How to Run Experiments in Parallel Using Async Functions
Please share instructions or examples how to run_experiment in parallel? async... Still getting sequential execution
Please share instructions or examples how to run_experiment in parallel? async... Still getting sequential execution
To run experiments in parallel (async) with Phoenix, use the run_experiment method from the Python client and set the concurrency parameter to greater than 1. This enables concurrent execution of tasks across dataset examples. Both synchronous and asynchronous tasks are supported, and you can use AsyncClient for async workflows. Example: LBKSfrom phoenix.client import AsyncClientLBKSclient = AsyncClient() await client.experiments.run_experiment( dataset=dataset, task=my_task, experiment_name="parallel-experiment", concurrency=5 # Number of parallel workers )LBKSMake sure your task function is async if using AsyncClient. If you still see sequential execution, check that concurrency is set and your environment supports async execution. For more, see Experiments Module in phoenix-client and example code.Would you like a step-by-step breakdown or troubleshooting tips?Sources:
- Experiments Module in phoenix-client
- phoenix.client.experiments example code
馃挕聽Hint: Mention RunLLM in the thread for followups.
Hey Igor B., I believe you should get parallel execution if you use the async run_experiment method on the new Phoenix client. It would look something like this when run in a notebook (in a script, you'll need to wrap the async invocation in a coroutine):
from phoenix.client import AsyncClient
async_client = AsyncClient()
model_name = "gpt-4o-mini" # loop through judge_models if you want or just try one for now
experiment = await async_client.experiments.run_experiment(
dataset=dataset,
task=partial(async_classify, model_name=model_name),
evaluators=[accuracy],
experiment_name=f"baseline-{model_name}",
experiment_description="Built-in hallucination eval with python SDK",
experiment_metadata={"sdk": "phoenix", "sdk_type": "python", "model": model_name},
concurrency=10,
# dry_run=10,
)We're still in the process of migrating our documentation to the new client.
unfortunately sill sequential run of tasks
One thing to call out. If you define a sync task, even if you use the async evaluate method, it will run synchronously. To get parallelism, your task needs to be a coroutine.
but when slightly modify task to
async def task(input: Any) -> str:
question = input["question"]
print(question, 'started processing')
#await asyncio.sleep(5)
print(question, 'finished processing')
return "answer"if runs synchronously
In general, async is most useful when the task is I/O-bound, e.g., when await suspends the task to wait for a network call, database query, or file operation. This allows other tasks to execute while the current task is blocked waiting for the I/O operation to complete, making efficient use of the single thread. For CPU-bound tasks that continuously perform calculations without I/O operations, async provides little benefit. Since these tasks lack natural suspension points where they would yield control, they monopolize the thread until completion, preventing other async tasks from running.