How to Run Experiments in Parallel Using Async Functions
Please share instructions or examples how to run_experiment in parallel? async... Still getting sequential execution
Please share instructions or examples how to run_experiment in parallel? async... Still getting sequential execution
馃挕聽Hint: Mention RunLLM in the thread for followups.
Hey Igor B., I believe you should get parallel execution if you use the async run_experiment method on the new Phoenix client. It would look something like this when run in a notebook (in a script, you'll need to wrap the async invocation in a coroutine):
from phoenix.client import AsyncClient
async_client = AsyncClient()
model_name = "gpt-4o-mini" # loop through judge_models if you want or just try one for now
experiment = await async_client.experiments.run_experiment(
dataset=dataset,
task=partial(async_classify, model_name=model_name),
evaluators=[accuracy],
experiment_name=f"baseline-{model_name}",
experiment_description="Built-in hallucination eval with python SDK",
experiment_metadata={"sdk": "phoenix", "sdk_type": "python", "model": model_name},
concurrency=10,
# dry_run=10,
)We're still in the process of migrating our documentation to the new client.
unfortunately sill sequential run of tasks
One thing to call out. If you define a sync task, even if you use the async evaluate method, it will run synchronously. To get parallelism, your task needs to be a coroutine.
but when slightly modify task to
async def task(input: Any) -> str:
question = input["question"]
print(question, 'started processing')
#await asyncio.sleep(5)
print(question, 'finished processing')
return "answer"if runs synchronously
In general, async is most useful when the task is I/O-bound, e.g., when await suspends the task to wait for a network call, database query, or file operation. This allows other tasks to execute while the current task is blocked waiting for the I/O operation to complete, making efficient use of the single thread. For CPU-bound tasks that continuously perform calculations without I/O operations, async provides little benefit. Since these tasks lack natural suspension points where they would yield control, they monopolize the thread until completion, preventing other async tasks from running.