Improving Concurrent Response Generation in Phoenix with Threading

Darshan T. · 2025-11-03T11:18:04.118Z

Hello Team, I've just started with phoenix. I'm using it for evaluations. I've created task function to generate response from my custom llm and using it with client.experiments.run_experiment(...) question is for repetitions=3, it is still processing 1 request at a time. Is there any way i can take advantage of threading with some x workers to generate concurrent response and make the process faster ? Thanks

Improving Concurrent Response Generation in Phoenix with Threading | Arize AI Community

15 comments

· Sorted by Oldest

RunLLM
APP
·
·
Yes, you can run experiments with concurrent workers by setting the concurrency parameter in client.experiments.run_experiment. The repetitions parameter controls how many times each example is run, while concurrency determines how many tasks are processed in parallel. For example, set concurrency=5 to use 5 workers: LBKSexperiment = client.experiments.run_experiment( dataset=dataset, task=my_task, experiment_name="greeting-experiment", repetitions=3, concurrency=5 # Number of concurrent workers )LBKSAccording to the documentation, this will process multiple requests in parallel, making the experiment faster. For more, see How to: Experiments - Repetitions and phoenix-client docs.Would you like more detail on how concurrency interacts with repetitions or how to optimize for your custom LLM?Sources: - https://github.com/Arize-ai/phoenix/blob/main/packages/phoenix-client/docs/source/index.md - How to: Experiments - Repetitions
💡 Hint: Mention RunLLM in the thread for followups.
Darshan T.
·
·
with AsyncClient sys:1: RuntimeWarning: coroutine 'AsyncExperiments.run_experiment' was never awaited
run it with async function but still it was processing 1 request at a time
Client has no arg called concurrency

Priyan

External

hi Darshan T., please use async_run_experiments

# Async experiments
from phoenix.client.experiments import async_run_experiment
from phoenix.client import AsyncClient

async def async_task(input):
    return f"Hello {input['name']}"

async_client = AsyncClient()
dataset = await async_client.datasets.get_dataset(dataset="my-dataset")

experiment = await async_run_experiment(
    dataset=dataset,
    task=async_task,
    experiment_name="greeting-experiment",
    concurrency=5  # Run 5 tasks concurrently
)

Priyan
External
·
·
concurrency up to 40 is generally good!
Darshan T.
·
·
SyntaxError: 'await' outside function if i wrap it around async function : httpcore.ConnectError: All connection attempts failed I verified Host and key is fine.
Priyan
External
·
·
Darshan T. I copied your exact code and it worked for me. Are you locally hosting or cloud hosting? As a reminder, even though you said you checked host and api key, I want to make sure you have the correct source of truth. You can find your hostname and API Keys in the Phoenix UI -> settings -> General did your dataset get pulled? (dataset = await async_client.get_dataset...)
Darshan T.
·
·
It is self-hosted
Priyan
External
·
·
what is printed when you print host? also, did the dataset get pulled, or is that also failing?
Darshan T.
·
·
Yes, it is able to pull the dataset. Host name is correct only

Priyan

External

can you pass in the async_client into async_run_experiment

experiment = await async_run_experiment(
        client=async_client,
        dataset=dataset,
        task=async_task,
        experiment_name="sample_test",
        concurrency=40
    )

Priyan
External
·
·
Darshan T.
Darshan T.
·
·
🔒[private user] yes that is working! Thanks one quick question. This concurrency is on response generation side (task method) or evaluation side or both?
Priyan
External
·
·
both!
Darshan T.
·
·
Cool! Let me play around it and again thanks for quick help!
Priyan
External
·
·
of course! hope it works out well.

RunLLM
APP
·
·
Yes, you can run experiments with concurrent workers by setting the concurrency parameter in client.experiments.run_experiment. The repetitions parameter controls how many times each example is run, while concurrency determines how many tasks are processed in parallel. For example, set concurrency=5 to use 5 workers: LBKSexperiment = client.experiments.run_experiment( dataset=dataset, task=my_task, experiment_name="greeting-experiment", repetitions=3, concurrency=5 # Number of concurrent workers )LBKSAccording to the documentation, this will process multiple requests in parallel, making the experiment faster. For more, see How to: Experiments - Repetitions and phoenix-client docs.Would you like more detail on how concurrency interacts with repetitions or how to optimize for your custom LLM?Sources: - https://github.com/Arize-ai/phoenix/blob/main/packages/phoenix-client/docs/source/index.md - How to: Experiments - Repetitions
💡 Hint: Mention RunLLM in the thread for followups.
Darshan T.
·
·
with AsyncClient sys:1: RuntimeWarning: coroutine 'AsyncExperiments.run_experiment' was never awaited
run it with async function but still it was processing 1 request at a time
Client has no arg called concurrency

Priyan

External

hi Darshan T., please use async_run_experiments

# Async experiments
from phoenix.client.experiments import async_run_experiment
from phoenix.client import AsyncClient

async def async_task(input):
    return f"Hello {input['name']}"

async_client = AsyncClient()
dataset = await async_client.datasets.get_dataset(dataset="my-dataset")

experiment = await async_run_experiment(
    dataset=dataset,
    task=async_task,
    experiment_name="greeting-experiment",
    concurrency=5  # Run 5 tasks concurrently
)

Priyan
External
·
·
concurrency up to 40 is generally good!
Darshan T.
·
·
SyntaxError: 'await' outside function if i wrap it around async function : httpcore.ConnectError: All connection attempts failed I verified Host and key is fine.
Priyan
External
·
·
Darshan T. I copied your exact code and it worked for me. Are you locally hosting or cloud hosting? As a reminder, even though you said you checked host and api key, I want to make sure you have the correct source of truth. You can find your hostname and API Keys in the Phoenix UI -> settings -> General did your dataset get pulled? (dataset = await async_client.get_dataset...)
Darshan T.
·
·
It is self-hosted
Priyan
External
·
·
what is printed when you print host? also, did the dataset get pulled, or is that also failing?
Darshan T.
·
·
Yes, it is able to pull the dataset. Host name is correct only

Priyan

External

can you pass in the async_client into async_run_experiment

experiment = await async_run_experiment(
        client=async_client,
        dataset=dataset,
        task=async_task,
        experiment_name="sample_test",
        concurrency=40
    )

Priyan
External
·
·
Darshan T.
Darshan T.
·
·
🔒[private user] yes that is working! Thanks one quick question. This concurrency is on response generation side (task method) or evaluation side or both?
Priyan
External
·
·
both!
Darshan T.
·
·
Cool! Let me play around it and again thanks for quick help!
Priyan
External
·
·
of course! hope it works out well.