Single Task Execution with Multiple Evaluators in Experimentation

·May 23, 2025 05:45 AM

Hi Team, I was running experiment with 1 example, 3 evaluators It suppose to run to get result only 1 time, right ? But with 3 evaluators, it keep run task to get result again again How could I run task only 1 time, then eval with 3 evaluator Thanks in advance

run_experiment( dataset, run_summarizer, evaluators=evaluators, experiment_name="subgraph_summarizer", concurrency=3, )

6 comments

· Sorted by Oldest

Mikyo
·
Hmm I’m not sure I know. Does it work with concurrency 1? If so it might be a bug. We can definitely help you out in Phoenix Support or GitHub!
❤️1
Sorratat S.
·
I am not sure, may be I got this error that might reactivate task again and again ? Worker timeout, requeuing
Mikyo
·
Can you try adding the timeout parameter? https://github.com/Arize-ai/phoenix/blob/main/src/phoenix/experiments/functions.py
💯1
Sorratat S.
·
Thanks so much, let's me try I checked Phoenix code timeout is set at 120 as default
Sorratat S.
·
I currently used arize-phoenix = "^8.32.1" which can not pass agument of timeout through run_experiment 😓
Mikyo
·
dang, I bet it's timing out?
😓1

Mikyo
·
Hmm I’m not sure I know. Does it work with concurrency 1? If so it might be a bug. We can definitely help you out in Phoenix Support or GitHub!
❤️1
Sorratat S.
·
I am not sure, may be I got this error that might reactivate task again and again ? Worker timeout, requeuing
Mikyo
·
Can you try adding the timeout parameter? https://github.com/Arize-ai/phoenix/blob/main/src/phoenix/experiments/functions.py
💯1
Sorratat S.
·
Thanks so much, let's me try I checked Phoenix code timeout is set at 120 as default
Sorratat S.
·
I currently used arize-phoenix = "^8.32.1" which can not pass agument of timeout through run_experiment 😓
Mikyo
·
dang, I bet it's timing out?
😓1