Issue with async_run_experiment: Error Message Explained

Jacob M. · 2025-10-31T22:03:13.088Z

Howdy! I have an issue with async_run_experiment. I was using run_experiment but need to have some concurrency. I switched to now run AsyncClient.experiments.run_experiment(). However, now I keep getting this error Retries exhausted after 1 attempts: string indices must be integers, not 'str' at this step in the process task_runs, _execution_details = await executor.execute(test_cases) It looks like maybe there's a bug in the async_run_experiment() function itself?

Issue with async_run_experiment: Error Message Explained | Arize AI Community

9 comments

· Sorted by Oldest

RunLLM
APP
·
·
The error `Retries exhausted after 1 attempts: string indices must be integers, not 'str'` when using `async_run_experiment` is typically caused by a mismatch in the expected structure of the dataset object—specifically, the new Phoenix client expects datasets in a different format (list vs. dict) than the old client. This is not a bug in `async_run_experiment` itself, but rather a client usage issue. To resolve, use the new client API: instantiate your client with `from phoenix.client import Client` and access datasets via `client.datasets.get_dataset(...)` instead of the old `px.Client().get_dataset(...)` pattern. See the code and explanation in the issue below: ``` from phoenix.client import Client client = Client() dataset = client.datasets.get_dataset(dataset=...) ``` According to <https://github.com/Arize-ai/phoenix/issues/9301|this issue discussion>, this change should resolve the error. Would you like more detail on the root cause or how to migrate your code? Sources: - <https://github.com/Arize-ai/phoenix/issues/9301|https://github.com/Arize-ai/phoenix/issues/9301>
💡 Hint: Mention RunLLM in the thread for followups.
Srilakshmi
External
·
·
hi Jacob, I believe that error usually means the dataset isn’t in the new client format. How are you uploading your dataset? you could try fetching it with the updated API:
from phoenix.client import Client, AsyncClient client = Client() dataset = client.datasets.get_dataset(dataset="<your_dataset_id>") async_client = AsyncClient() await async_client.experiments.run_experiment(dataset=dataset, task=task_fn)
let me know if that helps clear the string indices must be integers issue?
Jacob M.
·
·
That seemed to work.
❤️1
Jacob M.
·
·
I should have check here sooner. I spent a lot of time trying to debug this. Lol
Jacob M.
·
·
🔒[private user] - run_experiment would return an object with a summaries object which I could determine the average score of the experiment. async_run_experiment doesn't seem to have that.
Jacob M.
·
·
Any best practice on how to get the average score of the run without looping over the RanExperiment object and tallying up?
Srilakshmi
External
·
·
sorry for the late message here, yeah, async_run_experiment() doesn’t return the summaries field. best practice is to compute it client-side, i think something like this:
from statistics import mean avg_score = mean(t.metrics["score"] for t in task_runs if "score" in t.metrics)
should be able to mimic the same result as the sync summaries.average_score .
Jacob M.
·
·
Thank you.
Jacob M.
·
·
I'm hitting an issue now when trying a dry run with the async_run_experiment. An error message during the evaluation phase ValueError: Failed to fetch dataset for experiment: DRY_RUN

RunLLM
APP
·
·
The error `Retries exhausted after 1 attempts: string indices must be integers, not 'str'` when using `async_run_experiment` is typically caused by a mismatch in the expected structure of the dataset object—specifically, the new Phoenix client expects datasets in a different format (list vs. dict) than the old client. This is not a bug in `async_run_experiment` itself, but rather a client usage issue. To resolve, use the new client API: instantiate your client with `from phoenix.client import Client` and access datasets via `client.datasets.get_dataset(...)` instead of the old `px.Client().get_dataset(...)` pattern. See the code and explanation in the issue below: ``` from phoenix.client import Client client = Client() dataset = client.datasets.get_dataset(dataset=...) ``` According to <https://github.com/Arize-ai/phoenix/issues/9301|this issue discussion>, this change should resolve the error. Would you like more detail on the root cause or how to migrate your code? Sources: - <https://github.com/Arize-ai/phoenix/issues/9301|https://github.com/Arize-ai/phoenix/issues/9301>
💡 Hint: Mention RunLLM in the thread for followups.
Srilakshmi
External
·
·
hi Jacob, I believe that error usually means the dataset isn’t in the new client format. How are you uploading your dataset? you could try fetching it with the updated API:
from phoenix.client import Client, AsyncClient client = Client() dataset = client.datasets.get_dataset(dataset="<your_dataset_id>") async_client = AsyncClient() await async_client.experiments.run_experiment(dataset=dataset, task=task_fn)
let me know if that helps clear the string indices must be integers issue?
Jacob M.
·
·
That seemed to work.
❤️1
Jacob M.
·
·
I should have check here sooner. I spent a lot of time trying to debug this. Lol
Jacob M.
·
·
🔒[private user] - run_experiment would return an object with a summaries object which I could determine the average score of the experiment. async_run_experiment doesn't seem to have that.
Jacob M.
·
·
Any best practice on how to get the average score of the run without looping over the RanExperiment object and tallying up?
Srilakshmi
External
·
·
sorry for the late message here, yeah, async_run_experiment() doesn’t return the summaries field. best practice is to compute it client-side, i think something like this:
from statistics import mean avg_score = mean(t.metrics["score"] for t in task_runs if "score" in t.metrics)
should be able to mimic the same result as the sync summaries.average_score .
Jacob M.
·
·
Thank you.
Jacob M.
·
·
I'm hitting an issue now when trying a dry run with the async_run_experiment. An error message during the evaluation phase ValueError: Failed to fetch dataset for experiment: DRY_RUN