Howdy! I have an issue with async_run_experiment. I was using run_experiment but need to have some concurrency. I switched to now run AsyncClient.experiments.run_experiment(). However, now I keep getting this error Retries exhausted after 1 attempts: string indices must be integers, not 'str' at this step in the process task_runs, _execution_details = await executor.execute(test_cases) It looks like maybe there's a bug in the async_run_experiment() function itself?
The error Retries exhausted after 1 attempts: string indices must be integers, not 'str' when using async_run_experiment is typically caused by a mismatch in the expected structure of the dataset object鈥攕pecifically, the new Phoenix client expects datasets in a different format (list vs. dict) than the old client. This is not a bug in async_run_experiment itself, but rather a client usage issue. To resolve, use the new client API: instantiate your client with from phoenix.client import Client and access datasets via client.datasets.get_dataset(...) instead of the old px.Client().get_dataset(...) pattern. See the code and explanation in the issue below: LBKSfrom phoenix.client import ClientLBKSclient = Client() dataset = client.datasets.get_dataset(dataset=...)LBKSAccording to this issue discussion, this change should resolve the error.Would you like more detail on the root cause or how to migrate your code?Sources:
- https://github.com/Arize-ai/phoenix/issues/9301
馃挕聽Hint: Mention RunLLM in the thread for followups.
hi Jacob, I believe that error usually means the dataset isn鈥檛 in the new client format. How are you uploading your dataset? you could try fetching it with the updated API:
from phoenix.client import Client, AsyncClient
client = Client()
dataset = client.datasets.get_dataset(dataset="<your_dataset_id>")
async_client = AsyncClient()
await async_client.experiments.run_experiment(dataset=dataset, task=task_fn)let me know if that helps clear the string indices must be integers issue?
I should have check here sooner. I spent a lot of time trying to debug this. Lol
馃敀[private user] - run_experiment would return an object with a summaries object which I could determine the average score of the experiment. async_run_experiment doesn't seem to have that.
Any best practice on how to get the average score of the run without looping over the RanExperiment object and tallying up?
sorry for the late message here, yeah, async_run_experiment() doesn鈥檛 return the summaries field. best practice is to compute it client-side, i think something like this:
from statistics import mean
avg_score = mean(t.metrics["score"] for t in task_runs if "score" in t.metrics)should be able to mimic the same result as the sync summaries.average_score .
Thank you.
I'm hitting an issue now when trying a dry run with the async_run_experiment. An error message during the evaluation phase ValueError: Failed to fetch dataset for experiment: DRY_RUN
