Arize AI Community Icon

Home
Events

Arize AX Releases
Arize AX Support
Arize News
Arize Observe
Discussions
Introductions
Phoenix Support

Powered by Tightknit

Phoenix Support

Phoenix Support

Easy Timeout Issues with nest_asyncio: Hallucinations and More

·Aug 08, 2024 01:01 AM

the nest_asyncio seems to be really easy to timeout --> this was a run of hallucinations, qa, and relevance on 9 queries

4 comments

· Sorted by Oldest

RunLLM
APP
·
The issue with nest_asyncio timing out during evaluations is likely due to the concurrency and synchronization primitives used in the llm_classify function. Running evaluations in a Django environment, which typically uses worker threads, can exacerbate this issue.To work around this, you can run llm_classify synchronously by setting run_sync=True. This will avoid using any synchronization primitives that can only be created in the main thread, but it may be slower.LBKShallucination_classifications = llm_classify( dataframe=df, template=HALLUCINATION_PROMPT_TEMPLATE, model=azure_model, rails=rails, concurrency=20, run_sync=True )["label"].tolist()LBKSWould you like more details on this workaround?Sources: - https://arize-ai.slack.com/archives/C04R3GXC8HK/p1713961598.350669 - https://github.com/Arize-ai/phoenix/issues/2309 Note: I wasn’t able to find highly relevant data sources, but above are a few potentially relevant links.
💡 Hint: Mention RunLLM in the thread for followups.
Dustin N.
·
hi Yujian T. I've never seen nest_asyncio cause an issue itself, but this normally happens if you are being extremely rate limited for some reason. Because you're making so few queries this doesn't seem to be the case, can you run this on one example to see if it's working?
Yujian T.
·
this was on run_evals
Yujian T.
·
the 9 examples ran faster without nest_asyncio 🤷