Arize AI Community Icon

Home
Events

Arize AX Releases
Arize AX Support
Arize News
Arize Observe
Discussions
Introductions
Phoenix Support

Powered by Tightknit

Phoenix Support

Phoenix Support

Handling Phoenix Server Unreachability in Your RAG Application on GCP

·Jun 26, 2024 08:46 AM

Hi all, I'm working on a RAG application deployed on GCP with the phoenix server in a separate service. My question is, how can I set in the code of my app to not stall when the Phoenix server is unreachable?

7 comments

· Sorted by Oldest

Nabeegh A.
·
Hi Roméo I. - Could you clarify what exactly do you mean by "not stall". Do you want the exporter to wait until the service is reachable? Or not cause an error? With that said, I think this issue is not necessarily a phoenix issue. You can use some tools from the programming language you are using to tackle this.
Roméo I.
·
Hi Nabeegh, thanks for the reply, indeed it's not a phoenix issue, I just wanted to know the good practices in this situation. I found a work around 😉
Nabeegh A.
·
Sounds good Roméo I.. Could you share what workaround you used? Always interesting to see some clever tricks
Mikyo
·
Thanks Nabeegh for digging in! 🙏 Roméo I. Good to see ya. In general OTel shouldn't be causing your app to stall in any way. It's the basic requirements in the error handling principles of OTEL https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/error-handling.md#basic-error-handling-principles. The span processor implementation should make sure the any exception that might occur during export is caught and only logged for the visibility but never thrown into main application. You can verify it here for SimpleSpanProcessor and here for BatchSpanProcessor. Let me know if you don't find the above case to be true. Definitely would like to bottom this one out.
Roméo I.
·
Hi Mikyo, good to see you too ! Thanks for this very clear response as always 😉 So as you mentioned, the stalling problem was probably coming from somewhere else.
Roméo I.
·
Nabeegh A. Just for you to know, we made a health-check to our phoenix server before each call from our chatbot query and if the server doesn't respond we uninstrument our instrumentor. Sounds quite hacky when I say it out loud ahah.
Mikyo
·
Oh interesting. Phoenix does export prometheus metrics too if you are ever interested in monitoring it. Glad you got something working - LMK if you ever need a more graceful solution.
👍1