Arize AI Community Icon

Issues with Phoenix Upgrade: Resource Exhausted Warnings | Arize AI Community

Phoenix Support

Issues with Phoenix Upgrade: Resource Exhausted Warnings

·Oct 09, 2024 08:42 AM

Hello, we have been using phoenix for some time now. we recently had to upgrade the version of the package (and its dependencies accordingly). according to the documentation, we instantiate the instrumentation as follows:

from openinference.instrumentation.langchain import LangChainInstrumentor
from phoenix.otel import register

trace_provider = register()
instrumentor = LangChainInstrumentor()
instrumentor.instrument(trace_provider=trace_provider)

however, now we get some warnings on every call that also seem to "delay" the call and also seem to make the tracing inconsistent. the warnings look like that:

WARNING:opentelemetry.exporter.otlp.proto.grpc.exporter:Transient error StatusCode.RESOURCE_EXHAUSTED encountered while exporting traces to localhost:4317, retrying in 1s.
2024-10-09 08:33:44,263 - opentelemetry.exporter.otlp.proto.grpc.exporter - WARNING - Transient error StatusCode.RESOURCE_EXHAUSTED encountered while exporting traces to localhost:4317, retrying in 1s.
WARNING:opentelemetry.exporter.otlp.proto.grpc.exporter:Transient error StatusCode.RESOURCE_EXHAUSTED encountered while exporting traces to localhost:4317, retrying in 2s.
2024-10-09 08:33:45,270 - opentelemetry.exporter.otlp.proto.grpc.exporter - WARNING - Transient error StatusCode.RESOURCE_EXHAUSTED encountered while exporting traces to localhost:4317, retrying in 2s.
WARNING:opentelemetry.exporter.otlp.proto.grpc.exporter:Transient error StatusCode.RESOURCE_EXHAUSTED encountered while exporting traces to localhost:4317, retrying in 4s.
2024-10-09 08:33:47,278 - opentelemetry.exporter.otlp.proto.grpc.exporter - WARNING - Transient error StatusCode.RESOURCE_EXHAUSTED encountered while exporting traces to localhost:4317, retrying in 4s.
WARNING:opentelemetry.exporter.otlp.proto.grpc.exporter:Transient error StatusCode.RESOURCE_EXHAUSTED encountered while exporting traces to localhost:4317, retrying in 8s.
2

are we doing something wrong? many thanks in advance (RunLLM i edited my message :))

RunLLM
APP
·
The provided code snippet for instrumentation is correct. It uses the LangChainInstrumentor from openinference.instrumentation.langchain and the register function from phoenix.otel to set up tracing. Would you like more details on the setup or troubleshooting steps?Sources: - Phoenix GitHub - LangChain Integration
💡 Hint: Mention RunLLM in the thread for followups.
Roger Y.
·
Sorry you’re having trouble with the new version. Let me look into this for you. Can you let me know which version was working for you before, and which version you’re having problems with now?
Jason
·
Dvir B. My guess is there is some backup in the tracing data sent and that is backing up into the tracer itself. "register" uses the SimpleSpanProcessor by default which is blocking If you use:
register(batch=True)
That uses the asynchronous non blocking BatchSpanProcessor The fix above will cause the tracing to be non-blocking. It won't fix the reason the traces are backing up in the first place. We've seen come from a lot of things such as Network connections, etc...
Dvir B.
·
Roger Y. Hey and thanks for your response. we were working previously with 4.14.1 , and the newly installed version is 5.2.1 , which forced us to modify the instantiation of the instrumentation.
Dvir B.
·
Jason Thanks for your reply and suggestion. I tried that, and it seems like maybe there are maybe less warnings thrown (compared to the registration without batch=True,), but there are still a lot of warnings thrown, and the traces are not collected by the endpoint. Two important details that would maybe shed some more light:
the collector endpoint is locally hosted using python -m phoenix.server.main serve
It seems to occur in multi-modal calls to an LLM. when sending messages containing only text, traces are collected successfully (by the same endpoint) without any warning.
Jason
·
On multi modal we have seen issue with large size of traces not able to "upload" at network speed, generated at a rate faster then they can go up network, which then back pressures as traces build up. On the same node that seems less likely but should be wary we have seen that. This could also be something throughput related into the receiving DB. Are you Postgres or SQLLite
Dvir B.
·
sqlite
Dvir B.
·
can i somehow discard the images from the trace?
Roger Y.
·
you can try adding a config parameter as shown here (works for any instrumentor), but instead of the 200_000 number, you can set it to 0

·

Roger Y. thanks again! so, now my initialization looks like that:

trace_provider = register(batch=True)
instrumentor = LangChainInstrumentor()
instrumentor.instrument(trace_provider=trace_provider, config=TraceConfig(base64_image_max_length=0))

but i get the same behavior. I noticed that in the example you sent the initialization is performed as follows:

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor

endpoint = "http://127.0.0.1:4317"
tracer_provider = trace_sdk.TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(endpoint)))
tracer_provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))

should i modify my initialization? considering that the phoenix server is hosted locally

Roger Y.
·
yea, you can try that to see if it makes a difference. do you have the latest version of LangChainInstrumentor?
Roger Y.
·
you don’t need the ConsoleSpanExporter so you can remove that line
Roger Y.
·
and you can change SimpleSpanProcessor to BatchSpanProcessor
Dvir B.
·
and the endpoint variable there makes the env var PHOENIX_COLLECTOR_ENDPOINT unnecessary? they essentially should point to the same address?
Roger Y.
·
yes, that’s precisely where the env var eventually goes