Silent Tracing Failures in SpanAttributes: Error Logging Needed

Anton A. · 2024-08-20T20:50:10.603Z

Hey, when logging an incorrect value (i.e. [16]) to SpanAttributes.LLM_TOKEN_COUNT_COMPLETION there is a silent tracing failure and the trace doesn’t show up in the UI. Upon inspecting the docker logs of the deployment I found: Traceback (most recent call last): File "/phoenix/env/phoenix/db/bulk_inserter.py", line 200, in _insert_spans result = await insert_span(session, span, project_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/phoenix/env/phoenix/db/insertion/span.py", line 92, in insert_span cumulative_llm_token_count_completion += cast(int, accumulation[2] or 0) TypeError: 'int' object is not iterable Would be helpful if the client logged an error with a hint instead of just silently failing

Silent Tracing Failures in SpanAttributes: Error Logging Needed | Arize AI Community

10 comments

· Sorted by Oldest

RunLLM
APP
·
The issue arises because SpanAttributes.LLM_TOKEN_COUNT_COMPLETION expects a specific type, and logging an incorrect value like [16] causes a TypeError. This error is not handled gracefully, leading to a silent failure.To address this, you can modify the code to include error handling that logs a more informative message. Here’s a potential solution: 1. Modify the insert_span function to handle the TypeError and log an appropriate message.LBKSimport loggingLBKSasync def insert_span( session: AsyncSession, span: Span, project_name: str, ) -> Optional[SpanInsertionEvent]: try: # Existing code... cumulative_llm_token_count_completion = cast( int, get_attribute_value(span.attributes, SpanAttributes.LLM_TOKEN_COUNT_COMPLETION) or 0 ) if accumulation := ( await session.execute( select( func.sum(models.Span.cumulative_error_count), func.sum(models.Span.cumulative_llm_token_count_prompt), func.sum(models.Span.cumulative_llm_token_count_completion), ).where(models.Span.parent_id == span.context.span_id) ) ).first(): cumulative_error_count += cast(int, accumulation[0] or 0) cumulative_llm_token_count_prompt += cast(int, accumulation[1] or 0) cumulative_llm_token_count_completion += cast(int, accumulation[2] or 0) # More existing code... except TypeError as e: logging.error(f"TypeError encountered: {e}. Check the value of SpanAttributes.LLM_TOKEN_COUNT_COMPLETION.") raiseLBKSWould you like more details on how to implement this or any other aspect of Phoenix?Sources: - https://github.com/Arize-ai/phoenix/issues/3188 - https://github.com/Arize-ai/phoenix/issues/2012
💡 Hint: Mention RunLLM in the thread for followups.
Mikyo
·
Hmm, sorry you are facing that issue. It doesn't ring a bell immediately. Let me dig in a bit.
Mikyo
·
Hey Anton A. are you logging a number as the value? It looks like it might not be?
Anton A.
·
I was logging an array by accident
Anton A.
·
I was able to fix it, but it was hard to debug (needed to check the docker logs)
Mikyo
·
Ah gotcha. Yeah def. could be better. Let us know if you have any ideas
Anton A.
·
Could the span be logged with an error? What’s weird is that nothing showed in the phoenix UI when this error occurred
Anton A.
·
Either that or it would be good if there was some logging in the tracing client (altho this might be hard if there’s no validation on client-side)
Mikyo
·
Yeah client side validation might be harder - since it's still technically a valid span.
Mikyo
·
I'll file a ticket - thanks for the feedback!
💯1
1

RunLLM
APP
·
The issue arises because SpanAttributes.LLM_TOKEN_COUNT_COMPLETION expects a specific type, and logging an incorrect value like [16] causes a TypeError. This error is not handled gracefully, leading to a silent failure.To address this, you can modify the code to include error handling that logs a more informative message. Here’s a potential solution: 1. Modify the insert_span function to handle the TypeError and log an appropriate message.LBKSimport loggingLBKSasync def insert_span( session: AsyncSession, span: Span, project_name: str, ) -> Optional[SpanInsertionEvent]: try: # Existing code... cumulative_llm_token_count_completion = cast( int, get_attribute_value(span.attributes, SpanAttributes.LLM_TOKEN_COUNT_COMPLETION) or 0 ) if accumulation := ( await session.execute( select( func.sum(models.Span.cumulative_error_count), func.sum(models.Span.cumulative_llm_token_count_prompt), func.sum(models.Span.cumulative_llm_token_count_completion), ).where(models.Span.parent_id == span.context.span_id) ) ).first(): cumulative_error_count += cast(int, accumulation[0] or 0) cumulative_llm_token_count_prompt += cast(int, accumulation[1] or 0) cumulative_llm_token_count_completion += cast(int, accumulation[2] or 0) # More existing code... except TypeError as e: logging.error(f"TypeError encountered: {e}. Check the value of SpanAttributes.LLM_TOKEN_COUNT_COMPLETION.") raiseLBKSWould you like more details on how to implement this or any other aspect of Phoenix?Sources: - https://github.com/Arize-ai/phoenix/issues/3188 - https://github.com/Arize-ai/phoenix/issues/2012
💡 Hint: Mention RunLLM in the thread for followups.
Mikyo
·
Hmm, sorry you are facing that issue. It doesn't ring a bell immediately. Let me dig in a bit.
Mikyo
·
Hey Anton A. are you logging a number as the value? It looks like it might not be?
Anton A.
·
I was logging an array by accident
Anton A.
·
I was able to fix it, but it was hard to debug (needed to check the docker logs)
Mikyo
·
Ah gotcha. Yeah def. could be better. Let us know if you have any ideas
Anton A.
·
Could the span be logged with an error? What’s weird is that nothing showed in the phoenix UI when this error occurred
Anton A.
·
Either that or it would be good if there was some logging in the tracing client (altho this might be hard if there’s no validation on client-side)
Mikyo
·
Yeah client side validation might be harder - since it's still technically a valid span.
Mikyo
·
I'll file a ticket - thanks for the feedback!
💯1
1