RunLLM

To ensure each execution of your LangGraph workflow appears as a single trace (with all LLM/tool calls as child spans) in the Phoenix dashboard, you must invoke the workflow in a way that maintains a single root span per run. With Phoenix's OpenInference auto-instrumentation, this is achieved by calling your workflow as a callable (e.g., workflow(args)) rather than using .forward(args), which can flatten or separate traces. This ensures all node executions and LLM calls are grouped as spans under one trace for each workflow run, making the trace tree clear in the UI. No manual instrumentation is required if you follow this pattern and have auto-instrumentation enabled for LangGraph/LangChain.If you still see jumbled traces, double-check that you are not breaking context propagation (e.g., by using non-callable invocation patterns or by running nodes in separate processes without context propagation). For more, see the LangGraph Tracing guide and this Slack thread.Would you like more detail or a code example?Sources: - LangGraph Tracing - Phoenix Docs - Slack Discussion: dspy/langgraph trace grouping

Phoenix requires specific span attributes for cost tracking: llm.token_count.prompt, llm.token_count.completion, llm.token_count.total, llm.model_name, and llm.provider. If any are missing or mismatched, cost will not be calculated—even if token counts are present. Also, the model name and provider in your traces must match the patterns in your model pricing settings, including any custom entries you’ve added. Regex mismatches or typos in the model name pattern can prevent cost calculation. Double-check that all required attributes are present in your spans and that your model pricing configuration matches the trace data exactly. For more, see the cost tracking documentation and related troubleshooting discussion.If all attributes and pricing settings are correct but cost is still missing, inspect the raw span data in Phoenix to confirm the attributes are present and correctly formatted. Some instrumentations or LLM providers may not emit all required fields, or may use slightly different model names. Upgrading to the latest instrumentation packages and ensuring your model pricing regex matches the exact model name in the trace often resolves the issue. Would you like a step-by-step checklist or code example for debugging this further?Sources: - Cost Tracking - Phoenix - Phoenix GitHub Issue: Cost not shown with DSPy instrumentation

There have been no recent changes in OpenInference or the Google ADK instrumentation that would intentionally introduce delays or cause spans to appear as orphans for extended periods. Orphaned spans and delayed trace assembly in the UI typically result from missing or late-arriving parent spans, or ingestion/indexing delays on the backend. This can happen if spans are sent out of order, if the process exits before all spans are flushed, or if the backend (Arize AX) is experiencing temporary indexing lag. The OpenInference instrumentation itself does not queue or delay spans after export—traces are streamed as soon as they are generated (source, CHANGELOG).If you are seeing traces eventually resolve from orphaned to correctly nested after a long delay, this is most likely due to backend ingestion/indexing latency or temporary issues with trace assembly in Arize AX, not a change in the OpenInference or Google ADK client libraries. No recent OpenInference releases mention changes to trace export timing or parent-child span handling for Google ADK (CHANGELOG). Would you like more detail on troubleshooting or backend-specific checks?Sources: - https://google.github.io/adk-docs/observability/arize-ax/ - CHANGELOG

Phoenix currently does not support adding web-search as a tool in the prompt management UI or Playground; only function tools and custom tools are supported for tool calls. Replay of traces involving web-search is not possible because Phoenix's tool integration is limited to function-based tools, and web-search is not exposed as a selectable tool type in the UI or prompt schema [source, source, source].There is no documented workaround to enable web-search as a tool or to replay traces with web-search in Phoenix as of now. You may monitor for future updates or request this feature via GitHub issues if it is critical for your workflow [source]. Would you like more detail on prompt tool configuration or trace replay limitations?Sources: - https://arize.com/docs/phoenix/prompt-engineering/overview-prompts - https://arize.com/docs/phoenix/prompt-engineering/overview-prompts/prompts-in-code - https://arize-ai.github.io/phoenix/interfaces/_arizeai_phoenix-client.prompts_sdks_toOpenAI.ChatCompletionCreateParams.WebSearchOptions-1.html

It seems likely that Phoenix does not perform automatic deduplication of identical input/output data across deeply-nested spans; each span's attributes (including large context or results) are stored as-is, even if repeated in the trace. There is no mention of deduplication logic for span input/output in the available documentation or code discussions for Mastra or Phoenix in general. If you have repeated large payloads, they will be stored for each span that includes them, potentially increasing storage usage. Would you like more detail on how to manage or mitigate this?Regarding Postgres, my understanding is that Phoenix relies on Postgres' built-in TOAST (The Oversized-Attribute Storage Technique) for handling large row data. When a row exceeds 8KB, Postgres automatically TOASTs the data, storing large values out-of-line and keeping a pointer in the main table, so you do not need to configure this separately for Phoenix. This is standard Postgres behavior and is not altered by Phoenix's persistence layer configuration. Would you like more technical detail on how TOAST works or how Phoenix interacts with it?Sources: - https://github.com/Arize-ai/phoenix/pull/5250

Ollama models do not consistently support structured outputs (i.e., response schemas or function/tool calling) in the same way as OpenAI or Anthropic models. Many Ollama models either lack support for structured output or do not reliably follow output schemas, which can result in parsing errors or unstructured responses. This is a known limitation and is reflected in both user reports and official documentation—see, for example, the error message "does not support tools" when using certain Ollama models with Phoenix's Playground, and the need to check model compatibility for tool/function calling or structured outputs. You can find a list of Ollama models and their capabilities (including function calling support) in the model registry, but even models marked as supporting function calling may not always adhere to strict output schemas in practice. For best results, use models and providers known to support structured outputs, such as OpenAI or Anthropic, or experiment with prompt engineering to improve output consistency, though this is not always reliable with Ollama models. According to Phoenix release notes and GitHub issues, this is a common limitation.Would you like more detail on workarounds or model-specific capabilities?Sources: - https://github.com/Arize-ai/phoenix/issues/6346 - https://github.com/Arize-ai/openinference/blob/main/python/instrumentation/openinference-instrumentation-smolagents/tests/openinference/instrumentation/smolagents/cassettes/test_instrumentor/TestModels.test_litellm_reasoning_model_has_expected_attributes.yaml - Phoenix release notes

Yes, adding indexes to your database can improve query performance, especially for filters on metadata or attributes. For example, users have reported improved performance by adding a GIN index on metadata fields in PostgreSQL, and Phoenix maintainers have discussed adding expression indexes for common filters like session.id [source; source].However, you must ensure the index matches your query pattern (e.g., for JSONB fields, use a GIN index). Would you like detailed steps for adding a specific index in PostgreSQL?Sources: - https://github.com/Arize-ai/phoenix/issues/9460 - https://github.com/Arize-ai/phoenix/pull/11028

If you're on Phoenix v12.32.2 and still experiencing performance issues with span queries in the UI, it's likely due to the volume of spans or the complexity of your filters, not a version bug. Even with recent optimizations, large projects or metadata-heavy queries can cause slowdowns or timeouts [source].To mitigate, reduce the query time window, use more selective filters, or batch queries via the API with limits. For very large datasets, the API is more reliable than the UI for heavy queries. Would you like step-by-step guidance on batching or optimizing your queries?Sources: - https://github.com/Arize-ai/phoenix/issues/7451

Filtering spans in the Phoenix WebUI with a query like metadata['a'] == 'b' can be slow if your project contains a large number of spans, as metadata filters are applied server-side and may require scanning many records. Performance issues are common in such cases, especially with millions of spans or large attribute payloads [source].To improve performance, try reducing the time window, limiting the number of spans returned, or upgrading to the latest Phoenix version, which includes optimizations for span queries and lazy loading. For very large datasets, using the API with filters, batching, and limits is recommended over the UI for heavy queries [source]. Would you like more detail on batching or API-based querying?Sources: - https://github.com/Arize-ai/phoenix/issues/4006 - https://github.com/Arize-ai/phoenix/issues/7451

Thank you for the update! I'm glad to hear that setting up the OpenTelemetry collector with the remap plugin has resolved your issue and that traces are now being properly handled in both Phoenix and your other backend, with all UI/UX features working as expected.If you have any further questions about Phoenix, need help with advanced tracing, evaluation, or want to share more about your project as it evolves, feel free to reach out. Best of luck with your continued development!If you need more detailed guidance or run into any Phoenix-specific challenges, just let us know.