RunLLM

Phoenix does not provide a built-in model serving feature analogous to MLflow's model serving. Phoenix focuses on experimentation, evaluation, tracing, and observability for LLM and AI applications, but it does not offer a native mechanism to deploy or serve models or tasks as production endpoints. The recommended approach is to use your own serving infrastructure (e.g., Docker, ECS, Lambda) for deploying tasks or models, and then instrument those services with Phoenix for observability and evaluation purposes. There is no "Phoenix-native" serving or deployment abstraction—Phoenix is designed to observe and evaluate, not to serve models directly or manage production inference endpoints. You can continue wrapping your task logic in a container and deploying it as you currently do, then use Phoenix to trace, evaluate, and monitor the production service by instrumenting your code and sending traces to Phoenix. This workflow is consistent with Phoenix's intended usage and documentation; there are no additional serving features or deployment recommendations beyond this in the official docs or guides. For more, see Phoenix documentation on experiments and deployment and self-hosting/deployment options.Would you like more detail on how to instrument your production service for Phoenix observability?Sources: - Phoenix documentation on experiments and deployment - self-hosting/deployment options

For Google AgentEngine deployments, instrumentation must be configured inside the remote agent module, not just in your main application. You need to ensure that the Phoenix tracing setup (including the register function and the relevant instrumentor, e.g., GoogleADKInstrumentor().instrument(tracer_provider=tracer_provider)) is included in your agent module code that runs on AgentEngine. Also, make sure all required packages are listed in your deployment requirements and that environment variables for the Phoenix endpoint and API key are set in the remote environment. Setting GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=False is correct, as Phoenix uses OpenTelemetry directly, not the built-in telemetry. See the official Phoenix Google ADK Agent Engine deployment guide for details.If traces appear locally but not in AgentEngine, it's likely the remote agent module is missing the tracing setup or the environment variables are not correctly passed to the remote environment. Double-check that the tracing code is executed in the remote agent context and that the Phoenix endpoint and API key are available there. Would you like a step-by-step deployment example?Sources: - Phoenix Google ADK Agent Engine deployment guide

Your question is a bit vague—could you clarify what specifically is not being calculated in Phoenix with your PostgreSQL backend? Are you referring to metrics, evaluations, latency, or something else? Please provide more details or error messages.If you are experiencing missing or incorrect calculations (such as metrics not updating), known issues include frontend UI not reflecting filtered metrics even though the backend returns correct data, or backend errors due to malformed data or large datasets. For example, header metrics like Total Cost may not update with filters in the UI, but the backend GraphQL query works as expected. Also, performance bottlenecks or malformed data in PostgreSQL can cause calculation failures or missing results in the UI. Please specify the calculation or metric you are referring to for more targeted help. According to this issue and this issue, these are common causes.Would you like more detail or troubleshooting steps for a specific calculation or metric?Sources: - https://github.com/Arize-ai/phoenix/issues/10494 - https://github.com/Arize-ai/phoenix/issues/11243

Phoenix should not be set up as a middleware or proxy between your API gateway and workflow execution. Instead, Phoenix (and its OpenInference instrumentation) must be integrated directly within your workflow application code. You configure Phoenix tracing (project name, session IDs, etc.) inside the workflow execution logic, so that traces are emitted as your workflow runs—not at the API gateway or as a network-level proxy. Phoenix cannot capture traces without code-level instrumentation in the workflow itself; it does not function like an HTTP proxy such as NGINX or Apache for telemetry capture see discussion.In summary: instrument your workflow code to emit traces to Phoenix, typically by initializing the tracer (with project/session info) at the start of each workflow execution. The API gateway simply routes requests; Phoenix must be configured in the workflow backend, not as a standalone layer between the gateway and the workflow see docs see docs.Would you like more detail on how to set up tracing in your workflow code?Sources: - https://github.com/Arize-ai/phoenix/issues/8380 - see docs - see docs

Phoenix does not send your traces, evaluations, datasets, or any application data to Arize or third parties when self-hosted; all data remains within your infrastructure, and you control where and how long it is stored. In Phoenix Cloud, your data is stored on US servers by default, and you can self-host Phoenix for full data residency control if needed. Basic web analytics (not trace or evaluation data) are collected by default, but you can opt out via environment variables. For persistent storage, Phoenix supports both SQLite (default, local) and PostgreSQL (recommended for production), and you can configure data retention policies or delete data manually. No user data is sent externally unless you explicitly export or back up traces or datasets yourself. For more, see Phoenix Privacy and Data Retention.Would you like more detail?Sources: - Phoenix Privacy - Data Retention - Phoenix Cloud Data Residency - Phoenix Architecture

Phoenix does not currently have a built-in visual prompt-version diff viewer; users must manually compare prompt versions to see changes. The ability to view or diff prompt versions in the UI is a frequently requested enhancement, and the Phoenix team has acknowledged its value for prompt management and review workflows, but there is no official feature or timeline for this yet. Your contribution aligns with open issues and user needs, and the Phoenix team welcomes community contributions that improve prompt management usability, including diffing prompt versions, as seen in related GitHub issues and discussions (source, source, source).You are encouraged to adapt your code to Phoenix's contribution guidelines and open a pull request. The team recommends starting with a "good first issue" or discussing your approach in a GitHub issue before submitting a PR. Be sure to follow the development and code review guidelines, including clear PR descriptions, small focused changes, and adherence to the code of conduct (source, source). Would you like a step-by-step guide on preparing your contribution for a PR?Sources: - https://github.com/Arize-ai/phoenix/issues/8383 - https://arize.com/docs/phoenix/resources/contribute-to-phoenix - https://github.com/Arize-ai/openinference/blob/main/CONTRIBUTING - https://github.com/Arize-ai/phoenix/issues/6622 - https://arize.com/docs/phoenix/prompt-engineering/tutorial/compare-prompt-versions

When upgrading Phoenix, if the PHOENIX_SECRET environment variable changes (is regenerated or overwritten), all existing API keys become invalid, even if they still appear in the UI. Generating new API keys resolves the issue because they are signed with the new secret. To prevent this in future upgrades, ensure that PHOENIX_SECRET is persisted and not changed between deployments—manage it with a persistent Kubernetes Secret or external secrets manager if using Helm or similar tools. This behavior is expected and documented in user reports and official guidance.This issue is not related to API key expiration or database migrations, but specifically to the secret used for signing and validating API keys. Changing the secret invalidates all previously issued keys, requiring regeneration of new ones for continued access. For more, see the linked discussions and documentation below.Would you like more detail on how to persist PHOENIX_SECRET or manage secrets during upgrades?Sources: - GitHub Issue: API Keys invalid after updating - GitHub Issue: Upgrade from 11.3 to 11.29 broke api key

The new session.id index for spans in Phoenix is not created concurrently, which can lock large tables and disrupt operations. The current migration does not use the CONCURRENTLY option for index creation, so table locking is expected for large datasets. There is no documented way to force concurrent index creation in Phoenix migrations as of now; this would require a change in future PRs to use CREATE INDEX CONCURRENTLY where appropriate for PostgreSQL, but this is not yet implemented or configurable in the current codebase or migration scripts according to the referenced PR and its discussion.Would you like more detail on how to manually create the index concurrently or guidance on best practices for large table migrations?Sources: - feat(db): add session.id index for spans across sqlite and postgres

To ensure each execution of your LangGraph workflow appears as a single trace (with all LLM/tool calls as child spans) in the Phoenix dashboard, you must invoke the workflow in a way that maintains a single root span per run. With Phoenix's OpenInference auto-instrumentation, this is achieved by calling your workflow as a callable (e.g., workflow(args)) rather than using .forward(args), which can flatten or separate traces. This ensures all node executions and LLM calls are grouped as spans under one trace for each workflow run, making the trace tree clear in the UI. No manual instrumentation is required if you follow this pattern and have auto-instrumentation enabled for LangGraph/LangChain.If you still see jumbled traces, double-check that you are not breaking context propagation (e.g., by using non-callable invocation patterns or by running nodes in separate processes without context propagation). For more, see the LangGraph Tracing guide and this Slack thread.Would you like more detail or a code example?Sources: - LangGraph Tracing - Phoenix Docs - Slack Discussion: dspy/langgraph trace grouping

Phoenix requires specific span attributes for cost tracking: llm.token_count.prompt, llm.token_count.completion, llm.token_count.total, llm.model_name, and llm.provider. If any are missing or mismatched, cost will not be calculated—even if token counts are present. Also, the model name and provider in your traces must match the patterns in your model pricing settings, including any custom entries you’ve added. Regex mismatches or typos in the model name pattern can prevent cost calculation. Double-check that all required attributes are present in your spans and that your model pricing configuration matches the trace data exactly. For more, see the cost tracking documentation and related troubleshooting discussion.If all attributes and pricing settings are correct but cost is still missing, inspect the raw span data in Phoenix to confirm the attributes are present and correctly formatted. Some instrumentations or LLM providers may not emit all required fields, or may use slightly different model names. Upgrading to the latest instrumentation packages and ensuring your model pricing regex matches the exact model name in the trace often resolves the issue. Would you like a step-by-step checklist or code example for debugging this further?Sources: - Cost Tracking - Phoenix - Phoenix GitHub Issue: Cost not shown with DSPy instrumentation