Need Help with Real-Time Evaluation Implementations in Phoenix

Rafael · 2025-05-29T23:20:22.916Z

Hello everyone, I would like your help with real-time evaluation implementations. I have already managed to send my application traces to Phoenix, but I am having difficulty creating a job to run every x amount of time and perform the span evaluation.

Need Help with Real-Time Evaluation Implementations in Phoenix | Arize AI Community

6 comments

· Sorted by Oldest

Jason
·
Rafael Any details on the issue? Confirming you want to run this job yourself. We can provide some tips Wanted to note if your open to SaaS Arize Ax does have online Evals
Rafael
·
Hello @jason, I'm using Phoenix, but when I try to run the evaluation it calls the `get_qa_with_reference` method which always returns empty since I don't have a span with this span_kid.
Rafael
·
Yes, I am using the OpenInference specification to send my application traces. I needed to make a custom implementation, since I am using Spring AI
Rafael
·
Hello, I have the following trace structure, in some cases including the tools span.

Rafael

The attributes for the request span are.

{
    "input": {
        "value": "Hellow"
    },
    "output": {
        "value": "Hello, how can I help you?"
    },
    "session": {
        "id": "6c7916d6-ccbf-440a-a805-8a1bb3c4dc76"
    },
    "openinference": {
        "span": {
            "kind": "CHAIN"
        }
    }
}

Rafael

The other two spans follow the same attribute structure, as they are LLM.

{
    "llm": {
        "model_name": "gpt-4o-mini-2024-07-18",
        "token_count": {
            "total": 1630,
            "prompt": 1610,
            "completion": 20,
            "prompt_details": {
                "audio": 0,
                "cache_read": 1408
            },
            "completion_details": {
                "audio": 0,
                "reasoning": 0
            }
        },
        "input_messages": [
            {
                "message": {
                    "role": "system",
                    "content": "You are AgentMatcher..."
                }
            },
            {
                "message": {
                    "role": "user",
                    "content": "Hello"
                }
            }
        ],
        "output_messages": [
            {
                "message": {
                    "role": "assistant",
                    "content": "{\n  \"confidence\": 0.95,\n  \"selectedAgent\": \"agent\"\n}"
                }
            }
        ]
    },
    "sessionId": "6c7916d6-ccbf-440a-a805-8a1bb3c4dc76",
    "openinference": {
        "span": {
            "kind": "LLM"
        }
    }
}

Jason
·
Rafael Any details on the issue? Confirming you want to run this job yourself. We can provide some tips Wanted to note if your open to SaaS Arize Ax does have online Evals
Rafael
·
Hello @jason, I'm using Phoenix, but when I try to run the evaluation it calls the `get_qa_with_reference` method which always returns empty since I don't have a span with this span_kid.
Rafael
·
Yes, I am using the OpenInference specification to send my application traces. I needed to make a custom implementation, since I am using Spring AI
Rafael
·
Hello, I have the following trace structure, in some cases including the tools span.

Rafael

The attributes for the request span are.

{
    "input": {
        "value": "Hellow"
    },
    "output": {
        "value": "Hello, how can I help you?"
    },
    "session": {
        "id": "6c7916d6-ccbf-440a-a805-8a1bb3c4dc76"
    },
    "openinference": {
        "span": {
            "kind": "CHAIN"
        }
    }
}

Rafael

The other two spans follow the same attribute structure, as they are LLM.

{
    "llm": {
        "model_name": "gpt-4o-mini-2024-07-18",
        "token_count": {
            "total": 1630,
            "prompt": 1610,
            "completion": 20,
            "prompt_details": {
                "audio": 0,
                "cache_read": 1408
            },
            "completion_details": {
                "audio": 0,
                "reasoning": 0
            }
        },
        "input_messages": [
            {
                "message": {
                    "role": "system",
                    "content": "You are AgentMatcher..."
                }
            },
            {
                "message": {
                    "role": "user",
                    "content": "Hello"
                }
            }
        ],
        "output_messages": [
            {
                "message": {
                    "role": "assistant",
                    "content": "{\n  \"confidence\": 0.95,\n  \"selectedAgent\": \"agent\"\n}"
                }
            }
        ]
    },
    "sessionId": "6c7916d6-ccbf-440a-a805-8a1bb3c4dc76",
    "openinference": {
        "span": {
            "kind": "LLM"
        }
    }
}