Review Evaluator Results

Looking at the agent trace, we can see that evaluators are being computed for our traces:

In Splunk Agent Observability, return to the default agent stream and review the recent traces. Each trace now carries evaluator results alongside its spans.

We can see that one of the traces evaluated to False for both Context Adherence and Correctness. This is the trace that we sent earlier using the Log Hallucination button.

Let’s click on this trace and take a closer look.

Click on the LLM Response span, and notice there are two new evaluator categories on the right-hand side of the screen: Output Quality, which includes the Correctness evaluator, and RAG Quality, which includes the Context Adherence evaluator.

If we hover over false beside the Context Adherence evaluator, we can see the rationale for why this span received this score.

In this case, it explains that the assistant gave a dosage of 100 mg daily and side effects of rashes, itching, and swelling, which directly contradicts the context and adds unsupported information.

This type of finding turns “a patient complained” into “here is the specific request, the specific span, and the specific evaluator that caught it.”

You may be wondering: what if an evaulator gets the evaluation wrong?

We can provide feedback on any evaluator by clicking the Add feedback button:

And providing the Corrected value and Rationale:

This human feedback helps improve how the evaluator evaluates similar cases over time, which ultimately ensures that the scoring becomes more aligned to our real standards.