Trace and Investigate Agent Behavior
Explore a Trace
Open the Splunk Agent Observability and work through a real trace from the traffic you just generated.
Find your Instance ID
Execute the following command using the terminal connected to your EC2 instance:
echo $INSTANCEMake a note of the result, which will be something like shw-51ea. This will be
the name of your Agent Stream below.
Open your project and agent stream
1. In your browser, go to the Splunk Agent Observability console at https://console.multitenant.sao.splunkcloud.com and workshop org
2. Open the Splunk Agent Observability Workshop project.
3. Select the agent stream that matches the Instance ID you found above (such as shw-51ea).
Scan the trace list
Review the list of recent traces, one per message you sent. Note the high-level signals available at a glance, such as number of input tokens, output tokens, and spans.
Open a trace and read the span tree
Open the trace for the Lisinopril dosage question. You should see a single trace containing a
nested LLM span for the chatbot node and a tool span for search_medicine_qa
(retrieval). Expand the tree to follow the agent’s path end to end.
Inspect a span
Select the chat gpt-4.1-mini span and confirm it captured the system and user messages,
the Output, Token Counts, Latency, and Agent Cost. This
is the detail that lets you explain why the agent answered the way it did.
View the Trace Graph
Next, click on the Trace graph tab, which provides a visual, step-by-step view of how
this specific interaction executed across the system.
