Instrument the Application

Generate Traffic

4 minutes

With the callback attached, run the instrumented app and send a few requests. Every turn now produces a trace in Splunk Agent Observability. You’ll explore those traces in detail in the next chapter.

Exercise Run the app and generate traces
1

Deploy the healthcare assistant app

If you’re using the standalone version of Splunk Agent Observability for the workshop, run the following command to deploy the healthcare assistant app:

bash 
cd ~/workshop/healthcare-assistant/2-app-with-instrumentation
kubectl apply -f k8s.yaml

Alternatively, if you’re using Splunk Agent Observability within Splunk Observability Cloud for this workshop, please use the following command instead:

bash 
cd ~/workshop/healthcare-assistant/2-app-with-instrumentation
kubectl apply -f k8s-o11y.yaml

Ensure that the new application pod is running:

bash 
kubectl get pods -l app=healthcare-assistant

To access the application, click on the Shop URL link in the Connection Information section of Splunk Show event:

Connection Information

2

Send a few requests

Exercise both tool paths so you generate a RAG trace and a text-to-SQL trace:

What is the dosage and common side effects of Lisinopril?

Can you look up information for patient P001?

Note that each prompt returns the same answer that it did before instrumentation; the callback doesn’t change the app’s behavior, it just records it.

Tip

If you’d like to explore other medications that you can ask about, you can look at the following document:

bash 
cat ~/workshop/healthcare-assistant/docs/qa.csv
3

Trigger a hallucination

Next, click on the Log Hallucination button on the left-hand side of the application. This will send the same question as before:

What is the dosage and common side effects of Lisinopril?

But this time, the healthcare assistant responds to say that the common dosage is 100mg daily, which is much higher than the actual recommended dosage of just 10-40mg daily.

This is an inaccurate, and potentially dangerous, response that we’ll definitely want to know about!

4

Review the application logs

Use the following command to view the application logs:

bash 
kubectl logs -l app=healthcare-assistant

If everything is working as expected, you should see the following in the logs:

text 
Collecting usage statistics. To deactivate, set browser.gatherUsageStats to false.

2026-07-07 17:52:39.433 Uvicorn server started on :::8501

  You can now view your Streamlit app in your browser.

  Local URL: http://localhost:8501
  Network URL: http://10.42.0.18:8501
  External URL: http://98.92.157.71:8501

Tip

The app already enables SDK console logging at INFO near the top of ~/workshop/healthcare-assistant/2-app-with-instrumentation/agent.py. To see even more detail about what the SDK is doing, temporarily raise it to DEBUG:

python 
from splunk_ao.utils.log_config import enable_console_logging
import logging

enable_console_logging(level=logging.DEBUG)

Then rebuild the Docker image:

bash 
cd ~/workshop/healthcare-assistant
docker build -f 2-app-with-instrumentation/Dockerfile -t localhost:9999/healthcare-assistant:app-with-instrumentation-v2 .
docker push localhost:9999/healthcare-assistant:app-with-instrumentation-v2

Update the ~/workshop/healthcare-assistant/2-app-with-instrumentation/k8s.yaml file to reference the local image instead:

text 
image: localhost:9999/healthcare-assistant:app-with-instrumentation-v2

And redeploy the application:

bash 
cd ~/workshop/healthcare-assistant/2-app-with-instrumentation
kubectl apply -f k8s.yaml

Use the following command to view the application logs:

bash 
kubectl logs -l app=healthcare-assistant

What you just unlocked

In one small change you went from a black box to full capture: every prompt, response, tool call, retrieval, token count, and latency for every turn is now recorded. In the next chapter you’ll put that to work and investigate exactly what the agent did.

You sent three messages. Roughly how many traces did you create, and what determines that?

Click here to see the answer
Three traces, one per user turn. A fresh callback is attached per call, and the whole LangGraph turn runs as a single root run, so each message becomes one trace containing nested LLM and tool spans. You’ll review the traces in more detail in the next section.