Instrument the Application

Generate Traffic

4 minutes

With the callback attached, run the instrumented app and send a few requests. Every turn now produces a trace in Splunk Agent Observability. You’ll explore those traces in detail in the next chapter.

Exercise Run the app and generate traces
1

Deploy the healthcare assistant app

Run the following command to deploy the healthcare assistant app:

bash 
cd ~/workshop/healthcare-assistant/2-app-with-instrumentation
kubectl apply -f k8s.yaml

Ensure that the new application pod is running:

bash 
kubectl get pods -l app=healthcare-assistant

Using the IP address of your EC2 instance and port 81, open the healthcare assistant app using your browser. For example:

text 
  External URL: http://98.86.181.9:81
2

Send a few requests

Exercise both tool paths so you generate a RAG trace and a text-to-SQL trace:

What is the dosage and common side effects of Lisinopril?

Can you look up information for patient P001?

Note that each prompt returns the same answer that it did before instrumentation; the callback doesn’t change the app’s behavior, it just records it.

Tip

If you’d like to explore other medications that you can ask about, you can look at the following document:

bash 
cat ~/workshop/healthcare-assistant/docs/qa.csv
3

Trigger a hallucination

Next, click on the Log Hallucination button on the left-hand side of the application. This will send the same question as before:

What is the dosage and common side effects of Lisinopril?

But this time, the healthcare assistant responds to say that the common dosage is 100mg daily, which is much higher than the actual recommended dosage of just 10-40mg daily.

This is an inaccurate, and potentially dangerous, response that we’ll definitely want to know about!

4

Review the application logs

Use the following command to view the application logs:

bash 
kubectl logs -l app=healthcare-assistant

If everything is working as expected, you should see the following in the logs:

text 
Collecting usage statistics. To deactivate, set browser.gatherUsageStats to false.

2026-07-07 17:52:39.433 Uvicorn server started on :::8501

  You can now view your Streamlit app in your browser.

  Local URL: http://localhost:8501
  Network URL: http://10.42.0.18:8501
  External URL: http://98.92.157.71:8501

Tip

To see exactly what the SDK is doing, you can temporarily add the following near the top of ~/workshop/healthcare-assistant/2-app-with-instrumentation/agent.py:

python 
from galileo.utils.log_config import enable_console_logging

enable_console_logging()

Then rebuild the Docker image:

bash 
cd ~/workshop/healthcare-assistant
docker build -f 2-app-with-instrumentation/Dockerfile -t localhost:9999/healthcare-assistant:app-with-instrumentation .
docker push localhost:9999/healthcare-assistant:app-with-instrumentation

Update the ~/workshop/healthcare-assistant/2-app-with-instrumentation/k8s.yaml file to reference the local image instead:

text 
image: localhost:9999/healthcare-assistant:app-with-instrumentation

And redeploy the application:

bash 
cd ~/workshop/healthcare-assistant/2-app-with-instrumentation
kubectl apply -f k8s.yaml

Use the following command to view the application logs:

bash 
kubectl logs -l app=healthcare-assistant

What you just unlocked

In one small change you went from a black box to full capture: every prompt, response, tool call, retrieval, token count, and latency for every turn is now recorded. In the next chapter you’ll put that to work and investigate exactly what the agent did.

You sent three messages. Roughly how many traces did you create, and what determines that?

Click here to see the answer
Three traces, one per user turn. A fresh callback is attached per call, and the whole LangGraph turn runs as a single root run, so each message becomes one trace containing nested LLM and tool spans. You’ll review the traces in more detail in the next section.