Surface Emerging Issues with Signals

Explore Signals

5 minutes

Caution

Generating Signals requires multiple prompts to an LLM, which can result in significant usage costs if every workshop participant runs the process individually. For this reason, please observe as the workshop instructor completes the steps in this section, rather than performing them yourself.

Generate Signals for your agent stream and let the platform show you what’s trending wrong.

Exercise Review Signals
1

Generate Signals

In the Splunk Agent Observability console (https://console.multitenant.galileocloud.io, workshop org), open your project / default agent stream. Click on the AI Assistant icon.

Change the LLM used by the AI Assistant to be gpt-5 (Azure), then click the Generate Signals button.

Generate Signals

It will take a few moments to analyze the traces in this agent stream and generate signals.

2

Review Signals

We can see that several signals have been generated for our agent stream (the specific signals will vary from one agent stream to the next):

Signals overview

3

Open a signal for context

Select a signal and read its actionable context: what the pattern is, why it’s happening, and the recommended next step.

For example, let’s click on the signal named Database Metadata Leakage Risk:

Signal detail

This signals explains how the get_patient_info tool output includes raw SQL queries, database source, and table names, which could be leaked to end users if the LLM echoes this metadata.

It provides a suggestion action to remediate the issue, which is to strip database metadata (SQL queries, table names, source info) from tool outputs before returning them to the LLM.

4

Jump to the underlying traces

From the signal, we can pivot into the specific traces that make up the pattern by clicking on the View Affected Spans in Table button:

Signal to traces

This allows us to go from “there’s a recurring problem” to “here are the exact requests behind it” in a couple of clicks, exactly the targeted remediation Signals are designed to enable.

Why this matters

Without Signals, this kind of analysis means an engineer manually combing through traces after an incident, often for weeks. Signals compress that into minutes, and catch issues before they become incidents at all.

How do Signals complement the evaluators you enabled in the previous chapter?

Click here to see the answer

Evaluators score known quality dimensions you choose to measure (e.g., Context Adherence). Signals automatically surface unknown recurring failure patterns (planning loops, tool errors, routing failures) that you didn’t write an evaluator for. Together they cover both the problems you anticipated and the ones you didn’t.

We have the option to create a new evaluator from a specific signal, which allows us to track when the underlying issue happens again in the future so we can take appropriate action.