Historical Anomaly Detector

Objective

Create a historical baseline anomaly detector using the detector wizard and examine the generated alert message.

Step 1 – Create the Detector

Navigate to:

Alerts & Detectors → Create Detector → Custom Detector

ADD YOUR INITIALS before the proposed detector name.

Configure the following in the alert signal:

• Signal (A): system.cpu.utilization

• Filter: deployment.environment : astronomy-shop

Proceed to Alert Condition, choose Historical Anomaly and then

• Cycle length: 1d
• Alert when: Too high
• Trigger Sensitivity: High

Show advanced settings and review

Proceed to Alert Message.

Step 2 – Examine the Default Alert Message

Under Message Preview, click Customize and review the generated message:

{{#if anomalous}}
	Rule "{{{ruleName}}}" in detector "{{{detectorName}}}" triggered at {{timestamp}}.
{{else}}
	Rule "{{{ruleName}}}" in detector "{{{detectorName}}}" cleared at {{timestamp}}.
{{/if}}

{{#if anomalous}}
Triggering condition: {{{readableRule}}}
{{/if}}

Mean value of signal in the last {{event_annotations.current_window}}: {{inputs.summary.value}}
{{#if anomalous}}Trigger threshold: {{inputs.fire_top.value}}
{{else}}Clear threshold: {{inputs.clear_top.value}}.
{{/if}}

{{#notEmpty dimensions}}
Signal details:
{{{dimensions}}}
{{/notEmpty}}

{{#if anomalous}}
{{#if runbookUrl}}Runbook: {{{runbookUrl}}}{{/if}}
{{#if tip}}Tip: {{{tip}}}{{/if}}
{{/if}}

{{#if detectorTags}}Tags: {{detectorTags}}{{/if}}

{{#if detectorTeams}}
Teams:{{#each detectorTeams}} {{name}}{{#unless @last}},{{/unless}}{{/each}}.
{{/if}}

What This Message Is Doing

This message uses conditional blocks to render different content depending on whether the detector is triggering or clearing.

• {{#if anomalous}} renders content only when the detector is firing.
• The {{else}} branch renders when the detector clears.

This allows one template to handle both trigger and clear notifications.

Important Variables Available in Alert Messages

The following variables are automatically available:

• {{ruleName}} – Name of the alert rule
• {{detectorName}} – Name of the detector
• {{timestamp}} – Time of the event
• {{readableRule}} – Human-readable firing condition
• {{event_annotations.current_window}} – Evaluation window duration
• {{inputs.summary.value}} – Aggregated metric value for the evaluation window
• {{inputs.fire_top.value}} – Historical anomaly trigger threshold
• {{inputs.clear_top.value}} – Historical anomaly clear threshold
• {{dimensions}} – Dimension key/value pairs (host, environment, etc.)
• {{runbookUrl}} – Configured runbook link (if set)
• {{tip}} – Configured tip (if set)
• {{detectorTags}} – Tags assigned to the detector
• {{detectorTeams}} – Assigned teams

Any stream that is published in SignalFlow becomes available as: {{inputs.<stream_name>.value}}

Click Done Editing to close the custom message.

Proceed to Alert Recipients and do not select anything, we don’t actually want to send notifications for this scenario

When prompted about Missing Alert Notification Policy, choose Save

Edit in SignalFlow

Objective

Refactor the wizard-generated detector to:

• Separate threshold calculation from alert logic
• Compose multiple conditions in a single detect() statement
• Introduce a static operational guardrail
• Surface dynamic anomaly thresholds for reuse in alert messages

Edit in SignalFlow

From the detector action menu in the upper right hand corner (⋯), select Edit in SignalFlow

You should still be in the Detector UI for the detector you just saved, if not:

Navigate to:

Alerts & Detectors → Detectors

Locate your detector and open it, then Edit in SignalFlow.

Generated SignalFlow

Choose the SignalFlow tab and review the generated SignalFlow for the historical anomaly detector.

from signalfx.detectors.against_periods import against_periods

A = data(
  'system.cpu.utilization',
  filter=filter('deployment.environment', 'astronomy-shop')
).publish(label='A')

against_periods.detector_mean_std(
  stream=A,
  window_to_compare='10m',
  space_between_windows='1d',
  num_windows=4,
  fire_num_stddev=2.5,
  clear_num_stddev=2,
  discard_historical_outliers=True,
  orientation='above',
  auto_resolve_after='1h'
).publish('XYZ_AdvancedDetector')

Why We Are Refactoring

The wizard generated the detector using:

against_periods.detector_mean_std()

This helper function:

• Calculates historical baseline thresholds
• Applies fire and clear logic
• Applies orientation (above / below)
• Handles auto-resolve timing
• Publishes the detector in a single call

While convenient, this structure bundles threshold generation and alert behavior together.
To build multi-condition logic, we must separate threshold calculation from detection logic.

Step 1 – Replace the Import

Remove:

from signalfx.detectors.against_periods import against_periods

Replace with:

#import from SignalFx Library
from signalfx.detectors.against_periods import streams
from signalfx.detectors.against_periods import conditions

• streams generates reusable threshold streams.
• conditions enables logical composition in detect().

Step 2 – Rename the Signal Stream

Rename the signal from A to CPU for clarity.

Replace:

A = data('system.cpu.utilization', filter=filter('deployment.environment', 'astronomy-shop')).publish(label='A')

With:

#Calculate/filter CPU
CPU = data('system.cpu.utilization', filter=filter('deployment.environment', 'astronomy-shop')).publish(label='CPU')

Step 3 – Convert the Helper Call into Threshold Streams

The wizard-generated helper already contains the anomaly tuning we want to preserve:

• window_to_compare='10m'
• space_between_windows='1d'
• num_windows=4
• fire_num_stddev=2.5
• clear_num_stddev=2
• discard_historical_outliers=True

We will keep these values.

Locate:

against_periods.detector_mean_std(

Replace only the function name with:

#Use the streams.mean_std_thresholds function to establish the built in min/max fire and clear threshold conditions
fire_bot, clear_bot, clear_top, fire_top = streams.mean_std_thresholds(

Update the stream argument:

Replace:

stream=A,

With:

CPU,

Remove Helper-Only Alert Parameters

streams.mean_std_thresholds() generates threshold streams only.
It does not implement detector behaviors such as orientation or auto-resolve.

Remove:

orientation='above',
auto_resolve_after='1h'

Remove the Helper Publish

The helper call publishes a detector directly:

).publish('XYZ_AdvancedDetector')

streams.mean_std_thresholds() does not publish a detector.

Remove .publish('XYZ_AdvancedDetector')

Step 4 – Add Multi-Condition Detect Logic

Now that threshold generation and alert logic are separated, you must explicitly define the detection criteria.

First, define the static guardrail as its own stream by appending:

#Define static threshold for CPU as a variable
static_threshold = threshold(90)

This creates a constant threshold stream at 90%.
By defining it as a stream (instead of embedding threshold(90) directly inside detect()), it can be published, visualized, and referenced in alert messages.

Next, define the multi-condition detection logic:

#detect when CPU has exceeded the fire_top thresholds established AND CPU exceeds static threshold (90%) for 15 minutes; publish detector
detect(
  CPU > fire_top and when(CPU > static_threshold, lasting='15m')
).publish('custom_CPU_detector')

This detect statement evaluates two independent conditions:

1. Historical baseline anomaly
   CPU > fire_top
   The 10-minute moving average exceeds the dynamically calculated anomaly threshold.

2. Static operational guardrail with duration
   when(CPU > static_threshold, lasting='15m')
   CPU must remain above 90% for 15 consecutive minutes.

Both conditions must evaluate to true before the detector fires.

You now control exactly how anomaly behavior and operational thresholds interact. This introduces:

• Historical baseline anomaly: CPU > fire_top
• Static operational guardrail: CPU > static_threshold
• Sustained violation requirement: 15 minutes
• Explicit detector publication

Step 5 – Publish Threshold Streams for Preview and Messaging

To surface both thresholds for detector preview and alert messaging:

#publish the fire_top threshold and static_threshold for data visualization
fire_top.publish('CPU_top_threshold')
static_threshold.publish('CPU_static_threshold')

Result

#import from SignalFx Library
from signalfx.detectors.against_periods import streams
from signalfx.detectors.against_periods import conditions

#Calculate/filter CPU
CPU = data('system.cpu.utilization', filter=filter('deployment.environment', 'astronomy-shop')).publish(label='CPU')

#Use the streams.mean_std_thresholds function to establish the built in min/max fire and clear threshold conditions
fire_bot, clear_bot, clear_top, fire_top = streams.mean_std_thresholds(
  CPU,
  window_to_compare='10m',
  space_between_windows='1d',
  num_windows=4,
  fire_num_stddev=2.5,
  clear_num_stddev=2,
  discard_historical_outliers=True,
)

#Define static threshold for CPU as a variable
static_threshold = threshold(90)

#detect when CPU has exceeded the fire_top thresholds established AND CPU exceeds static threshold (90%) for 15 minutes; publish detector
detect(
  CPU > fire_top and when(CPU > static_threshold, lasting='15m')
).publish('custom_CPU_detector')

#publish the fire_top threshold and static_threshold for data visualization
fire_top.publish('CPU_top_threshold')
static_threshold.publish('CPU_static_threshold')

static_threshold.publish('CPU_static_threshold')

{{inputs.CPU_static_threshold.value}}

Update Alert Message and Alert Rule

Objective

Customize the alert message to accurately reflect the multi-condition detection logic by:

• Explaining why the wizard-generated message is removed
• Referencing published threshold streams
• Explicitly communicating both the historical anomaly and static guardrail conditions

Step 1 – Save the Detector

Click Save in the upper right corner.

Step 2 – Edit Alert Message

Navigate to the detector’s Alert Rules tab.

Click Edit on the existing Alert Rule.

Select the Alert message tab and click Customize.

Replace the message body with:

{{#if anomalous}}
	Rule "{{{ruleName}}}" in detector "{{{detectorName}}}" triggered at {{timestamp}}.
{{else}}
	Rule "{{{ruleName}}}" in detector "{{{detectorName}}}" cleared at {{timestamp}}.
{{/if}}

{{#if anomalous}}
Triggering condition: {{{readableRule}}}
{{/if}}

Mean value of signal in the last {{event_annotations.current_window}}: {{inputs.CPU.value}}

{{#if anomalous}}
Historical anomaly threshold: {{inputs.CPU_top_threshold.value}}
Static guardrail threshold: {{inputs.CPU_static_threshold.value}}
{{else}}
Clear threshold: {{inputs.clear_top.value}}
{{/if}}

{{#notEmpty dimensions}}
Signal details:
{{{dimensions}}}
{{/notEmpty}}

{{#if anomalous}}
{{#if runbookUrl}}Runbook: {{{runbookUrl}}}{{/if}}
{{#if tip}}Tip: {{{tip}}}{{/if}}
{{/if}}

{{#if detectorTags}}Tags: {{detectorTags}}{{/if}}

{{#if detectorTeams}}
Teams:{{#each detectorTeams}} {{name}}{{#unless @last}},{{/unless}}{{/each}}.
{{/if}}

You are now explicitly referencing:

• {{inputs.CPU_top_threshold.value}} → dynamic anomaly threshold
• {{inputs.CPU_static_threshold.value}} → static 90% guardrail

These variables are available because both streams were published in SignalFlow.

Click Done Editing to save the custom message.

Click ProceedAlert Recipients..

Step 3 – Update the Alert Rule Description

In the Activate… step, update the Description to:

The 10m moving average of system.cpu.utilization (assumed to be cyclical over 1d periods) is more than 2.5 standard deviation(s) above its historical norm and has exceeded 90% for 15 minutes.

Click Update Alert Rule to save changes.

Wrap Up

You have now:

• Used the wizard to configure historical baseline parameters
• Refactored the generated SignalFlow to expose threshold streams
• Added multi-condition alert logic (historical anomaly + static guardrail)
• Published both anomaly and static thresholds for reuse
• Updated the alert message and description to clearly communicate the detection logic