3. Dropping Spans

5 minutes

In this section, we will explore how to use the Filter Processor to selectively drop spans based on certain conditions.

Specifically, we will drop traces based on the span name, which is commonly used to filter out unwanted spans such as health checks or internal communication traces. In this case, we will be filtering out spans that contain "/_healthz", typically associated with health check requests and usually are quite “noisy”.

Exercise

Important

Change ALL terminal windows to the 3-dropping-spans directory and run the clear command.

Copy *.yaml from the 2-building-resilience directory into 3-dropping-spans. Your updated directory structure will now look like this:

.
├── agent.yaml
└── gateway.yaml

Next, we will configure the filter processor and the respective pipelines.

3.1 Configuration

Exercise

Switch to your Gateway terminal window and open the gateway.yaml file. Update the processors section with the following configuration:

Add a filter processor:
Configure the gateway to exclude spans with the name /_healthz. The error_mode: ignore directive ensures that any errors encountered during filtering are ignored, allowing the pipeline to continue running smoothly. The traces section defines the filtering rules, specifically targeting spans named /_healthz for exclusion.

  filter/health:                       # Defines a filter processor
    error_mode: ignore                 # Ignore errors
    traces:                            # Filtering rules for traces
      span:                            # Exclude spans named "/_healthz"
       - 'name == "/_healthz"'

Add the filter processor to the traces pipeline:
Include the filter/health processor in the traces pipeline. For optimal performance, place the filter as early as possible—right after the memory_limiter and before the batch processor. Here’s how the configuration should look:

traces:
  receivers:
    - otlp
  processors:
    - memory_limiter
    - filter/health             # Filters data based on rules
    - resource/add_mode
    - batch
  exporters:
    - debug
    - file/traces

This setup ensures that health check related spans (/_healthz) are filtered out early in the pipeline, reducing unnecessary noise in your telemetry data.

3.2 Test Filter Processor

To test your configuration, you’ll need to generate some trace data that includes a span named "/_healthz".

Exercise

Start the Gateway: In your Gateway terminal window start the Gateway.

../otelcol --config ./gateway.yaml

Start the Agent: In your Agent terminal window start the Agent.

../otelcol --config ./agent.yaml

Start the Loadgen: In the Loadgen terminal window, execute the following command to start the load generator with health check spans enabled:

../loadgen -health -count 5

The debug output in the Agent terminal will show _healthz spans:

InstrumentationScope healthz 1.0.0
Span #0
   Trace ID       : 0cce8759b5921c8f40b346b2f6e2f4b6
   Parent ID      :
   ID             : bc32bd0e4ddcb174
   Name           : /_healthz
   Kind           : Server
   Start time     : 2025-07-11 08:47:50.938703979 +0000 UTC
   End time       : 2025-07-11 08:47:51.938704091 +0000 UTC
   Status code    : Ok
   Status message : Success

They will not be present in the Gateway debug as they are dropped by the filter processor that was configured earlier.

Verify agent.out: Using jq, in the Test terminal, confirm the name of the spans received by the Agent:

jq -c '.resourceSpans[].scopeSpans[].spans[] | "Span \(input_line_number) found with name \(.name)"' ./agent.out

"Span 1 found with name /movie-validator"
"Span 2 found with name /_healthz"
"Span 3 found with name /movie-validator"
"Span 4 found with name /_healthz"
"Span 5 found with name /movie-validator"
"Span 6 found with name /_healthz"
"Span 7 found with name /movie-validator"
"Span 8 found with name /_healthz"
"Span 9 found with name /movie-validator"
"Span 10 found with name /_healthz"

Check the Gateway Debug output: Using jq confirm the name of the spans received by the Gateway:

jq -c '.resourceSpans[].scopeSpans[].spans[] | "Span \(input_line_number) found with name \(.name)"' ./gateway-traces.out

The gateway-metrics.out file will not contain any spans named /_healthz.

"Span 1 found with name /movie-validator"
"Span 2 found with name /movie-validator"
"Span 3 found with name /movie-validator"
"Span 4 found with name /movie-validator"
"Span 5 found with name /movie-validator"

Tip

To ensure optimal performance with the Filter processor, thoroughly understand your incoming data format and rigorously test your configuration. Use the most specific filtering criteria possible to minimize the risk of inadvertently dropping important data.

This configuration can be extended to filter spans based on various attributes, tags, or custom criteria, enhancing the OpenTelemetry Collector’s flexibility and efficiency for your specific observability requirements.

Important

Stop the Agent and the Gateway processes by pressing Ctrl-C in their respective terminals.

3. Dropping Spans

Subsections of 3. Dropping Spans

3.1 Configuration

3.2 Test Filter Processor