Advanced Collector Configuration

90 minutes Authors Robert Castley, Charity Anderson, Pieter Hagen & Geoff Higginbottom

The goal of this workshop is to help you gain confidence in creating and modifying OpenTelemetry Collector configuration files. You’ll start with a minimal agent.yaml file and progressively build it out to handle several advanced, real-world scenarios.

A key focus of this workshop is learning how to configure the OpenTelemetry Collector to store telemetry data locally, rather than sending it to a third-party vendor backend. This approach not only simplifies debugging and troubleshooting but is also ideal for testing and development environments where you want to avoid sending data to production systems.

To make the most of this workshop, you should have:

A basic understanding of the OpenTelemetry Collector and its configuration file structure.
Proficiency in editing YAML files.

Everything in this workshop is designed to run locally, ensuring a hands-on and accessible learning experience. Let’s dive in and start building!

Workshop Overview

During this workshop, we will cover the following topics:

Setting up the agent locally: Add metadata, and introduce the debug and file exporters.
Configuring a gateway: Route traffic from the agent to the gateway.
Configuring the FileLog receiver: Collect log data from various log files.
Enhancing agent resilience: Basic configurations for fault tolerance.
Configuring processors:
- Filter out noise by dropping specific spans (e.g., health checks).
- Remove unnecessary tags, and handle sensitive data.
- Transform data using OTTL (OpenTelemetry Transformation Language) in the pipeline before exporting.
Configuring Connectors:
- Route data to different endpoints based on the values received.
- Convert log and span data to metrics.

By the end of this workshop, you’ll be familiar with configuring the OpenTelemetry Collector for a variety of real-world use cases.

Pre-requisites

90 minutes

Prerequisites

Proficiency in editing YAML files using vi, vim, nano, or your preferred text editor.
Supported Environments:
- Splunk Workshop Instance (preferred)
- Apple Mac (Apple Silicon). Installation of jq is required - https://jqlang.org/download/

Exercise

Create a workshop directory: In your environment create a new directory (e.g., advanced-otel-workshop). We will refer to this directory as [WORKSHOP] for the remainder of the workshop.

Download workshop binaries: Change into your [WORKSHOP] directory and download the OpenTelemetry Collector and Load Generator binaries:

curl -L https://github.com/signalfx/splunk-otel-collector/releases/download/v0.120.0/otelcol_linux_amd64 -o otelcol && \
curl -L https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/loadgen/build/loadgen-linux-amd64 -o loadgen

Update file permissions: Once downloaded, update the file permissions to make both executable:

chmod +x otelcol loadgen && \
./otelcol -v && \
./loadgen --help

curl -L https://github.com/signalfx/splunk-otel-collector/releases/download/v0.120.0/otelcol_darwin_arm64 -o otelcol && \
curl -L https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/loadgen/build/loadgen-darwin-arm64 -o loadgen

macOS Users

Before running the binaries on macOS, you need to remove the quarantine attribute that macOS applies to downloaded files. This step ensures they can execute without restrictions.

Run the following command in your terminal:

xattr -dr com.apple.quarantine otelcol && \
xattr -dr com.apple.quarantine loadgen

Update file permissions: Once downloaded, update the file permissions to make both executable:

chmod +x otelcol loadgen && \
./otelcol -v && \
./loadgen --help

[WORKSHOP]
├── otelcol      # OpenTelemetry Collector binary
└── loadgen      # Load Generator binary

1. Agent Configuration

10 minutes

Tip

During this workshop, you will be using up to four terminal windows simultaneously. To stay organized, consider customizing each terminal or shell with unique names and colors. This will help you quickly identify and switch between them as needed.

We will refer to these terminals as: Agent, Gateway, Spans and Logs.

Exercise

In the Agent terminal window, change into the [WORKSHOP] directory and create a new subdirectory named 1-agent.
```
mkdir 1-agent && \
cd 1-agent
```
Create a file named agent.yaml. This file will define the basic structure of an OpenTelemetry Collector configuration.

Copy and paste the following initial configuration into agent.yaml:

# Extensions
extensions:
  health_check:                        # Health Check Extension
    endpoint: 0.0.0.0:13133            # Health Check Endpoint

# Receivers
receivers:
  hostmetrics:                         # Host Metrics Receiver
    collection_interval: 3600s         # Collection Interval (1hr)
    scrapers:
      cpu:                             # CPU Scraper
  otlp:                                # OTLP Receiver
    protocols:
      http:                            # Configure HTTP protocol
        endpoint: "0.0.0.0:4318"       # Endpoint to bind to

# Exporters
exporters:
  debug:                               # Debug Exporter
    verbosity: detailed                # Detailed verbosity level

# Processors
processors:
  memory_limiter:                      # Limits memory usage
    check_interval: 2s                 # Check interval
    limit_mib: 512                     # Memory limit in MiB
  resourcedetection:                   # Resource Detection Processor
    detectors: [system]                # Detect system resources
    override: true                     # Overwrites existing attributes
  resource/add_mode:                   # Resource Processor
    attributes:
    - action: insert                   # Action to perform
      key: otelcol.service.mode        # Key name
      value: "agent"                   # Key value

# Connectors
connectors:

# Service Section - Enabled Pipelines
service:
  extensions:
  - health_check                       # Health Check Extension
  pipelines:
    traces:
      receivers:
      - otlp                           # OTLP Receiver
      processors:
      - memory_limiter                 # Memory Limiter processor
      - resourcedetection              # Add system attributes to the data
      - resource/add_mode              # Add collector mode metadata
      exporters:
      - debug                          # Debug Exporter
    metrics:
      receivers:
      - otlp
      processors:
      - memory_limiter
      - resourcedetection
      - resource/add_mode
      exporters:
      - debug
    logs:
      receivers:
      - otlp
      processors:
      - memory_limiter
      - resourcedetection
      - resource/add_mode
      exporters:
      - debug

Your directory structure should now look like this:

.
└── agent.yaml  # OpenTelemetry Collector configuration file

1.1 Validation & Load Generation

In this workshop, we’ll use https://otelbin.io to quickly validate YAML syntax and ensure your OpenTelemetry configurations are accurate. This step helps avoid errors before running tests during the session.

Exercise

Here’s how to validate your configuration:

Open https://otelbin.io and replace the existing configuration by pasting your YAML into the left pane.
Info
If are on a Mac and not using a Splunk Workshop instance, you can quickly copy the contents of the agent.yaml file to your clipboard by running the following command:
```
cat agent.yaml | pbcopy
```
At the top of the page, make sure Splunk OpenTelemetry Collector is selected as the validation target. If you don’t select this option, then you will see warnings in the UI stating Receiver "hostmetrics" is unused. (Line 8).
Once validated, refer to the image representation below to confirm your pipelines are set up correctly.

In most cases, we’ll display only the key pipeline. However, if all three pipelines (Traces, Metrics, and Logs) share the same structure, we’ll note this instead of showing each one individually.

%%{init:{"fontFamily":"monospace"}}%%
graph LR
    %% Nodes
      REC1(&nbsp;&nbsp;otlp&nbsp;&nbsp;<br>fa:fa-download):::receiver
      PRO1(memory_limiter<br>fa:fa-microchip):::processor
      PRO2(resourcedetection<br>fa:fa-microchip):::processor
      PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
      EXP1(&ensp;debug&ensp;<br>fa:fa-upload):::exporter
    %% Links
    subID1:::sub-traces
    subgraph " "
      subgraph subID1[**Traces/Metrics/Logs**]
      direction LR
      REC1 --> PRO1
      PRO1 --> PRO2
      PRO2 --> PRO3
      PRO3 --> EXP1
      end
    end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-traces stroke:#fff,stroke-width:1px, color:#fff,stroke-dasharray: 3 3;

Load Generation Tool

For the is workshop we have specifically developed a loadgen tool. loadgen is a flexible load generator for simulating traces and logging activities. It supports base, health, and security traces by default, along with optional logging of random quotes to a file, either in plain text or JSON format.

The output generated by loadgen mimic those produced by an OpenTelemetry instrumentation library, allowing us to test the Collector’s processing logic and offers a simple yet powerful way to mimic real-world scenarios.

1.2 Test Agent Configuration

You’re ready to start the OpenTelemetry Collector with the newly created agent.yaml. This exercise sets the foundation for understanding how data flows through the OpenTelemetry Collector.

Exercise

Start the Agent: In the Agent terminal window run the following command:

../otelcol --config=agent.yaml

Verify debug output: If everything is configured correctly, the first and last lines of the output will look like:

2025/01/13T12:43:51 settings.go:478: Set config to [agent.yaml]
<snip to the end>
2025-01-13T12:43:51.747+0100 info service@v0.120.0/service.go:261 Everything is ready. Begin running and processing data.

Send Test Span: Instead of instrumenting an application, we’ll simulate sending trace data to the OpenTelemetry Collector using the loadgen tool.

In the Spans terminal window, change into the 1-agent directory and run the following command to send a single span:

../loadgen -count 1

Sending traces. Use Ctrl-C to stop.
Response: {"partialSuccess":{}}

Base trace sent with traceId: 1aacb1db8a6d510f10e52f154a7fdb90 and spanId: 7837a3a2d3635d9f

{"partialSuccess":{}}: Indicates 100% success, as the partialSuccess field is empty. In case of a partial failure, this field will include details about any failed parts.

Verify Debug Output:

In the Agent terminal window check the collector’s debug output:

2025-03-06T10:11:35.174Z        info    Traces  {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 1}
2025-03-06T10:11:35.174Z        info    ResourceSpans #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.6.1
Resource attributes:
     -> service.name: Str(cinema-service)
     -> deployment.environment: Str(production)
     -> host.name: Str(workshop-instance)
     -> os.type: Str(linux)
     -> otelcol.service.mode: Str(agent)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope cinema.library 1.0.0
InstrumentationScope attributes:
     -> fintest.scope.attribute: Str(Starwars, LOTR)
Span #0
    Trace ID       : 0ef4daa44a259a7199a948231bc383c0
    Parent ID      :
    ID             : e8fdd442c36cbfb1
    Name           : /movie-validator
    Kind           : Server
    Start time     : 2025-03-06 10:11:35.163557 +0000 UTC
    End time       : 2025-03-06 10:11:36.163557 +0000 UTC
    Status code    : Ok
    Status message : Success
Attributes:
     -> user.name: Str(George Lucas)
     -> user.phone_number: Str(+1555-867-5309)
     -> user.email: Str(george@deathstar.email)
     -> user.password: Str(LOTR>StarWars1-2-3)
     -> user.visa: Str(4111 1111 1111 1111)
     -> user.amex: Str(3782 822463 10005)
     -> user.mastercard: Str(5555 5555 5555 4444)
     -> payment.amount: Double(86.48)
        {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces"}

Important

Stop the agent in the Agent terminal window using Ctrl-C.

1.3 File Exporter

To capture more than just debug output on the screen, we also want to generate output during the export phase of the pipeline. For this, we’ll add a File Exporter to write OTLP data to files for comparison.

The difference between the OpenTelemetry debug exporter and the file exporter lies in their purpose and output destination:

Feature	Debug Exporter	File Exporter
Output Location	Console/Log	File on disk
Purpose	Real-time debugging	Persistent offline analysis
Best for	Quick inspection during testing	Temporary storage and sharing
Production Use	No	Rare, but possible
Persistence	No	Yes

In summary, the Debug Exporter is great for real-time, in-development troubleshooting, while the File Exporter is better suited for storing telemetry data locally for later use.

Exercise

In the Agent terminal window ensure the collector is not running then edit the agent.yaml and configure the File Exporter:

Configuring a file exporter: The File Exporter writes telemetry data to files on disk.

  file:                                # File Exporter
    path: "./agent.out"                # Save path (OTLP/JSON)
    append: false                      # Overwrite the file each time

Update the Pipelines Section: Add the file exporter to the traces pipeline only:

  pipelines:
    traces:
      receivers:
      - otlp                           # OTLP Receiver
      processors:
      - memory_limiter                 # Memory Limiter processor
      - resourcedetection              # Add system attributes to the data
      - resource/add_mode              # Add collector mode metadata
      exporters:
      - debug                          # Debug Exporter
      - file                           # File Exporter
    metrics:
      receivers:
      - otlp
      processors:
      - memory_limiter
      - resourcedetection
      - resource/add_mode
      exporters:
      - debug
    logs:
      receivers:
      - otlp
      processors:
      - memory_limiter
      - resourcedetection
      - resource/add_mode
      exporters:
      - debug

Validate the agent configuration using https://otelbin.io:

%%{init:{"fontFamily":"monospace"}}%%
graph LR
    %% Nodes
      REC1(&nbsp;&nbsp;otlp&nbsp;&nbsp;<br>fa:fa-download):::receiver
      PRO1(memory_limiter<br>fa:fa-microchip):::processor
      PRO2(resourcedetection<br>fa:fa-microchip):::processor
      PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
      EXP1(&ensp;debug&ensp;<br>fa:fa-upload):::exporter
      EXP2(&ensp;file&ensp;<br>fa:fa-upload):::exporter
    %% Links
    subID1:::sub-traces
    subgraph " "
      subgraph subID1[**Traces**]
      direction LR
      REC1 --> PRO1
      PRO1 --> PRO2
      PRO2 --> PRO3
      PRO3 --> EXP1
      PRO3 --> EXP2
      end
    end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3;

1.3.1 Test File Exporter

Exercise

Restart your agent: Find your Agent terminal window, and (re)start the agent using the modified configuration:

../otelcol --config=agent.yaml

2025-01-13T12:43:51.747+0100 info service@v0.120.0/service.go:261 Everything is ready. Begin running and processing data.

Send a Trace: From the Spans terminal window send another span and verify you get the same output on the console as we saw previously:

../loadgen -count 1

Verify that the agent.out file is written: Check that a file named agent.out is written in the current directory.

.
├── agent.out    # OTLP/Json output created by the File Exporter
└── agent.yaml

Verify the span format:

Verify the format used by the File Exporter to write the span to agent.out.
The output will be a single line in OTLP/JSON format.
To view the contents of agent.out, you can use the cat ./agent.out command. For a more readable formatted view, pipe the output to jq like this: cat ./agent.out | jq:

{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"cinema-service"}},{"key":"deployment.environment","value":{"stringValue":"production"}},{"key":"host.name","value":{"stringValue":"workshop-instance"}},{"key":"os.type","value":{"stringValue":"linux"}},{"key":"otelcol.service.mode","value":{"stringValue":"agent"}}]},"scopeSpans":[{"scope":{"name":"cinema.library","version":"1.0.0","attributes":[{"key":"fintest.scope.attribute","value":{"stringValue":"Starwars, LOTR"}}]},"spans":[{"traceId":"d824a28db5aa5f5a3011f19c452e5af0","spanId":"ab4cde146f77eacf","parentSpanId":"","name":"/movie-validator","kind":2,"startTimeUnixNano":"1741256991405300000","endTimeUnixNano":"1741256992405300000","attributes":[{"key":"user.name","value":{"stringValue":"George Lucas"}},{"key":"user.phone_number","value":{"stringValue":"+1555-867-5309"}},{"key":"user.email","value":{"stringValue":"george@deathstar.email"}},{"key":"user.password","value":{"stringValue":"LOTR\u003eStarWars1-2-3"}},{"key":"user.visa","value":{"stringValue":"4111 1111 1111 1111"}},{"key":"user.amex","value":{"stringValue":"3782 822463 10005"}},{"key":"user.mastercard","value":{"stringValue":"5555 5555 5555 4444"}},{"key":"payment.amount","value":{"doubleValue":56.24}}],"status":{"message":"Success","code":1}}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.6.1"}]}

{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "stringValue": "cinema-service"
            }
          },
          {
            "key": "deployment.environment",
            "value": {
              "stringValue": "production"
            }
          },
          {
            "key": "host.name",
            "value": {
              "stringValue": "RCASTLEY-M-YQRY.local"
            }
          },
          {
            "key": "os.type",
            "value": {
              "stringValue": "darwin"
            }
          },
          {
            "key": "otelcol.service.mode",
            "value": {
              "stringValue": "agent"
            }
          }
        ]
      },
      "scopeSpans": [
        {
          "scope": {
            "name": "cinema.library",
            "version": "1.0.0",
            "attributes": [
              {
                "key": "fintest.scope.attribute",
                "value": {
                  "stringValue": "Starwars, LOTR"
                }
              }
            ]
          },
          "spans": [
            {
              "traceId": "d824a28db5aa5f5a3011f19c452e5af0",
              "spanId": "ab4cde146f77eacf",
              "parentSpanId": "",
              "name": "/movie-validator",
              "kind": 2,
              "startTimeUnixNano": "1741256991405300000",
              "endTimeUnixNano": "1741256992405300000",
              "attributes": [
                {
                  "key": "user.name",
                  "value": {
                    "stringValue": "George Lucas"
                  }
                },
                {
                  "key": "user.phone_number",
                  "value": {
                    "stringValue": "+1555-867-5309"
                  }
                },
                {
                  "key": "user.email",
                  "value": {
                    "stringValue": "george@deathstar.email"
                  }
                },
                {
                  "key": "user.password",
                  "value": {
                    "stringValue": "LOTR>StarWars1-2-3"
                  }
                },
                {
                  "key": "user.visa",
                  "value": {
                    "stringValue": "4111 1111 1111 1111"
                  }
                },
                {
                  "key": "user.amex",
                  "value": {
                    "stringValue": "3782 822463 10005"
                  }
                },
                {
                  "key": "user.mastercard",
                  "value": {
                    "stringValue": "5555 5555 5555 4444"
                  }
                },
                {
                  "key": "payment.amount",
                  "value": {
                    "doubleValue": 56.24
                  }
                }
              ],
              "status": {
                "message": "Success",
                "code": 1
              }
            }
          ]
        }
      ],
      "schemaUrl": "https://opentelemetry.io/schemas/1.6.1"
    }
  ]
}

Important

Stop the agent in the Agent terminal window using Ctrl-C.

2. Gateway Configuration

10 minutes

The OpenTelemetry Gateway is designed to receive, process, and export telemetry data. It acts as an intermediary between telemetry sources (e.g. applications, services) and backends (e.g., observability platforms like Prometheus, Jaeger, or Splunk Observability Cloud).

The gateway is useful because it centralizes telemetry data collection, enabling features like data filtering, transformation, and routing to multiple destinations. It also reduces the load on individual services by offloading telemetry processing and ensures consistent data formats across distributed systems. This makes it easier to manage, scale, and analyze telemetry data in complex environments.

Exercise

In the Gateway terminal window, change into the [WORKSHOP] directory and create a new subdirectory named 2-gateway.

Important

Change ALL terminal windows to the [WORKSHOP]/2-gateway directory.

Back in the Gateway terminal window, copy agent.yaml from the 1-agent directory into 2-gateway.
Create a file called gateway.yaml and add the following initial configuration:

###########################         This section holds all the
## Configuration section ##         configurations that can be 
###########################         used in this OpenTelemetry Collector
extensions:                       # List of extensions
  health_check:                   # Health check extension
    endpoint: 0.0.0.0:14133       # Custom port to avoid conflicts

receivers:
  otlp:                           # OTLP receiver
    protocols:
      http:                       # HTTP protocol
        endpoint: "0.0.0.0:5318"  # Custom port to avoid conflicts
        include_metadata: true    # Required for token pass-through

exporters:                        # List of exporters
  debug:                          # Debug exporter
    verbosity: detailed           # Enable detailed debug output
  file/traces:                    # Exporter Type/Name
    path: "./gateway-traces.out"  # Path for OTLP JSON output
    append: false                 # Overwrite the file each time
  file/metrics:                   # Exporter Type/Name
    path: "./gateway-metrics.out" # Path for OTLP JSON output
    append: false                 # Overwrite the file each time
  file/logs:                      # Exporter Type/Name
    path: "./gateway-logs.out"    # Path for OTLP JSON output
    append: false                 # Overwrite the file each time

connectors:

processors:                       # List of processors
  memory_limiter:                 # Limits memory usage
    check_interval: 2s            # Memory check interval
    limit_mib: 512                # Memory limit in MiB
  batch:                          # Batches data before exporting
    metadata_keys:                # Groups data by token
    - X-SF-Token
  resource/add_mode:              # Adds metadata
    attributes:
    - action: upsert              # Inserts or updates a key
      key: otelcol.service.mode   # Key name
      value: "gateway"            # Key value

###########################
### Activation Section  ###
###########################
service:                          # Service configuration
  telemetry:
    metrics:
      level: none                 # Disable metrics
  extensions: [health_check]      # Enabled extensions
  pipelines:                      # Configured pipelines
    traces:                       # Traces pipeline
      receivers:
      - otlp                      # OTLP receiver
      processors:                 # Processors for traces
      - memory_limiter
      - resource/add_mode
      - batch
      exporters:
      - debug                     # Debug exporter
      - file/traces
    metrics:                      # Metrics pipeline
      receivers:
      - otlp                      # OTLP receiver
      processors:                 # Processors for metrics
      - memory_limiter
      - resource/add_mode
      - batch
      exporters:
      - debug                     # Debug exporter
      - file/metrics
    logs:                         # Logs pipeline
      receivers:
      - otlp                      # OTLP receiver
      processors:                 # Processors for logs
      - memory_limiter
      - resource/add_mode
      - batch
      exporters:
      - debug                     # Debug exporter
      - file/logs

Note

When the gateway is started it will generate three files: gateway-traces.out, gateway-metrics.out, and gateway-logs.out. These files will eventually contain the telemetry data received by the gateway.

.
├── agent.yaml
└── gateway.yaml

2.1 Start Gateway

The configuration for the gateway does not need any additional configuration changes to function. This has been done to save time and focus on the core concepts of the Gateway.

Validate the gateway configuration using otelbin.io. For reference, the logs: section of your pipelines will look similar to this:

%%{init:{"fontFamily":"monospace"}}%%
graph LR
    %% Nodes
      REC1(&nbsp;&nbsp;otlp&nbsp;&nbsp;<br>fa:fa-download):::receiver
      PRO1(memory_limiter<br>fa:fa-microchip):::processor
      PRO2(resource<br>fa:fa-microchip<br>add_mode):::processor
      PRO3(batch<br>fa:fa-microchip):::processor
      EXP1(&ensp;file&ensp;<br>fa:fa-upload<br>logs):::exporter
      EXP2(&ensp;debug&ensp;<br>fa:fa-upload):::exporter
    %% Links
    subID1:::sub-logs
    subgraph " "
      subgraph subID1[**Logs**]
      direction LR
      REC1 --> PRO1
      PRO1 --> PRO2
      PRO2 --> PRO3
      PRO3 --> EXP2
      PRO3 --> EXP1
      end
    end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-logs stroke:#34d399,stroke-width:1px, color:#34d399,stroke-dasharray: 3 3;

Exercise

Start the Gateway: In the Gateway terminal window, run the following command to start the gateway:

../otelcol --config=gateway.yaml

If everything is configured correctly, the first and last lines of the output should look like:

2025/01/15 15:33:53 settings.go:478: Set config to [gateway.yaml]
<snip to the end>
2025-01-13T12:43:51.747+0100 info service@v0.120.0/service.go:261 Everything is ready. Begin running and processing data.

Next, we will configure the agent to send data to the newly created gateway.

2.2 Configure Agent

Exercise

Add the otlphttp exporter: The OTLP/HTTP Exporter is used to send data from the agent to the gateway using the OTLP/HTTP protocol.

Switch to your Agent terminal window.
Validate that the newly generated gateway-logs.out, gateway-metrics.out, and gateway-traces.out are present in the directory.
Open the agent.yaml file in your editor.
Add the otlphttp exporter configuration to the exporters: section:

  otlphttp:                            # Exporter Type
    endpoint: "http://localhost:5318"  # Gateway OTLP endpoint

Add a Batch Processor configuration: The Batch Processor will accept spans, metrics, or logs and place them into batches. Batching helps better compress the data and reduce the number of outgoing connections required to transmit the data. It is highly recommended configuring the batch processor on every collector.

Add the batch processor configuration to the processors: section:

  batch:                               # Processor Type

Update the pipelines:

Enable Hostmetrics Receiver:
- Add hostmetrics to the metrics pipeline. The HostMetrics Receiver will generate host CPU metrics once per hour with the current configuration.
Enable Batch Processor:
- Add the batch processor (after the resource/add_mode processor) to the traces, metrics, and logs pipelines.
Enable OTLPHTTP Exporter:
- Add the otlphttp exporter to the traces, metrics, and logs pipelines.

  pipelines:
    traces:
      receivers:
      - otlp                           # OTLP Receiver
      processors:
      - memory_limiter                 # Memory Limiter processor
      - resourcedetection              # Add system attributes to the data
      - resource/add_mode              # Add collector mode metadata
      - batch                          # Batch processor
      exporters:
      - debug                          # Debug Exporter
      - file                           # File Exporter
      - otlphttp                       # OTLP/HTTP Exporter
    metrics:
      receivers:
      - otlp
      - hostmetrics                    # Host Metrics Receiver
      processors:
      - memory_limiter
      - resourcedetection
      - resource/add_mode
      - batch
      exporters:
      - debug
      - otlphttp
    logs:
      receivers:
      - otlp
      processors:
      - memory_limiter
      - resourcedetection
      - resource/add_mode
      - batch
      exporters:
      - debug
      - otlphttp

Validate the agent configuration using otelbin.io. For reference, the metrics: section of your pipelines will look similar to this:

%%{init:{"fontFamily":"monospace"}}%%
graph LR
    %% Nodes
      REC1(&nbsp;&nbsp;&nbsp;&nbsp;otlp&nbsp;&nbsp;&nbsp;&nbsp;<br>fa:fa-download):::receiver
      REC2(hostmetrics<br>fa:fa-download):::receiver
      PRO1(memory_limiter<br>fa:fa-microchip):::processor
      PRO2(resourcedetection<br>fa:fa-microchip):::processor
      PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
      PRO4(batch<br>fa:fa-microchip):::processor
      EXP1(otlphttp<br>fa:fa-upload):::exporter
      EXP2(&ensp;debug&ensp;<br>fa:fa-upload):::exporter
    %% Links
    subID1:::sub-metrics
    subgraph " "
      subgraph subID1[**Metrics**]
      direction LR
      REC1 --> PRO1
      REC2 --> PRO1
      PRO1 --> PRO2
      PRO2 --> PRO3
      PRO3 --> PRO4
      PRO4 --> EXP2
      PRO4 --> EXP1
      end
    end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-metrics stroke:#38bdf8,stroke-width:1px, color:#38bdf8,stroke-dasharray: 3 3;

2.3 Send metrics from the Agent to the Gateway

Exercise

Start the Agent: In the Agent terminal window start the agent with the updated configuration:

../otelcol --config=agent.yaml

Verify CPU Metrics:

Check that when the agent starts, it immediately starts sending CPU metrics.
Both the agent and the gateway will display this activity in their debug output. The output should resemble the following snippet:

<snip>
NumberDataPoints #37
Data point attributes:
    -> cpu: Str(cpu0)
    -> state: Str(system)
StartTimestamp: 2024-12-09 14:18:28 +0000 UTC
Timestamp: 2025-01-15 15:27:51.319526 +0000 UTC
Value: 9637.660000

At this stage, the agent continues to collect CPU metrics once per hour or upon each restart and sends them to the gateway.

The gateway processes these metrics and exports them to a file named ./gateway-metrics.out. This file stores the exported metrics as part of the pipeline service.

Verify Data arrived at Gateway: To confirm that CPU metrics, specifically for cpu0, have successfully reached the gateway, we’ll inspect the gateway-metrics.out file using the jq command.

The following command filters and extracts the system.cpu.time metric, focusing on cpu0. It displays the metric’s state (e.g., user, system, idle, interrupt) along with the corresponding values.

Run the command below in the Tests terminal to check the system.cpu.time metric:

jq '.resourceMetrics[].scopeMetrics[].metrics[] | select(.name == "system.cpu.time") | .sum.dataPoints[] | select(.attributes[0].value.stringValue == "cpu0") | {cpu: .attributes[0].value.stringValue, state: .attributes[1].value.stringValue, value: .asDouble}' gateway-metrics.out

{
  "cpu": "cpu0",
  "state": "user",
  "value": 123407.02
}
{
  "cpu": "cpu0",
  "state": "system",
  "value": 64866.6
}
{
  "cpu": "cpu0",
  "state": "idle",
  "value": 216427.87
}
{
  "cpu": "cpu0",
  "state": "interrupt",
  "value": 0
}

2.4 Send traces from the Agent to the Gateway

Exercise

Send a Test Trace:

Validate agent and gateway are still running.
In the Spans terminal window, run the following command to send 5 spans and validate the output of the agent and gateway debug logs:

../loadgen -count 5

2025-03-06T11:49:00.456Z        info    Traces  {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 1}
2025-03-06T11:49:00.456Z        info    ResourceSpans #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.6.1
Resource attributes:
     -> service.name: Str(cinema-service)
     -> deployment.environment: Str(production)
     -> host.name: Str(workshop-instance)
     -> os.type: Str(linux)
     -> otelcol.service.mode: Str(agent)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope cinema.library 1.0.0
InstrumentationScope attributes:
     -> fintest.scope.attribute: Str(Starwars, LOTR)
Span #0
    Trace ID       : 97fb4e5b13400b5689e3306da7cff077
    Parent ID      :
    ID             : 413358465e5b4f15
    Name           : /movie-validator
    Kind           : Server
    Start time     : 2025-03-06 11:49:00.431915 +0000 UTC
    End time       : 2025-03-06 11:49:01.431915 +0000 UTC
    Status code    : Ok
    Status message : Success
Attributes:
     -> user.name: Str(George Lucas)
     -> user.phone_number: Str(+1555-867-5309)
     -> user.email: Str(george@deathstar.email)
     -> user.password: Str(LOTR>StarWars1-2-3)
     -> user.visa: Str(4111 1111 1111 1111)
     -> user.amex: Str(3782 822463 10005)
     -> user.mastercard: Str(5555 5555 5555 4444)
     -> payment.amount: Double(87.01)
        {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces"}

Verify the Gateway has handled the spans: Once the gateway processes incoming spans, it writes the trace data to a file named gateway-traces.out. To confirm that the spans have been successfully handled, you can inspect this file.

Using the jq command, you can extract and display key details about each span, such as its spanId and its position in the file. Also, we can extract the attributes that the Hostmetrics Receiver added to the spans.

jq -c '.resourceSpans[] as $resource | $resource.scopeSpans[].spans[] | "Span \(input_line_number) found with spanId \(.spanId), hostname \($resource.resource.attributes[] | select(.key == "host.name") | .value.stringValue), os \($resource.resource.attributes[] | select(.key == "os.type") | .value.stringValue)"' ./gateway-traces.out

"Span 1 found with spanId d71fe6316276f97d, hostname workshop-instance, os linux"
"Span 2 found with spanId e8d19489232f8c2a, hostname workshop-instance, os linux"
"Span 3 found with spanId 9dfaf22857a6bd05, hostname workshop-instance, os linux"
"Span 4 found with spanId c7f544a4b5fef5fc, hostname workshop-instance, os linux"
"Span 5 found with spanId 30bb49261315969d, hostname workshop-instance, os linux"

Important

Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.

3. FileLog Setup

10 minutes

The FileLog Receiver in the OpenTelemetry Collector is used to ingest logs from files.

It monitors specified files for new log entries and streams those logs into the Collector for further processing or exporting. It is also useful for testing and development purposes.

For this part of the workshop, the loadgen will generate logs using random quotes:

lotrQuotes := []string{
    "One does not simply walk into Mordor.",
    "Even the smallest person can change the course of the future.",
    "All we have to decide is what to do with the time that is given us.",
    "There is some good in this world, and it's worth fighting for.",
}

starWarsQuotes := []string{
    "Do or do not, there is no try.",
    "The Force will be with you. Always.",
    "I find your lack of faith disturbing.",
    "In my experience, there is no such thing as luck.",
}

The FileLog receiver in the agent will read these log lines and send them to the gateway.

Exercise

In the Logs terminal window, change into the [WORKSHOP] directory and create a new subdirectory named 3-filelog.
Next, copy *.yaml from 2-gateway into 3-filelog.

Important

Change ALL terminal windows to the [WORKSHOP]/3-filelog directory.

Start the loadgen and this will begin writing lines to a file named quotes.log:

../loadgen -logs

Writing logs to quotes.log. Press Ctrl+C to stop.

.
├── agent.yaml
├── gateway.yaml
└── quotes.yaml

3.1 Configuration

Exercise

In the Agent terminal window edit the agent.yaml and configure the FileLog receiver.

Configure FileLog Receiver: The filelog receiver reads log data from a file and includes custom resource attributes in the log data:

  filelog/quotes:                      # Receiver Type/Name
    include: ./quotes.log              # The file to read log data from
    include_file_path: true            # Include file path in the log data
    include_file_name: false           # Exclude file name from the log data
    resource:                          # Add custom resource attributes
      com.splunk.source: ./quotes.log  # Source of the log data
      com.splunk.sourcetype: quotes    # Source type of the log data

Update logs pipeline: Add thefilelog/quotes receiver to the logs pipeline only:

    logs:
      receivers:
      - otlp
      - filelog/quotes                 # Filelog Receiver
      processors:
      - memory_limiter
      - resourcedetection
      - resource/add_mode
      - batch
      exporters:
      - debug
      - otlphttp

Validate Configuration: Paste the updated agent.yaml into otelbin.io. For reference, the logs: section of your pipelines will look similar to this:

%%{init:{"fontFamily":"monospace"}}%%
graph LR
    %% Nodes
      REC1(&nbsp;&nbsp;otlp&nbsp;&nbsp;<br>fa:fa-download):::receiver
      REC2(filelog<br>fa:fa-download<br>quotes):::receiver
      PRO1(memory_limiter<br>fa:fa-microchip):::processor
      PRO2(resourcedetection<br>fa:fa-microchip):::processor
      PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
      PRO4(batch<br>fa:fa-microchip):::processor
      EXP1(&ensp;debug&ensp;<br>fa:fa-upload):::exporter
      EXP2(otlphttp<br>fa:fa-upload):::exporter
    %% Links
    subID1:::sub-logs
    subgraph " "
      subgraph subID1[**Logs**]
      direction LR
      REC1 --> PRO1
      REC2 --> PRO1
      PRO1 --> PRO2
      PRO2 --> PRO3
      PRO3 --> PRO4
      PRO4 --> EXP1
      PRO4 --> EXP2
      end
    end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-logs stroke:#34d399,stroke-width:1px, color:#34d399,stroke-dasharray: 3 3;

3.2 Test FileLog Receiver

Exercise

Start the Gateway: In your Gateway terminal window start the gateway.

Start the Agent: In your Agent terminal window start the agent.

A continuous stream of log data from the quotes.log will be in the agent and gateway debug logs:

Timestamp: 1970-01-01 00:00:00 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Str(2025-03-06 15:18:32 [ERROR] - There is some good in this world, and it's worth fighting for. LOTR)
Attributes:
     -> log.file.path: Str(quotes.log)
Trace ID:
Span ID:
Flags: 0
LogRecord #1

Stop the loadgen: In the Logs terminal window, stop the loadgen using Ctrl-C.

Verify the gateway: Check if the gateway has written a ./gateway-logs.out file.

At this point, your directory structure will appear as follows:

.
├── agent.out
├── agent.yaml
├── gateway-logs.out     # Output from the logs pipeline
├── gateway-metrics.out  # Output from the metrics pipeline
├── gateway-traces.out   # Output from the traces pipeline
├── gateway.yaml
└── quotes.log           # File containing Random log lines

Examine a log line: In gateway-logs.out compare a log line with the snippet below. Verify that the log entry includes the same attributes as we have seen in metrics and traces data previously:

{"resourceLogs":[{"resource":{"attributes":[{"key":"com.splunk.source","value":{"stringValue":"./quotes.log"}},{"key":"com.splunk.sourcetype","value":{"stringValue":"quotes"}},{"key":"host.name","value":{"stringValue":"workshop-instance"}},{"key":"os.type","value":{"stringValue":"linux"}},{"key":"otelcol.service.mode","value":{"stringValue":"gateway"}}]},"scopeLogs":[{"scope":{},"logRecords":[{"observedTimeUnixNano":"1741274312475540000","body":{"stringValue":"2025-03-06 15:18:32 [DEBUG] - All we have to decide is what to do with the time that is given us. LOTR"},"attributes":[{"key":"log.file.path","value":{"stringValue":"quotes.log"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1741274312475560000","body":{"stringValue":"2025-03-06 15:18:32 [DEBUG] - Your focus determines your reality. SW"},"attributes":[{"key":"log.file.path","value":{"stringValue":"quotes.log"}}],"traceId":"","spanId":""}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.6.1"}]}

{
  "resourceLogs": [
    {
      "resource": {
        "attributes": [
          {
            "key": "com.splunk.source",
            "value": {
              "stringValue": "./quotes.log"
            }
          },
          {
            "key": "com.splunk.sourcetype",
            "value": {
              "stringValue": "quotes"
            }
          },
          {
            "key": "host.name",
            "value": {
              "stringValue": "workshop-instance"
            }
          },
          {
            "key": "os.type",
            "value": {
              "stringValue": "linux"
            }
          },
          {
            "key": "otelcol.service.mode",
            "value": {
              "stringValue": "gateway"
            }
          }
        ]
      },
      "scopeLogs": [
        {
          "scope": {},
          "logRecords": [
            {
              "observedTimeUnixNano": "1741274312475540000",
              "body": {
                "stringValue": "2025-03-06 15:18:32 [DEBUG] - All we have to decide is what to do with the time that is given us. LOTR"
              },
              "attributes": [
                {
                  "key": "log.file.path",
                  "value": {
                    "stringValue": "quotes.log"
                  }
                }
              ],
              "traceId": "",
              "spanId": ""
            },
            {
              "observedTimeUnixNano": "1741274312475560000",
              "body": {
                "stringValue": "2025-03-06 15:18:32 [DEBUG] - Your focus determines your reality. SW"
              },
              "attributes": [
                {
                  "key": "log.file.path",
                  "value": {
                    "stringValue": "quotes.log"
                  }
                }
              ],
              "traceId": "",
              "spanId": ""
            }
          ]
        }
      ],
      "schemaUrl": "https://opentelemetry.io/schemas/1.6.1"
    }
  ]
}

You may also have noticed that every log line contains empty placeholders for "traceId":"" and "spanId":"". The FileLog receiver will populate these fields only if they are not already present in the log line. For example, if the log line is generated by an application instrumented with an OpenTelemetry instrumentation library, these fields will already be included and will not be overwritten.

Important

Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.

4. Building In Resilience

10 minutes

The OpenTelemetry Collector’s FileStorage Extension enhances the resilience of your telemetry pipeline by providing reliable checkpointing, managing retries, and handling temporary failures effectively.

With this extension enabled, the OpenTelemetry Collector can store intermediate states on disk, preventing data loss during network disruptions and allowing it to resume operations seamlessly.

Note

This solution will work for metrics as long as the connection downtime is brief—up to 15 minutes. If the downtime exceeds this, Splunk Observability Cloud will drop data due to datapoints being out of order.

For logs, there are plans to implement a more enterprise-ready solution in one of the upcoming Splunk OpenTelemetry Collector releases.

Exercise

Inside the [WORKSHOP] directory, create a new subdirectory named 4-resilience.
Next, copy *.yaml from the 3-filelog directory into 4-resilience.

Important

Change ALL terminal windows to the [WORKSHOP]/4-resilience directory.

Your updated directory structure will now look like this:

.
├── agent.yaml
└── gateway.yaml

4.1 File Storage Configuration

In this exercise, we will update the extensions: section of the agent.yaml file. This section is part of the OpenTelemetry configuration YAML and defines optional components that enhance or modify the OpenTelemetry Collector’s behavior.

While these components do not process telemetry data directly, they provide valuable capabilities and services to improve the Collector’s functionality.

Exercise

Update the agent.yaml: In the Agent terminal window, add the file_storage extension and name it checkpoint:

  file_storage/checkpoint:             # Extension Type/Name
    directory: "./checkpoint-dir"      # Define directory
    create_directory: true             # Create directory
    timeout: 1s                        # Timeout for file operations
    compaction:                        # Compaction settings
      on_start: true                   # Start compaction at Collector startup
      # Define compaction directory
      directory: "./checkpoint-dir/tmp"
      max_transaction_size: 65536      # Max. size limit before compaction occurs

Add file_storage to existing otlphttp exporter: Modify the otlphttp: exporter to configure retry and queuing mechanisms, ensuring data is retained and resent if failures occur:

  otlphttp: 
    endpoint: "http://localhost:5318"
    retry_on_failure:
      enabled: true                    # Enable retry on failure
    sending_queue:                     # 
      enabled: true                    # Enable sending queue
      num_consumers: 10                # No. of consumers
      queue_size: 10000                # Max. queue size
      storage: file_storage/checkpoint # File storage extension

Update the services section: Add the file_storage/checkpoint extension to the existing extensions: section. This will cause the extension to be enabled:

service:
  extensions:
  - health_check
  - file_storage/checkpoint            # Enabled extensions for this collector

Update the metrics pipeline: For this exercise we are going to remove the hostmetrics receiver from the Metric pipeline to reduce debug and log noise:

    metrics:
      receivers:
      - otlp
      # - hostmetrics                  # Hostmetrics Receiver

Validate the agent configuration using otelbin.io. For reference, the metrics: section of your pipelines will look similar to this:

%%{init:{"fontFamily":"monospace"}}%%
graph LR
    %% Nodes
      REC1(&nbsp;&nbsp;otlp&nbsp;&nbsp;<br>fa:fa-download):::receiver
      PRO1(memory_limiter<br>fa:fa-microchip):::processor
      PRO2(resourcedetection<br>fa:fa-microchip):::processor
      PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
      PRO4(batch<br>fa:fa-microchip):::processor
      EXP1(&ensp;debug&ensp;<br>fa:fa-upload):::exporter
      EXP2(otlphttp<br>fa:fa-upload):::exporter
    %% Links
    subID1:::sub-metrics
    subgraph " "
      subgraph subID1[**Metrics**]
      direction LR
      REC1 --> PRO1
      PRO1 --> PRO2
      PRO2 --> PRO3
      PRO3 --> PRO4
      PRO4 --> EXP1
      PRO4 --> EXP2
      end
    end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-metrics stroke:#38bdf8,stroke-width:1px, color:#38bdf8,stroke-dasharray: 3 3;

4.2 Setup environment for Resilience Testing

Next, we will configure our environment to be ready for testing the File Storage configuration.

Exercise

Start the Gateway: In the Gateway terminal window navigate to the [WORKSHOP]/4-resilience directory and run:

../otelcol --config=gateway.yaml

Start the Agent: In the Agent terminal window navigate to the [WORKSHOP]/4-resilience directory and run:

../otelcol --config=agent.yaml

Send five test spans: In the Spans terminal window navigate to the [WORKSHOP]/4-resilience directory and run:

../loadgen -count 5

Both the agent and gateway should display debug logs, and the gateway should create a ./gateway-traces.out file.

If everything functions correctly, we can proceed with testing system resilience.

4.3 Simulate Failure

To assess the Agent’s resilience, we’ll simulate a temporary gateway outage and observe how the agent handles it:

Summary:

Send Traces to the Agent – Generate traffic by sending traces to the agent.
Stop the Gateway – This will trigger the agent to enter retry mode.
Restart the Gateway – The agent will recover traces from its persistent queue and forward them successfully. Without the persistent queue, these traces would have been lost permanently.

Exercise

Simulate a network failure: In the Gateway terminal stop the gateway with Ctrl-C and wait until the gateway console shows that it has stopped:

2025-01-28T13:24:32.785+0100  info  service@v0.120.0/service.go:309  Shutdown complete.

Send traces: In the Spans terminal window send five more traces using the loadgen.

Notice that the agent’s retry mechanism is activated as it continuously attempts to resend the data. In the agent’s console output, you will see repeated messages similar to the following:

2025-01-28T14:22:47.020+0100  info  internal/retry_sender.go:126  Exporting failed. Will retry the request after interval.  {"kind": "exporter", "data_type": "traces", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"http://localhost:5318/v1/traces\": dial tcp 127.0.0.1:5318: connect: connection refused", "interval": "9.471474933s"}

Stop the Agent: In the Agent terminal window, use Ctrl-C to stop the agent. Wait until the agent’s console confirms it has stopped:

2025-01-28T14:40:28.702+0100  info  extensions/extensions.go:66  Stopping extensions...
2025-01-28T14:40:28.702+0100  info  service@v0.120.0/service.go:309  Shutdown complete.

Tip

Stopping the agent will halt its retry attempts and prevent any future retry activity.

If the agent runs for too long without successfully delivering data, it may begin dropping traces, depending on the retry configuration, to conserve memory. By stopping the agent, any metrics, traces, or logs currently stored in memory are lost before being dropped, ensuring they remain available for recovery.

This step is essential for clearly observing the recovery process when the agent is restarted.

4.4 Recovery

In this exercise, we’ll test how the OpenTelemetry Collector recovers from a network outage by restarting the Gateway. When the gateway becomes available again, the agent will resume sending data from its last checkpointed state, ensuring no data loss.

Exercise

Restart the Gateway: In the Gateway terminal window run:

../otelcol --config=gateway.yaml

Restart the Agent: In the Agent terminal window run:

../otelcol --config=agent.yaml

After the agent is up and running, the File_Storage extension will detect buffered data in the checkpoint folder.
It will start to dequeue the stored spans from the last checkpoint folder, ensuring no data is lost.

Exercise

Verify the Agent Debug output
Note that the Agent Debug Screen does NOT change and still shows the following line indicating no new data is being exported.

2025-02-07T13:40:12.195+0100    info    service@v0.120.0/service.go:253 Everything is ready. Begin running and processing data.

Watch the Gateway Debug output
You should see from the gateway debug screen, it has started receiving the previously missed traces without requiring any additional action on your part.

2025-02-07T12:44:32.651+0100    info    service@v0.120.0/service.go:253 Everything is ready. Begin running and processing data.
2025-02-07T12:47:46.721+0100    info    Traces  {"kind": "exporter", "data_type": "traces", "name": "debug", "resource spans": 4, "spans": 4}
2025-02-07T12:47:46.721+0100    info    ResourceSpans #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.6.1
Resource attributes:

Check the gateway-traces.out file
Using jq, count the number of traces in the recreated gateway-traces.out. It should match the number you send when the gateway was down.

jq '.resourceSpans | length | "\(.) resourceSpans found"' gateway-traces.out

Important

Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.

Conclusion

This exercise demonstrated how to enhance the resilience of the OpenTelemetry Collector by configuring the file_storage extension, enabling retry mechanisms for the otlp exporter, and using a file-backed queue for temporary data storage.

By implementing file-based checkpointing and queue persistence, you ensure the telemetry pipeline can gracefully recover from temporary interruptions, making it a more robust and reliable for production environments.

5. Dropping Spans

10 minutes

In this section, we will explore how to use the Filter Processor to selectively drop spans based on certain conditions.

Specifically, we will drop traces based on the span name, which is commonly used to filter out unwanted spans such as health checks or internal communication traces. In this case, we will be filtering out spans whose name is "/_healthz", typically associated with health check requests and usually are quite “noisy”.

Exercise

Inside the [WORKSHOP] directory, create a new subdirectory named 5-dropping-spans.
Next, copy *.yaml from the 4-resilience directory into 5-dropping-spans.

Important

Change ALL terminal windows to the [WORKSHOP]/5-dropping-spans directory.

Your updated directory structure will now look like this:

.
├── agent.yaml
└── gateway.yaml

Next, we will configure the filter processor and the respective pipelines.

5.1 Configuration

Exercise

Switch to your Gateway terminal window and open the gateway.yaml file. Update the processors section with the following configuration:

Add a filter processor:
Configure the gateway to exclude spans with the name /_healthz. The error_mode: ignore directive ensures that any errors encountered during filtering are ignored, allowing the pipeline to continue running smoothly. The traces section defines the filtering rules, specifically targeting spans named /_healthz for exclusion.

filter/health:                       # Defines a filter processor
  error_mode: ignore                 # Ignore errors
  traces:                            # Filtering rules for traces
    span:                            # Exclude spans named "/_healthz"
      - 'name == "/_healthz"'

Add the filter processor to the traces pipeline:
Include the filter/health processor in the traces pipeline. For optimal performance, place the filter as early as possible—right after the memory_limiter and before the batch processor. Here’s how the configuration should look:

traces:
  receivers:
    - otlp
  processors:
    - memory_limiter
    - filter/health             # Filters data based on rules
    - resource/add_mode
    - batch
  exporters:
    - debug
    - file/traces

This setup ensures that health check-related spans (/_healthz) are filtered out early in the pipeline, reducing unnecessary noise in your telemetry data.

Validate the agent configuration using otelbin.io. For reference, the traces: section of your pipelines will look similar to this:

%%{init:{"fontFamily":"monospace"}}%%
graph LR
    %% Nodes
      REC1(&nbsp;&nbsp;otlp&nbsp;&nbsp;<br>fa:fa-download):::receiver
      PRO1(memory_limiter<br>fa:fa-microchip):::processor
      PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
      PRO4(filter<br>fa:fa-microchip<br>health):::processor
      PRO5(batch<br>fa:fa-microchip):::processor
      EXP1(&ensp;debug&ensp;<br>fa:fa-upload):::exporter
      EXP2(&ensp;&ensp;file&ensp;&ensp;<br>fa:fa-upload<br>traces):::exporter
    %% Links
    subID1:::sub-traces
    subgraph " "
      subgraph subID1[**Traces**]
      direction LR
      REC1 --> PRO1
      PRO1 --> PRO4
      PRO4 --> PRO3
      PRO3 --> PRO5
      PRO5 --> EXP1
      PRO5 --> EXP2
      end
    end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3;

5.2 Test Filter Processor

To test your configuration, you’ll need to generate some trace data that includes a span named "/_healthz".

Exercise

Start the Gateway: In your Gateway terminal window start the gateway.

./otelcol --config ./gateway.yaml

Start the Agent: In your Agent terminal window start the agent.

./otelcol --config ./agent.yaml

Start the Loadgen: In the Spans terminal window run the loadgen with the flag to also send healthz spans along with base spans:

../loadgen -health -count 5

Verify agent.out: Using jq confirm the name of the spans received by the agent:

jq -c '.resourceSpans[].scopeSpans[].spans[] | "Span \(input_line_number) found with name \(.name)"' ./agent.out

"Span 1 found with name /movie-validator"
"Span 2 found with name /_healthz"
"Span 3 found with name /movie-validator"
"Span 4 found with name /_healthz"
"Span 5 found with name /movie-validator"
"Span 6 found with name /_healthz"
"Span 7 found with name /movie-validator"
"Span 8 found with name /_healthz"
"Span 9 found with name /movie-validator"
"Span 10 found with name /_healthz"

Check the Gateway Debug output: Using jq confirm the name of the spans received by the gateway:

jq -c '.resourceSpans[].scopeSpans[].spans[] | "Span \(input_line_number) found with name \(.name)"' ./gateway-traces.out

The gateway-metrics.out file will not contain any spans named /_healthz.

"Span 1 found with name /movie-validator"
"Span 2 found with name /movie-validator"
"Span 3 found with name /movie-validator"
"Span 4 found with name /movie-validator"
"Span 5 found with name /movie-validator"

Tip

When using the Filter processor make sure you understand the look of your incoming data and test the configuration thoroughly. In general, use as specific a configuration as possible to lower the risk of the wrong data being dropped.

You can further extend this configuration to filter out spans based on different attributes, tags, or other criteria, making the OpenTelemetry Collector more customizable and efficient for your observability needs.

Important

Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.

6. Redacting Sensitive Data

10 minutes

In this section, you’ll learn how to configure the OpenTelemetry Collector to remove specific tags and redact sensitive data from telemetry spans. This is crucial for protecting sensitive information such as credit card numbers, personal data, or other security-related details that must be anonymized before being processed or exported.

We’ll walk through configuring key processors in the OpenTelemetry Collector, including:

Attributes Processor: Modifies or removes specific span attributes.
Redaction Processor: Ensures sensitive data is sanitized before being stored or transmitted.

Exercise

Inside the [WORKSHOP] directory, create a new subdirectory named 6-sensitive-data.
Next, copy *.yaml from the 5-dropping-spans directory into 6-sensitive-data.

Important

Change ALL terminal windows to the [WORKSHOP]/6-sensitive-data directory.

Your updated directory structure will now look like this:

.
├── agent.yaml
└── gateway.yaml

6.1 Configuration

In this step, we’ll modify agent.yaml to include the attributes and redaction processors. These processors will help ensure that sensitive data within span attributes is properly handled before being logged or exported.

Previously, you may have noticed that some span attributes displayed in the console contained personal and sensitive data. We’ll now configure the necessary processors to filter out and redact this information effectively.

<snip>
Attributes:
     -> user.name: Str(George Lucas)
     -> user.phone_number: Str(+1555-867-5309)
     -> user.email: Str(george@deathstar.email)
     -> user.account_password: Str(LOTR>StarWars1-2-3)
     -> user.visa: Str(4111 1111 1111 1111)
     -> user.amex: Str(3782 822463 10005)
     -> user.mastercard: Str(5555 5555 5555 4444)
  {"kind": "exporter", "data_type": "traces", "name": "debug"}

Exercise

Switch to your Agent terminal window and open the agent.yaml file in your editor. We’ll add two processors to enhance the security and privacy of your telemetry data: the Attributes Processor and the Redaction Processor.

Add an attributes Processor: The Attributes Processor allows you to modify span attributes (tags) by updating, deleting, or hashing their values. This is particularly useful for obfuscating sensitive information before it is exported.

In this step, we’ll:

Update the user.phone_number attribute to a static value ("UNKNOWN NUMBER").
Hash the user.email attribute to ensure the original email is not exposed.
Delete the user.password attribute to remove it entirely from the span.

  attributes/update:
    actions:                           # Actions
      - key: user.phone_number         # Target key
        action: update                 # Update action
        value: "UNKNOWN NUMBER"        # New value
      - key: user.email                # Target key
        action: hash                   # Hash the email value
      - key: user.password             # Target key
        action: delete                 # Delete the password

Add a redaction Processor: The The Redaction Processor detects and redacts sensitive data in span attributes based on predefined patterns, such as credit card numbers or other personally identifiable information (PII).

In this step:

We set allow_all_keys: true to ensure all attributes are processed (if set to false, only explicitly allowed keys are retained).
We define blocked_values with regular expressions to detect and redact Visa and MasterCard credit card numbers.
The summary: debug option logs detailed information about the redaction process for debugging purposes.

  redaction/redact:
    allow_all_keys: true               # If false, only allowed keys will be retained
    blocked_values:                    # List of regex patterns to block
      - '\b4[0-9]{3}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b'       # Visa
      - '\b5[1-5][0-9]{2}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b'  # MasterCard
    summary: debug                     # Show debug details about redaction

Update the traces Pipeline: Integrate both processors into the traces pipeline. Make sure that you comment out the redaction processor at first (we will enable it later in a separate exercise):

    traces:
      receivers:
      - otlp
      processors:
      - memory_limiter
      - attributes/update              # Update, hash, and remove attributes
      #- redaction/redact              # Redact sensitive fields using regex
      - resourcedetection
      - resource/add_mode
      - batch
      exporters:
      - debug
      - file
      - otlphttp

Validate the agent configuration using otelbin.io. For reference, the traces: section of your pipelines will look similar to this:

%%{init:{"fontFamily":"monospace"}}%%
graph LR
    %% Nodes
      REC1(&nbsp;&nbsp;otlp&nbsp;&nbsp;<br>fa:fa-download):::receiver
      PRO1(memory_limiter<br>fa:fa-microchip):::processor
      PRO2(resourcedetection<br>fa:fa-microchip):::processor
      PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
      PRO5(batch<br>fa:fa-microchip):::processor
      PRO6(attributes<br>fa:fa-microchip<br>update):::processor
      EXP1(otlphttp<br>fa:fa-upload):::exporter
      EXP2(&ensp;&ensp;debug&ensp;&ensp;<br>fa:fa-upload):::exporter
      EXP3(file<br>fa:fa-upload):::exporter

    %% Links
    subID1:::sub-traces
    subgraph " "
      subgraph subID1[**Traces**]
      direction LR
      REC1 --> PRO1
      PRO1 --> PRO6
      PRO6 --> PRO2
      PRO2 --> PRO3
      PRO3 --> PRO5
      PRO5 --> EXP2
      PRO5 --> EXP3
      PRO5 --> EXP1
      end
    end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3;

6.2 Test Attribute Processor

In this exercise, we will delete the user.account_password, update the user.phone_number attribute and hash the user.email in the span data before it is exported by the agent.

Exercise

Start the Gateway: In your Gateway terminal window start the gateway.

../otelcol --config=gateway.yaml

Start the Agent: In your Agent terminal window start the agent.

../otelcol --config=agent.yaml

Start the Load Generator: In the Spans terminal window start the loadgen:

../loadgen -count 1

Check the debug output: For both the agent and gateway debug output, confirm that user.account_password has been removed, and both user.phone_number & user.email have been updated.

   -> user.name: Str(George Lucas)
   -> user.phone_number: Str(UNKNOWN NUMBER)
   -> user.email: Str(62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287)
   -> payment.amount: Double(51.71)
   -> user.visa: Str(4111 1111 1111 1111)
   -> user.amex: Str(3782 822463 10005)
   -> user.mastercard: Str(5555 5555 5555 4444)

    -> user.name: Str(George Lucas)
    -> user.phone_number: Str(+1555-867-5309)
    -> user.email: Str(george@deathstar.email)
    -> user.password: Str(LOTR>StarWars1-2-3)
    -> user.visa: Str(4111 1111 1111 1111)
    -> user.amex: Str(3782 822463 10005)
    -> user.mastercard: Str(5555 5555 5555 4444)
    -> payment.amount: Double(95.22)

Check file output: Using jq validate that user.account_password has been removed, and user.phone_number & user.email have been updated in gateway-taces.out:

jq '.resourceSpans[].scopeSpans[].spans[].attributes[] | select(.key == "user.password" or .key == "user.phone_number" or .key == "user.email") | {key: .key, value: .value.stringValue}' ./gateway-traces.out

Notice that the user.account_password has been removed, and the user.phone_number & user.email have been updated:

{
  "key": "user.phone_number",
  "value": "UNKNOWN NUMBER"
}
{
  "key": "user.email",
  "value": "62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287"
}

Important

Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.

6.3 Test Redaction Processor

The redaction processor gives precise control over which attributes and values are permitted or removed from telemetry data.

In this exercise, we will redact the user.visa & user.mastercard values in the span data before it is exported by the agent.

Exercise

Prepare the terminals: Delete the *.out files and clear the screen.

Start the Gateway: In your Gateway terminal window start the gateway.

../otelcol --config=gateway.yaml

Enable the redaction/redact processor: In the Agent terminal window, edit agent.yaml and remove the # we inserted in the previous exercise.

    traces:
      receivers:
      - otlp
      processors:
      - memory_limiter
      - attributes/update              # Update, hash, and remove attributes
      - redaction/redact               # Redact sensitive fields using regex
      - resourcedetection
      - resource/add_mode
      - batch
      exporters:
      - debug
      - file
      - otlphttp

Start the Agent: In your Agent terminal window start the agent.

../otelcol --config=agent.yaml

Start the Load Generator: In the Spans terminal window start the loadgen:

../loadgen -count 1

Check the debug output: For both the agent and gateway confirm the values for user.visa & user.mastercard have been updated. Notice user.amex attribute value was NOT redacted because a matching regex pattern was not added to blocked_values

   -> user.name: Str(George Lucas)
   -> user.phone_number: Str(UNKNOWN NUMBER)
   -> user.email: Str(62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287)
   -> payment.amount: Double(69.71)
   -> user.visa: Str(****)
   -> user.amex: Str(3782 822463 10005)
   -> user.mastercard: Str(****)
   -> redaction.masked.keys: Str(user.mastercard,user.visa)
   -> redaction.masked.count: Int(2)

    -> user.name: Str(George Lucas)
    -> user.phone_number: Str(+1555-867-5309)
    -> user.email: Str(george@deathstar.email)
    -> user.password: Str(LOTR>StarWars1-2-3)
    -> user.visa: Str(4111 1111 1111 1111)
    -> user.amex: Str(3782 822463 10005)
    -> user.mastercard: Str(5555 5555 5555 4444)
    -> payment.amount: Double(65.54)

Note

By including summary:debug in the redaction processor, the debug output will include summary information about which matching keys values were redacted, along with the count of values that were masked.

     -> redaction.masked.keys: Str(user.mastercard,user.visa)
     -> redaction.masked.count: Int(2)

Check file output: Using jq verify that user.visa & user.mastercard have been updated in the gateway-traces.out.

jq '.resourceSpans[].scopeSpans[].spans[].attributes[] | select(.key == "user.visa" or .key == "user.mastercard" or .key == "user.amex") | {key: .key, value: .value.stringValue}' ./gateway-traces.out

Notice that user.amex has not been redacted because a matching regex pattern was not added to blocked_values:

{
  "key": "user.visa",
  "value": "****"
}
{
  "key": "user.amex",
  "value": "3782 822463 10005"
}
{
  "key": "user.mastercard",
  "value": "****"
}

These are just a few examples of how attributes and redaction processors can be configured to protect sensitive data.

Important

Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.

7. Transform Data

10 minutes

The Transform Processor lets you modify telemetry data—logs, metrics, and traces—as it flows through the pipeline. Using the OpenTelemetry Transformation Language (OTTL), you can filter, enrich, and transform data on the fly without touching your application code.

In this exercise we’ll update agent.yaml to include a Transform Processor that will:

Filter log resource attributes.
Parse JSON structured log data into attributes.
Set log severity levels based on the log message body.

You may have noticed that in previous logs, fields like SeverityText and SeverityNumber were undefined (this is typical of the filelog receiver). However, the severity is embedded within the log body:

<snip>
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str(2025-01-31 15:49:29 [WARN] - Do or do not, there is no try.)
</snip>

Logs often contain structured data encoded as JSON within the log body. Extracting these fields into attributes allows for better indexing, filtering, and querying. Instead of manually parsing JSON in downstream systems, OTTL enables automatic transformation at the telemetry pipeline level.

Exercise

Inside the [WORKSHOP] directory, create a new subdirectory named 7-transform-data.
Next, copy *.yaml from the 6-sensitve-data directory into 7-transform-data.

Important

Change ALL terminal windows to the [WORKSHOP]/7-transform-data directory.

Your updated directory structure will now look like this:

.
├── agent.yaml
└── gateway.yaml

7.1 Configuration

Exercise

Add a transform processor: Switch to your Agent terminal window and edit the agent.yaml and add the following transform processor:

  transform/logs:                      # Processor Type/Name
    log_statements:                    # Log Processing Statements
      - context: resource              # Log Context
        statements:                    # List of attribute keys to keep
          - keep_keys(attributes, ["com.splunk.sourcetype", "host.name", "otelcol.service.mode"])

By using the -context: resource key we are targeting the resourceLog attributes of logs.

This configuration ensures that only the relevant resource attributes (com.splunk.sourcetype, host.name, otelcol.service.mode) are retained, improving log efficiency and reducing unnecessary metadata.

Adding a Context Block for Log Severity Mapping: To properly set the severity_text and severity_number fields of a log record, we add a log context block within log_statements. This configuration extracts the level value from the log body, maps it to severity_text, and assigns the corresponding severity_number based on the log level:

      - context: log                   # Log Context
        statements:                    # Transform Statements Array
          - set(cache, ParseJSON(body)) where IsMatch(body, "^\\{")  # Parse JSON log body into a cache object
          - flatten(cache, "")                                        # Flatten nested JSON structure
          - merge_maps(attributes, cache, "upsert")                   # Merge cache into attributes, updating existing keys
          - set(severity_text, attributes["level"])                   # Set severity_text from the "level" attribute
          - set(severity_number, 1) where severity_text == "TRACE"    # Map severity_text to severity_number
          - set(severity_number, 5) where severity_text == "DEBUG"
          - set(severity_number, 9) where severity_text == "INFO"
          - set(severity_number, 13) where severity_text == "WARN"
          - set(severity_number, 17) where severity_text == "ERROR"
          - set(severity_number, 21) where severity_text == "FATAL"

The merge_maps function is used to combine two maps (dictionaries) into one. In this case, it merges the cache object (containing parsed JSON data from the log body) into the attributes map.

Parameters:
- attributes: The target map where the data will be merged.
- cache: The source map containing the parsed JSON data.
- "upsert": This mode ensures that if a key already exists in the attributes map, its value will be updated with the value from cache. If the key does not exist, it will be inserted.

This step is crucial because it ensures that all relevant fields from the log body (e.g., level, message, etc.) are added to the attributes map, making them available for further processing or exporting.

Summary of Key Transformations:

Parse JSON: Extracts structured data from the log body.
Flatten JSON: Converts nested JSON objects into a flat structure.
Merge Attributes: Integrates extracted data into log attributes.
Map Severity Text: Assigns severity_text from the log’s level attribute.
Assign Severity Numbers: Converts severity levels into standardized numerical values.

You should have a single transform processor containing two context blocks: one for resource and one for log.

This configuration ensures that log severity is correctly extracted, standardized, and structured for efficient processing.

Tip

This method of mapping all JSON fields to top-level attributes should only be used for testing and debugging OTTL. It will result in high cardinality in a production scenario.

Update the logs pipeline: Add the transform/logs: processor into the logs: pipeline:

    logs:
      receivers:
      - otlp
      - filelog/quotes
      processors:
      - memory_limiter
      - resourcedetection
      - resource/add_mode
      - transform/logs                 # Transform logs processor
      - batch
      exporters:
      - debug
      - otlphttp

Validate the agent configuration using https://otelbin.io. For reference, the logs: section of your pipelines will look similar to this:

%%{init:{"fontFamily":"monospace"}}%%
graph LR
    %% Nodes
      REC1(&nbsp;&nbsp;otlp&nbsp;&nbsp;<br>fa:fa-download):::receiver
      REC2(filelog<br>fa:fa-download<br>quotes):::receiver
      PRO1(memory_limiter<br>fa:fa-microchip):::processor
      PRO2(resourcedetection<br>fa:fa-microchip):::processor
      PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
      PRO4(transform<br>fa:fa-microchip<br>logs):::processor
      PRO5(batch<br>fa:fa-microchip):::processor
      EXP1(otlphttp<br>fa:fa-upload):::exporter
      EXP2(&ensp;&ensp;debug&ensp;&ensp;<br>fa:fa-upload):::exporter
    %% Links
    subID1:::sub-logs
    subgraph " "
      subgraph subID1[**Logs**]
      direction LR
      REC1 --> PRO1
      REC2 --> PRO1
      PRO1 --> PRO2
      PRO2 --> PRO3
      PRO3 --> PRO4
      PRO4 --> PRO5
      PRO5 --> EXP2
      PRO5 --> EXP1
      end
    end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-logs stroke:#34d399,stroke-width:1px, color:#34d399,stroke-dasharray: 3 3;

7.2 Setup Environment

Exercise

Start the Gateway: In the Gateway terminal run:

../otelcol --config=gateway.yaml

Start the Agent: In the Agent terminal run:

../otelcol --config=agent.yaml

Start the Load Generator: Open the Logs terminal window and run the loadgen.

Important

To ensure the logs are structured in JSON format, include the -json flag when starting the script.

../loadgen -logs -json -count 5

The loadgen will write 5 log lines to ./quotes.log in JSON format.

7.3 Test Transform Processor

This test verifies that the com.splunk/source and os.type metadata have been removed from the log resource attributes before being exported by the agent. Additionally, the test ensures that:

The log body is parsed to extract severity information.
- SeverityText and SeverityNumber are set on the LogRecord.
JSON fields from the log body are promoted to log attributes.

This ensures proper metadata filtering, severity mapping, and structured log enrichment before export.

Exercise

Check the debug output: For both the agent and gateway confirm that com.splunk/source and os.type have been removed:

Resource attributes:
   -> com.splunk.sourcetype: Str(quotes)
   -> host.name: Str(workshop-instance)
   -> otelcol.service.mode: Str(agent)

Resource attributes:
   -> com.splunk.source: Str(./quotes.log)
   -> com.splunk.sourcetype: Str(quotes)
   -> host.name: Str(workshop-instance)
   -> os.type: Str(linux)
   -> otelcol.service.mode: Str(agent)

For both the agent and gateway confirm that SeverityText and SeverityNumber in the LogRecord is now defined with the severity level from the log body. Confirm that the JSON fields from the body can be accessed as top-level log Attributes:

<snip>
SeverityText: WARN
SeverityNumber: Warn(13)
Body: Str({"level":"WARN","message":"Your focus determines your reality.","movie":"SW","timestamp":"2025-03-07 11:17:26"})
Attributes:
     -> log.file.path: Str(quotes.log)
     -> level: Str(WARN)
     -> message: Str(Your focus determines your reality.)
     -> movie: Str(SW)
     -> timestamp: Str(2025-03-07 11:17:26)
</snip>

<snip>
SeverityText:
SeverityNumber: Unspecified(0)
Body: Str({"level":"WARN","message":"Your focus determines your reality.","movie":"SW","timestamp":"2025-03-07 11:17:26"})
Attributes:
     -> log.file.path: Str(quotes.log)
</snip>

Check file output: In the new gateway-logs.out file verify the data has been transformed:

jq '[.resourceLogs[].scopeLogs[].logRecords[] | {severityText, severityNumber, body: .body.stringValue}]' gateway-logs.out

[
  {
    "severityText": "DEBUG",
    "severityNumber": 5,
    "body": "{\"level\":\"DEBUG\",\"message\":\"All we have to decide is what to do with the time that is given us.\",\"movie\":\"LOTR\",\"timestamp\":\"2025-03-07 11:56:29\"}"
  },
  {
    "severityText": "WARN",
    "severityNumber": 13,
    "body": "{\"level\":\"WARN\",\"message\":\"The Force will be with you. Always.\",\"movie\":\"SW\",\"timestamp\":\"2025-03-07 11:56:29\"}"
  },
  {
    "severityText": "ERROR",
    "severityNumber": 17,
    "body": "{\"level\":\"ERROR\",\"message\":\"One does not simply walk into Mordor.\",\"movie\":\"LOTR\",\"timestamp\":\"2025-03-07 11:56:29\"}"
  },
  {
    "severityText": "DEBUG",
    "severityNumber": 5,
    "body": "{\"level\":\"DEBUG\",\"message\":\"Do or do not, there is no try.\",\"movie\":\"SW\",\"timestamp\":\"2025-03-07 11:56:29\"}"
  }
]
[
  {
    "severityText": "ERROR",
    "severityNumber": 17,
    "body": "{\"level\":\"ERROR\",\"message\":\"There is some good in this world, and it's worth fighting for.\",\"movie\":\"LOTR\",\"timestamp\":\"2025-03-07 11:56:29\"}"
  }
]

Important

Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.

8. Routing Data

10 minutes

The Routing Connector in OpenTelemetry is a powerful feature that allows you to direct data (traces, metrics, or logs) to different pipelines based on specific criteria. This is especially useful in scenarios where you want to apply different processing or exporting logic to subsets of your telemetry data.

For example, you might want to send production data to one exporter while directing test or development data to another. Similarly, you could route certain spans based on their attributes, such as service name, environment, or span name, to apply custom processing or storage logic.

Exercise

Inside the [WORKSHOP] directory, create a new subdirectory named 8-routing-data.
Next, copy *.yaml from the 7-transform-data directory into 8-routing-data.
Change all terminal windows to the [WORKSHOP]/8-routing-data directory.

Your updated directory structure will now look like this:

.
├── agent.yaml
└── gateway.yaml

Next, we will configure the routing connector and the respective pipelines.

8.1 Configure the Routing Connector

In this exercise, you will configure the Routing Connector in the gateway.yaml file. This setup enables the gateway to route traces based on the deployment.environment attribute in the spans you send. By implementing this, you can process and handle traces differently depending on their attributes.

Exercise

In OpenTelemetry configuration files, connectors have their own dedicated section, similar to receivers and processors.

Add the routing connector: In the Gateway terminal window edit gateway.yaml and add the following below the connectors: section:

  routing:
    default_pipelines: [traces/standard] # Default pipeline if no rule matches
    error_mode: ignore                   # Ignore errors in routing
    table:                               # Define routing rules
      # Routes spans to a target pipeline if the resourceSpan attribute matches the rule
      - statement: route() where attributes["deployment.environment"] == "security-applications"
        pipelines: [traces/security]     # Target pipeline

The rules above apply to traces, but this approach also applies to metrics and logs, allowing them to be routed based on attributes in resourceMetrics or resourceLogs.

Configure file: exporters: The routing connector requires separate targets for routing. Add two file exporters, file/traces/security and file/traces/standard, to ensure data is directed correctly.

  file/traces/standard:                    # Exporter for regular traces
    path: "./gateway-traces-standard.out"  # Path for saving trace data
    append: false                          # Overwrite the file each time
  file/traces/security:                    # Exporter for security traces
    path: "./gateway-traces-security.out"  # Path for saving trace data
    append: false                          # Overwrite the file each time

With the routing configuration complete, the next step is to configure the pipelines to apply these routing rules.

8.2 Configuring the Pipelines

Exercise

Update the traces pipeline to use routing:

To enable routing, update the original traces: pipeline by using routing as the only exporter. This ensures all span data is sent through the routing connector for evaluation.

Remove all processors and replace it with an empty array ([]). These are now defined in the traces/standard and traces/security pipelines.

  pipelines:
    traces:                           # Original traces pipeline
      receivers: 
      - otlp                          # OTLP Receiver
      processors: []
      exporters: 
      - routing                       # Routing Connector

Add both the standard and security traces pipelines:

Configure the Security pipeline: This pipeline will handle all spans that match the routing rule for security. This uses routing as its receiver. Place it below the existing traces: pipeline:

    traces/security:              # New Security Traces/Spans Pipeline
      receivers: 
      - routing                   # Receive data from the routing connector
      processors:
      - memory_limiter            # Memory Limiter Processor
      - resource/add_mode         # Adds collector mode metadata
      - batch
      exporters:
      - debug                     # Debug Exporter 
      - file/traces/security      # File Exporter for spans matching rule

Add the Standard pipeline: This pipeline processes all spans that do not match the routing rule. This pipeline is also using routing as its receiver. Add this below the traces/security one:

    traces/standard:              # Default pipeline for unmatched spans
      receivers: 
      - routing                   # Receive data from the routing connector
      processors:
      - memory_limiter            # Memory Limiter Processor
      - resource/add_mode         # Adds collector mode metadata
      - batch
      exporters:
      - debug                     # Debug exporter
      - file/traces/standard      # File exporter for unmatched spans

Validate the agent configuration using otelbin.io. For reference, the traces: section of your pipelines will look similar to this:

%%{init:{"fontFamily":"monospace"}}%%
graph LR
    %% Nodes
      REC1(&nbsp;&nbsp;&nbsp;otlp&nbsp;&nbsp;&nbsp;<br>fa:fa-download):::receiver
      PRO1(memory_limiter<br>fa:fa-microchip):::processor
      PRO2(memory_limiter<br>fa:fa-microchip):::processor
      PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
      PRO4(resource<br>fa:fa-microchip<br>add_mode):::processor
      PRO5(batch<br>fa:fa-microchip):::processor
      PRO6(batch<br>fa:fa-microchip):::processor
      EXP1(&nbsp;&ensp;debug&nbsp;&ensp;<br>fa:fa-upload):::exporter
      EXP2(&emsp;&emsp;file&emsp;&emsp;<br>fa:fa-upload<br>traces):::exporter
      EXP3(&nbsp;&ensp;debug&nbsp;&ensp;<br>fa:fa-upload):::exporter
      EXP4(&emsp;&emsp;file&emsp;&emsp;<br>fa:fa-upload<br>traces):::exporter
      ROUTE1(&nbsp;routing&nbsp;<br>fa:fa-route):::con-export
      ROUTE2(&nbsp;routing&nbsp;<br>fa:fa-route):::con-receive
      ROUTE3(&nbsp;routing&nbsp;<br>fa:fa-route):::con-receive
    %% Links
    subID1:::sub-traces
    subID2:::sub-traces
    subID3:::sub-traces
    subgraph " "
    direction LR
      subgraph subID1[**Traces**]
      REC1 --> ROUTE1
      end
      subgraph subID2[**Traces/standard**]
      ROUTE1 --> ROUTE2
      ROUTE2 --> PRO1
      PRO1 --> PRO3
      PRO3 --> PRO5
      PRO5 --> EXP1
      PRO5 --> EXP2
      end
      subgraph subID3[**Traces/security**]
      ROUTE1 --> ROUTE3
      ROUTE3 --> PRO2
      PRO2 --> PRO4
      PRO4 --> PRO6
      PRO6 --> EXP3
      PRO6 --> EXP4
      end
    end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3;

8.3 Test Routing Connector

Exercise

In this section, we will test the routing rule configured for the Gateway. The expected result is that thespan generated by the loadgen will be sent to the gateway-traces-security.out file.

Start the Gateway: In your Gateway terminal window start the gateway.

../otelcol --config gateway.yaml

Start the Agent: In your Agent terminal window start the agent.

../otelcol --config agent.yaml

Send a Regular Span: In the Spans terminal window send a regular span using the loadgen:

../loadgen -count 1

Both the agent and gateway will display debug information. The gateway will also generate a new gateway-traces-standard.out file, as this is now the designated destination for regular spans.

Tip

If you check gateway-traces-standard.out, it will contain the span sent by loadgen. You will also see an empty gateway-traces-security.out file, as the routing configuration creates output files immediately, even if no matching spans have been processed yet.

Send a Security Span: In the Spans terminal window send a security span using the security flag:

../loadgen -security -count 1

Again, both the agent and gateway should display debug information, including the span you just sent. This time, the gateway will write a line to the gateway-traces-security.out file, which is designated for spans where the deployment.environment resource attribute matches "security-applications". The gateway-traces-standard.out should be unchanged.

jq -c '.resourceSpans[] as $resource | $resource.scopeSpans[].spans[] | {spanId: .spanId, deploymentEnvironment: ($resource.resource.attributes[] | select(.key == "deployment.environment") | .value.stringValue)}' gateway-traces-security.out

{"spanId":"cb799e92e26d5782","deploymentEnvironment":"security-applications"}

You can repeat this scenario multiple times, and each trace will be written to its corresponding output file.

Important

Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.

Conclusion

In this section, we successfully tested the routing connector in the gateway by sending different spans and verifying their destinations.

Regular spans were correctly routed to gateway-traces-standard.out, confirming that spans without a matching deployment.environment attribute follow the default pipeline.
Security-related spans were routed to gateway-traces-security.out, demonstrating that the routing rule based on "deployment.environment": "security-applications" works as expected.

By inspecting the output files, we confirmed that the OpenTelemetry Collector correctly evaluates span attributes and routes them to the appropriate destinations. This validates that routing rules can effectively separate and direct telemetry data for different use cases.

You can now extend this approach by defining additional routing rules to further categorize spans, metrics, and logs based on different attributes.

Create metrics with Count Connector

10 minutes

In this section, we’ll explore how to use the Count Connector to extract attribute values from logs and convert them into meaningful metrics.

Specifically, we’ll use the Count Connector to track the number of “Star Wars” and “Lord of the Rings” quotes appearing in our logs, turning them into measurable data points.

Exercise

Inside the [WORKSHOP] directory, create a new subdirectory named 9-sum-count.
Next, copy all contents from the 8-routing-data directory into 9-sum-count.
After copying, remove any *.out and *.log files.
Change all terminal windows to the [WORKSHOP]/9-sum-count directory.

.
├── agent.yaml
└── gateway.yaml

Update the agent.yaml to change the frequency that we read logs. Find the filelog/quotes receiver in the agent.yaml and add a poll_interval attribute:

  filelog/quotes:                      # Receiver Type/Name
    poll_interval: 10s                 # Only read every ten seconds

The reason for the delay is that the Count Connector in the OpenTelemetry Collector counts logs only within each processing interval. This means that every time the data is read, the count resets to zero for the next interval. With the default Filelog reciever interval of 200ms it reads every line the loadgen writes, giving us counts of 1. With this interval we make sure we have multiple entries to count.

The Collector can maintain a running count for each read interval by omitting conditions, as shown below. However, it’s best practice to let your backend handle running counts since it can track them over a longer time period.

Exercise

Add the Count Connector

Include the Count Connector in the connector’s section of your configuration and define the metrics counters we want to use:

connectors:
  count:
    logs:
      logs.full.count:
        description: "Running count of all logs read in interval"
      logs.sw.count:
        description: "StarWarsCount"
        conditions:
        - attributes["movie"] == "SW"
      logs.lotr.count:
        description: "LOTRCount"
        conditions:
        - attributes["movie"] == "LOTR"
      logs.error.count:
        description: "ErrorCount"
        conditions:
        - attributes["level"] == "ERROR"

Explanation of the Metrics Counters
- logs.full.count: Tracks the total number of logs processed during each read interval.
  Since this metric has no filtering conditions, every log that passes through the system is included in the count.
- logs.sw.count Counts logs that contain a quote from a Star Wars movie.
- logs.lotr.count: Counts logs that contain a quote from a Lord of the Rings movie.
- logs.error.count: Represents a real-world scenario by counting logs with a severity level of ERROR for the read interval.
Configure the Count Connector in the pipelines
In the pipeline configuration below, the connector exporter is added to the logs section, while the connector receiver is added to the metrics section.

  pipelines:
    traces:
      receivers:
      - otlp
      processors:
      - memory_limiter
      - attributes/update              # Update, hash, and remove attributes
      - redaction/redact               # Redact sensitive fields using regex
      - resourcedetection
      - resource/add_mode
      - batch
      exporters:
      - debug
      - file
      - otlphttp
    metrics:
      receivers:
      - count                          # Count Connector - Receiver
      - otlp
      #- hostmetrics                    # Host Metrics Receiver
      processors:
      - memory_limiter
      - resourcedetection
      - resource/add_mode
      - batch
      exporters:
      - debug
      - otlphttp
    logs:
      receivers:
      - otlp
      - filelog/quotes
      processors:
      - memory_limiter
      - resourcedetection
      - resource/add_mode
      - transform/logs                 # Transform logs processor
      - batch
      exporters:
      - count                          # Count Connector - Exporter
      - debug
      - otlphttp

We count logs based on their attributes. If your log data is stored in the log body instead of attributes, you’ll need to use a Transform processor in your pipeline to extract key/value pairs and add them as attributes.

In this workshop, we’ve already added merge_maps(attributes, cache, "upsert") in the 07-transform section. This ensures that all relevant data is included in the log attributes for processing.

When selecting fields to create attributes from, be mindful—adding all fields indiscriminately is generally not ideal for production environments. Instead, choose only the fields that are truly necessary to avoid unnecessary data clutter.

Exercise

Validate the agent configuration using otelbin.io. For reference, the logs and metrics: sections of your pipelines will look like this:

%%{init:{"fontFamily":"monospace"}}%%
graph LR
    %% Nodes
    REC1(otlp<br>fa:fa-download):::receiver
    REC2(filelog<br>fa:fa-download<br>quotes):::receiver
    REC3(otlp<br>fa:fa-download):::receiver
    PRO1(memory_limiter<br>fa:fa-microchip):::processor
    PRO2(memory_limiter<br>fa:fa-microchip):::processor
    PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
    PRO4(resource<br>fa:fa-microchip<br>add_mode):::processor
    PRO5(batch<br>fa:fa-microchip):::processor
    PRO6(batch<br>fa:fa-microchip):::processor
    PRO7(resourcedetection<br>fa:fa-microchip):::processor
    PRO8(resourcedetection<br>fa:fa-microchip):::processor
    PRO9(transfrom<br>fa:fa-microchip<br>logs):::processor
    EXP1(&nbsp;&ensp;debug&nbsp;&ensp;<br>fa:fa-upload):::exporter
    EXP2(&emsp;&emsp;otlphttp&emsp;&emsp;<br>fa:fa-upload):::exporter
    EXP3(&nbsp;&ensp;debug&nbsp;&ensp;<br>fa:fa-upload):::exporter
    EXP4(&emsp;&emsp;otlphttp&emsp;&emsp;<br>fa:fa-upload):::exporter
    ROUTE1(&nbsp;count&nbsp;<br>fa:fa-route):::con-export
    ROUTE2(&nbsp;count&nbsp;<br>fa:fa-route):::con-receive

    %% Links
    subID1:::sub-logs
    subID2:::sub-metrics
    subgraph " " 
      direction LR
      subgraph subID1[**Logs**]
      direction LR
      REC1 --> PRO1
      REC2 --> PRO1
      PRO1 --> PRO7
      PRO7 --> PRO3
      PRO3 --> PRO9
      PRO9 --> PRO5
      PRO5 --> ROUTE1
      PRO5 --> EXP1
      PRO5 --> EXP2
      end
      
      subgraph subID2[**Metrics**]
      direction LR
      ROUTE1 --> ROUTE2       
      ROUTE2 --> PRO2
      REC3 --> PRO2
      PRO2 --> PRO8
      PRO8 --> PRO4
      PRO4 --> PRO6
      PRO6 --> EXP3
      PRO6 --> EXP4
      end
    end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-logs stroke:#34d399,stroke-width:1px, color:#34d399,stroke-dasharray: 3 3;
classDef sub-metrics stroke:#38bdf8,stroke-width:1px, color:#38bdf8,stroke-dasharray: 3 3;

9.1 Testing the Count Connector

Exercise

Start the Gateway:
In the Gateway terminal window navigate to the [WORKSHOP]/9-sum-count directory and run:

../otelcol --config=gateway.yaml

Start the Agent:
In the Agent terminal window navigate to the [WORKSHOP]/9-sum-count directory and run:

../otelcol --config=agent.yaml

Send 12 Logs lines with the Loadgen:
In the Spans terminal window navigate to the [WORKSHOP]/9-sum-count directory.
Send 12 log lines, they should be read in two intervals. Do this with the following loadgen command:

../loadgen -logs -json -count 12

Both the agent and gateway will display debug information, showing they are processing data. Wait until the loadgen completes.

Verify metrics have been generated
As the logs are processed, the Agent generates metrics and forwards them to the Gateway, which then writes them to gateway-metrics.out.

To check if the metrics logs.full.count, logs.sw.count, logs.lotr.count, and logs.error.count are present in the output, run the following jq query:

jq '.resourceMetrics[].scopeMetrics[].metrics[]
    | select(.name == "logs.full.count" or .name == "logs.sw.count" or .name == "logs.lotr.count" or .name == "logs.error.count")
    | {name: .name, value: (.sum.dataPoints[0].asInt // "-")}' gateway-metrics.out

{
  "name": "logs.sw.count",
  "value": "2"
}
{
  "name": "logs.lotr.count",
  "value": "2"
}
{
  "name": "logs.full.count",
  "value": "4"
}
{
  "name": "logs.error.count",
  "value": "2"
}
{
  "name": "logs.error.count",
  "value": "1"
}
{
  "name": "logs.sw.count",
  "value": "2"
}
{
  "name": "logs.lotr.count",
  "value": "6"
}
{
  "name": "logs.full.count",
  "value": "8"
}

Tip

Note: the logs.full.count normally is equal to logs.sw.count + logs.lotr.count, while the logs.error.count will be a random number.

Important

Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.

Create metrics with Sum Connector

10 minutes

In this section, we’ll explore how the Sum Connector can extract values from spans and convert them into metrics.

We’ll specifically use the credit card charges from our base spans and leverage the Sum Connector to retrieve the total charges as a metric.

The connector can be used to collect (sum) attribute values from spans, span events, metrics, data points, and log records. It captures each individual value, transforms it into a metric, and passes it along. However, it’s the backend’s job to use these metrics and their attributes for calculations and further processing.

Exercise

Switch to your Agent terminal window and open the agent.yaml file in your editor.

Add the Sum Connector
Include the Sum Connector in the connectors section of your configuration and define the metrics counters:

  sum:
    spans:
       user.card-charge:
        source_attribute: payment.amount
        conditions:
          - attributes["payment.amount"] != "NULL"
        attributes:
          - key: user.name

In the example above, we check for the payment.amount attribute in spans. If it has a valid value, the Sum connector generates a metric called user.card-charge and includes the user.name as an attribute. This enables the backend to track and display a user’s total charges over an extended period, such as a billing cycle.

In the pipeline configuration below, the connector exporter is added to the traces section, while the connector receiver is added to the metrics section.

Exercise

Configure the Count Connector in the pipelines

  pipelines:
    traces:
      receivers:
      - otlp
      processors:
      - memory_limiter
      - attributes/update              # Update, hash, and remove attributes
      - redaction/redact               # Redact sensitive fields using regex
      - resourcedetection
      - resource/add_mode
      - batch
      exporters:
      - debug
      - file
      - otlphttp
      - sum                            # Sum Connector   - Receiver
    metrics:
      receivers:
      - sum                            # Sum Connector   - 
      - count                          # Count Connector - Receiver
      - otlp
      #- hostmetrics                    # Host Metrics Receiver
      processors:
      - memory_limiter
      - resourcedetection
      - resource/add_mode
      - batch
      exporters:
      - debug
      - otlphttp
    logs:
      receivers:
      - otlp
      - filelog/quotes
      processors:
      - memory_limiter
      - resourcedetection
      - resource/add_mode
      - transform/logs                 # Transform logs processor
      - batch
      exporters:
      - count                          # Count Connector - Exporter
      - debug
      - otlphttp

Validate the agent configuration using otelbin.io. For reference, the traces and metrics: sections of your pipelines will look like this:

%%{init:{"fontFamily":"monospace"}}%%
graph LR
    %% Nodes
    REC1(otlp<br>fa:fa-download<br> ):::receiver
    REC3(otlp<br>fa:fa-download<br> ):::receiver
    PRO1(memory_limiter<br>fa:fa-microchip<br> ):::processor
    PRO2(memory_limiter<br>fa:fa-microchip<br> ):::processor
    PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
    PRO4(resource<br>fa:fa-microchip<br>add_mode):::processor
    PRO5(batch<br>fa:fa-microchip<br> ):::processor
    PRO6(batch<br>fa:fa-microchip<br> ):::processor
    PRO7(resourcedetection<br>fa:fa-microchip<br> ):::processor
    PRO8(resourcedetection<br>fa:fa-microchip<br>):::processor

    PROA(attributes<br>fa:fa-microchip<br>redact):::processor
    PROB(redaction<br>fa:fa-microchip<br>update):::processor
    EXP1(&nbsp;&ensp;debug&nbsp;&ensp;<br>fa:fa-upload<br> ):::exporter
    EXP2(&emsp;&emsp;file&emsp;&emsp;<br>fa:fa-upload<br> ):::exporter
    EXP3(&nbsp;&ensp;debug&nbsp;&ensp;<br>fa:fa-upload<br> ):::exporter
    EXP4(&emsp;&emsp;otlphttp&emsp;&emsp;<br>fa:fa-upload<br> ):::exporter
    EXP5(&emsp;&emsp;otlphttp&emsp;&emsp;<br>fa:fa-upload<br> ):::exporter
    ROUTE1(&nbsp;sum&nbsp;<br>fa:fa-route<br> ):::con-export
    ROUTE2(&nbsp;count&nbsp;<br>fa:fa-route<br> ):::con-receive
    ROUTE3(&nbsp;sum&nbsp;<br>fa:fa-route<br> ):::con-receive

    %% Links
    subID1:::sub-traces
    subID2:::sub-metrics
    subgraph " " 
      direction LR
      subgraph subID1[**Traces**]
      direction LR
      REC1 --> PRO1
      PRO1 --> PROA
      PROA --> PROB
      PROB --> PRO7
      PRO7 --> PRO3
      PRO3 --> PRO5 
      PRO5 --> EXP1
      PRO5 --> EXP2
      PRO5 --> EXP5
      PRO5 --> ROUTE1
      end
      
      subgraph subID2[**Metrics**]
      direction LR
      ROUTE1 --> ROUTE3
      ROUTE3 --> PRO2       
      ROUTE2 --> PRO2
      REC3 --> PRO2
      PRO2 --> PRO8
      PRO8 --> PRO4
      PRO4 --> PRO6
      PRO6 --> EXP3
      PRO6 --> EXP4
      end
    end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-logs stroke:#34d399,stroke-width:1px, color:#34d399,stroke-dasharray: 3 3;
classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3;
classDef sub-metrics stroke:#38bdf8,stroke-width:1px, color:#38bdf8,stroke-dasharray: 3 3;

9.3 Testing the Count Connector

Exercise

Start the Gateway:
In the Gateway terminal window navigate to the [WORKSHOP]/9-sum-count directory and run:

../otelcol --config=gateway.yaml

Start the Agent:
In the Agent terminal window navigate to the [WORKSHOP]/9-sum-count directory and run:

../otelcol --config=agent.yaml

Start the Loadgen:
In the Spans terminal window navigate to the [WORKSHOP]/9-sum-count directory. Send 8 spans with the following loadgen command:

../loadgen -count 8

Both the agent and gateway will display debug information, showing they are processing data. Wait until the loadgen completes.

Verify that metrics
While processing the span, the Agent, has generated metrics and passed them on to the Gateway. The Gateway has written them into gateway-metrics.out.

To confirm the presence of user.card-chargeand verify each one has an attribute user.name in the metrics output, run the following jq query:

jq -r '.resourceMetrics[].scopeMetrics[].metrics[] | select(.name == "user.card-charge") | .sum.dataPoints[] | "\(.attributes[] | select(.key == "user.name").value.stringValue)\t\(.asDouble)"' gateway-metrics.out | while IFS=$'\t' read -r name charge; do
    printf "%-20s %s\n" "$name" "$charge"
done

George Lucas         67.49
Frodo Baggins        87.14
Thorin Oakenshield   90.98
Luke Skywalker       51.37
Luke Skywalker       65.56
Thorin Oakenshield   67.5
Thorin Oakenshield   66.66
Peter Jackson        94.39

Important

Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.

Advanced Collector Configuration

Workshop Overview

Subsections of Advanced Collector Configuration

Pre-requisites

Prerequisites

1. Agent Configuration

Subsections of 1. Agent Setup

1.1 Validation & Load Generation

Load Generation Tool

1.2 Test Agent Configuration

1.3 File Exporter

Subsections of 1.3 File Exporter

1.3.1 Test File Exporter

2. Gateway Configuration

Subsections of 2. Gateway Setup

2.1 Start Gateway

2.2 Configure Agent

2.3 Send metrics from the Agent to the Gateway

2.4 Send traces from the Agent to the Gateway

3. FileLog Setup

Subsections of 3. FileLog Setup

3.1 Configuration

3.2 Test FileLog Receiver

4. Building In Resilience

Subsections of 4. Building Resilience

4.1 File Storage Configuration

4.2 Setup environment for Resilience Testing

4.3 Simulate Failure

4.4 Recovery

Conclusion

5. Dropping Spans

Subsections of 5. Dropping Spans

5.1 Configuration

5.2 Test Filter Processor

6. Redacting Sensitive Data

Subsections of 6. Sensitive Data

6.1 Configuration

6.2 Test Attribute Processor

6.3 Test Redaction Processor

7. Transform Data

Subsections of 7. Transform Data

7.1 Configuration

7.2 Setup Environment

7.3 Test Transform Processor

8. Routing Data

Subsections of 8. Routing Data

8.1 Configure the Routing Connector

8.2 Configuring the Pipelines

8.3 Test Routing Connector

Conclusion

Create metrics with Count Connector

Subsections of 9. Count & Sum Connector

9.1 Testing the Count Connector

Create metrics with Sum Connector

9.3 Testing the Count Connector

Wrap-up