1. Verify Agent Configuration

15 minutes  

Welcome! In this section, we’ll begin with a fully functional OpenTelemetry setup that includes both an Agent and a Gateway.

We’ll start by quickly reviewing their configuration files to get familiar with the overall structure and to highlight key sections that control the telemetry pipeline.

Tip

Throughout the workshop, you’ll work with multiple terminal windows. To keep things organized, give each terminal a unique name or color. This will help you easily recognize and switch between them during the exercises.

We will refer to these terminals as: Agent, Gateway, Loadgen, and Test.

Exercise

  1. Create your first terminal window and name it Agent. Navigate to the directory for the first exercise [WORKSHOP]/1-agent-gateway and verify that the required files have been generated:

    cd 1-agent-gateway
ls -l

  2. You should see the following files in the directory. If not, re-run the setup-workshop.sh script as described in the Pre-requisites section:

    ​
    .
├── agent.yaml
└── gateway.yaml

Understanding the Agent configuration

Let’s review the key components of the agent.yaml file used in this workshop. We’ve made some important additions to support metrics, traces, and logs.

Receivers

The receivers section defines how the Agent ingests telemetry data. In this setup, three types of receivers have been configured:

  • Host Metrics Receiver

    hostmetrics:                         # Host Metrics Receiver
  collection_interval: 3600s         # Collection Interval (1hr)
  scrapers:
    cpu:                             # CPU Scraper

    Collects CPU usage from the local system every hour. We’ll use this to generate example metric data.

  • OTLP Receiver (HTTP protocol)

    otlp:                                # OTLP Receiver
  protocols:
    http:                            # Configure HTTP protocol
      endpoint: "0.0.0.0:4318"       # Endpoint to bind to

    Enables the agent to receive metrics, traces, and logs over HTTP on port 4318. This is used to send data to the collector in future exercises.

  • FileLog Receiver

    filelog/quotes:                      # Receiver Type/Name
  include: ./quotes.log              # The file to read log data from
  include_file_path: true            # Include file path in the log data
  include_file_name: false           # Exclude file name from the log data
  resource:                          # Add custom resource attributes
    com.splunk.source: ./quotes.log  # Source of the log data
    com.splunk.sourcetype: quotes    # Source type of the log data

    Enables the agent to tail a local log file (quotes.log) and convert it to structured log events, enriched with metadata such as source and sourceType.

Exporters

  • Debug Exporter

      debug:                               # Exporter Type
    verbosity: detailed                # Enabled detailed debug output

  • OTLPHTTP Exporter

      otlphttp:                            # Exporter Type
    endpoint: "http://localhost:5318"  # Gateway OTLP endpoint

    The debug exporter sends data to the console for visibility and debugging during the workshop while the otlphttp exporter forwards all telemetry to the local Gateway instance.

    This dual-export strategy ensures you can see the raw data locally while also sending it downstream for further processing and export.

Last Modified Jul 9, 2025

Subsections of 1. Agent Configuration

1.1 Verify Gateway Configuration

The OpenTelemetry Gateway serves as a central hub for receiving, processing, and exporting telemetry data. It sits between your telemetry sources (such as applications and services) and your observability backends like Splunk Observability Cloud.

By centralizing telemetry traffic, the gateway enables advanced features such as data filtering, enrichment, transformation, and routing to one or more destinations. It helps reduce the burden on individual services by offloading telemetry processing and ensures consistent, standardized data across distributed systems.

This makes your observability pipeline easier to manage, scale, and analyze—especially in complex, multi-service environments.

Exercise

Open or create your second terminal window and name it Gateway. Navigate to the first exercise directory [WORKSHOP]/1-agent-gateway then check the contents of the gateway.yaml file.

This file outlines the core structure of the OpenTelemetry Collector as deployed in Gateway mode.

Understanding the Gateway Configuration

Let’s explore the gateway.yaml file that defines how the OpenTelemetry Collector is configured in Gateway mode during this workshop. This Gateway is responsible for receiving telemetry from the Agent, then processing and exporting it for inspection or forwarding.

  • OTLP Receiver (Custom Port)

    receivers:
  otlp:
    protocols:
      http:
        endpoint: "0.0.0.0:5318"

    The port 5318 matches the otlphttp exporter in the Agent configuration, ensuring that all telemetry data sent by the Agent is accepted by the Gateway.

Note

This separation of ports avoids conflicts and keeps responsibilities clear between agent and gateway roles.

  • File Exporters

    The Gateway uses three file exporters to output telemetry data to local files. These exporters are defined as:

    exporters:                        # List of exporters
  debug:                          # Debug exporter
    verbosity: detailed           # Enable detailed debug output
  file/traces:                    # Exporter Type/Name
    path: "./gateway-traces.out"  # Path for OTLP JSON output for traces
    append: false                 # Overwrite the file each time
  file/metrics:                   # Exporter Type/Name
    path: "./gateway-metrics.out" # Path for OTLP JSON output for metrics
    append: false                 # Overwrite the file each time
  file/logs:                      # Exporter Type/Name
    path: "./gateway-logs.out"    # Path for OTLP JSON output for logs
    append: false                 # Overwrite the file each time

    Each exporter writes a specific signal type to its corresponding file.

    These files are created once the gateway is started and will be populated with real telemetry as the agent sends data. You can monitor these files in real time to observe the flow of telemetry through your pipeline.

Last Modified Jul 9, 2025

1.2 Validate & Test Configuration

Now, we can start the Gateway and the Agent, which is configured to automatically send Host Metrics at startup. We do this to verify that data is properly routed from the Agent to the Gateway.

Exercise

Gateway: In the Gateway terminal window, run the following command to start the Gateway:

​
../otelcol --config=gateway.yaml

If everything is configured correctly, the collector will start and state Everything is ready. Begin running and processing data. in the output, similar to the following:

2025-06-09T09:22:11.944+0100    info    service@v0.126.0/service.go:289 Everything is ready. Begin running and processing data. {"resource": {}}

Once the Gateway is running, it will listen for incoming data on port 5318 and export the received data to the following files:

  • gateway-traces.out
  • gateway-metrics.out
  • gateway-logs.out

Start the Agent: In the Agent terminal window start the agent with the agent configuration:

​
../otelcol --config=agent.yaml

Verify CPU Metrics:

  1. Check that when the Agent starts, it immediately starts sending CPU metrics.
  2. Both the Agent and the Gateway will display this activity in their debug output. The output should resemble the following snippet:
<snip>
NumberDataPoints #31
Data point attributes:
     -> cpu: Str(cpu3)
     -> state: Str(wait)
StartTimestamp: 2025-07-07 16:49:42 +0000 UTC
Timestamp: 2025-07-09 09:36:21.190226459 +0000 UTC
Value: 77.380000
        {"resource": {}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics"}

At this stage, the Agent continues to collect CPU metrics once per hour or upon each restart and sends them to the gateway. The Gateway processes these metrics and exports them to a file named gateway-metrics.out. This file stores the exported metrics as part of the pipeline service.

Verify Data arrived at Gateway: To confirm that CPU metrics, specifically for cpu0, have successfully reached the gateway, we’ll inspect the gateway-metrics.out file using the jq command.

The following command filters and extracts the system.cpu.time metric, focusing on cpu0. It displays the metric’s state (e.g., user, system, idle, interrupt) along with the corresponding values.

Open or create your third terminal window and name it Tests. Run the command below in the Tests terminal to check the system.cpu.time metric:

​
jq '.resourceMetrics[].scopeMetrics[].metrics[] | select(.name == "system.cpu.time") | .sum.dataPoints[] | select(.attributes[0].value.stringValue == "cpu0") | {cpu: .attributes[0].value.stringValue, state: .attributes[1].value.stringValue, value: .asDouble}' gateway-metrics.out
{
  "cpu": "cpu0",
  "state": "user",
  "value": 123407.02
}
{
  "cpu": "cpu0",
  "state": "system",
  "value": 64866.6
}
{
  "cpu": "cpu0",
  "state": "idle",
  "value": 216427.87
}
{
  "cpu": "cpu0",
  "state": "interrupt",
  "value": 0
}
Important

Stop the Agent and the Gateway processes by pressing Ctrl-C in their respective terminals.