Learn the concepts of the OpenTelemetry Collector and how to use it to send data to Splunk Observability Cloud.
Practice setting up the OpenTelemetry Collector configuration from scratch and go though several advanced configuration scenarios's.
Organizations getting started with OpenTelemetry may begin by sending data directly to an observability backend. While this works well for initial testing, using the OpenTelemetry collector as part of your observability architecture provides numerous benefits and is recommended for any production deployment.
In this workshop, we will be focusing on using the OpenTelemetry collector and starting with the fundamentals of configuring the receivers, processors, and exporters ready to use with Splunk Observability Cloud. The journey will take attendees from novices to being able to start adding custom components to help solve for their business observability needs for their distributed platform.
Throughout the workshop there will be expandable Ninja Sections, these will be more hands on and go into further technical detail that you can explore within the workshop or in your own time.
Please note that the content in these sections may go out of date due to the frequent development being made to the OpenTelemetry project. Links will be provided in the event details are out of sync, please let us know if you spot something that needs updating.
This interactive workshop is for developers and system administrators who are interested in learning more about architecture and deployment of the OpenTelemetry Collector.
By the end of this talk, attendees will be able to:
%%{
init:{
"theme":"base",
"themeVariables": {
"primaryColor": "#ffffff",
"clusterBkg": "#eff2fb",
"defaultLinkColor": "#333333"
}
}
}%%
flowchart LR;
subgraph Collector
A[OTLP] --> M(Receivers)
B[JAEGER] --> M(Receivers)
C[Prometheus] --> M(Receivers)
end
subgraph Processors
M(Receivers) --> H(Filters, Attributes, etc)
E(Extensions)
end
subgraph Exporters
H(Filters, Attributes, etc) --> S(OTLP)
H(Filters, Attributes, etc) --> T(JAEGER)
H(Filters, Attributes, etc) --> U(Prometheus)
end
The first step in installing the Open Telemetry Collector is downloading it. For our lab, we will use the wget command to download the .deb package from the OpenTelemetry Github repository.
Obtain the .deb package for your platform from the OpenTelemetry Collector Contrib releases page
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.111.0/otelcol-contrib_0.111.0_linux_amd64.debInstall the .deb package using dpkg. Take a look at the dpkg Output tab below to see what the example output of a successful install will look like:
sudo dpkg -i otelcol-contrib_0.111.0_linux_amd64.debSelecting previously unselected package otelcol-contrib.
(Reading database ... 89232 files and directories currently installed.)
Preparing to unpack otelcol-contrib_0.111.0_linux_amd64.deb ...
Unpacking otelcol-contrib (0.111.0) ...
Setting up otelcol-contrib (0.111.0) ...
Created symlink /etc/systemd/system/multi-user.target.wants/otelcol-contrib.service → /lib/systemd/system/otelcol-contrib.service.The collector should now be running. We will verify this as root using systemctl command. To exit the status just press q.
sudo systemctl status otelcol-contrib● otelcol-contrib.service - OpenTelemetry Collector Contrib
Loaded: loaded (/lib/systemd/system/otelcol-contrib.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2024-10-07 10:27:49 BST; 52s ago
Main PID: 17113 (otelcol-contrib)
Tasks: 13 (limit: 19238)
Memory: 34.8M
CPU: 155ms
CGroup: /system.slice/otelcol-contrib.service
└─17113 /usr/bin/otelcol-contrib --config=/etc/otelcol-contrib/config.yaml
Oct 07 10:28:36 petclinic-rum-testing otelcol-contrib[17113]: Descriptor:
Oct 07 10:28:36 petclinic-rum-testing otelcol-contrib[17113]: -> Name: up
Oct 07 10:28:36 petclinic-rum-testing otelcol-contrib[17113]: -> Description: The scraping was successful
Oct 07 10:28:36 petclinic-rum-testing otelcol-contrib[17113]: -> Unit:
Oct 07 10:28:36 petclinic-rum-testing otelcol-contrib[17113]: -> DataType: Gauge
Oct 07 10:28:36 petclinic-rum-testing otelcol-contrib[17113]: NumberDataPoints #0
Oct 07 10:28:36 petclinic-rum-testing otelcol-contrib[17113]: StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Oct 07 10:28:36 petclinic-rum-testing otelcol-contrib[17113]: Timestamp: 2024-10-07 09:28:36.942 +0000 UTC
Oct 07 10:28:36 petclinic-rum-testing otelcol-contrib[17113]: Value: 1.000000
Oct 07 10:28:36 petclinic-rum-testing otelcol-contrib[17113]: {"kind": "exporter", "data_type": "metrics", "name": "debug"}Because we will be making multiple configuration file changes, setting environment variables and restarting the collector, we need to stop the collector service and disable it from starting on boot.
sudo systemctl stop otelcol-contrib && sudo systemctl disable otelcol-contribOpenTelemetry is configured through YAML files. These files have default configurations that we can modify to meet our needs. Let’s look at the default configuration that is supplied:
cat /etc/otelcol-contrib/config.yaml | |
Congratulations! You have successfully downloaded and installed the OpenTelemetry Collector. You are well on your way to becoming an OTel Ninja. But first let’s walk through configuration files and different distributions of the OpenTelemetry Collector.
Splunk does provide its own, fully supported, distribution of the OpenTelemetry Collector. This distribution is available to install from the Splunk GitHub Repository or via a wizard in Splunk Observability Cloud that will build out a simple installation script to copy and paste. This distribution includes many additional features and enhancements that are not available in the OpenTelemetry Collector Contrib distribution.
We will now walk through each section of the configuration file and modify it to send host metrics to Splunk Observability Cloud.
Now that we have the OpenTelemetry Collector installed, let’s take a look at extensions for the OpenTelemetry Collector. Extensions are optional and available primarily for tasks that do not involve processing telemetry data. Examples of extensions include health monitoring, service discovery, and data forwarding.
%%{
init:{
"theme": "base",
"themeVariables": {
"primaryColor": "#ffffff",
"clusterBkg": "#eff2fb",
"defaultLinkColor": "#333333"
}
}
}%%
flowchart LR;
style E fill:#e20082,stroke:#333,stroke-width:4px,color:#fff
subgraph Collector
A[OTLP] --> M(Receivers)
B[JAEGER] --> M(Receivers)
C[Prometheus] --> M(Receivers)
end
subgraph Processors
M(Receivers) --> H(Filters, Attributes, etc)
E(Extensions)
end
subgraph Exporters
H(Filters, Attributes, etc) --> S(OTLP)
H(Filters, Attributes, etc) --> T(JAEGER)
H(Filters, Attributes, etc) --> U(Prometheus)
end
Extensions are configured in the same config.yaml file that we referenced in the installation step. Let’s edit the config.yaml file and configure the extensions. Note that the pprof and zpages extensions are already configured in the default config.yaml file. For the purpose of this workshop, we will only be updating the health_check extension to expose the port on all network interfaces on which we can access the health of the collector.
sudo vi /etc/otelcol-contrib/config.yamlextensions:
health_check:
endpoint: 0.0.0.0:13133Start the collector:
otelcol-contrib --config=file:/etc/otelcol-contrib/config.yamlThis extension enables an HTTP URL that can be probed to check the status of the OpenTelemetry Collector. This extension can be used as a liveness and/or readiness probe on Kubernetes. To learn more about the curl command, check out the curl man page.
Open a new terminal session and SSH into your instance to run the following command:
curl http://localhost:13133{"status":"Server available","upSince":"2024-10-07T11:00:08.004685295+01:00","uptime":"12.56420005s"}Performance Profiler extension enables the golang net/http/pprof endpoint. This is typically used by developers to collect performance profiles and investigate issues with the service. We will not be covering this in this workshop.
zPages are an in-process alternative to external exporters. When included, they collect and aggregate tracing and metrics information in the background; this data is served on web pages when requested. zPages are an extremely useful diagnostic feature to ensure the collector is running as expected.
ServiceZ gives an overview of the collector services and quick access to the pipelinez, extensionz, and featurez zPages. The page also provides build and runtime information.
Example URL: http://localhost:55679/debug/servicez (change localhost to reflect your own environment).
PipelineZ provides insights on the running pipelines running in the collector. You can find information on type, if data is mutated, and you can also see information on the receivers, processors and exporters that are used for each pipeline.
Example URL: http://localhost:55679/debug/pipelinez (change localhost to reflect your own environment).
ExtensionZ shows the extensions that are active in the collector.
Example URL: http://localhost:55679/debug/extensionz (change localhost to reflect your own environment).
Now that we’ve covered extensions, let’s check our configuration changes.
Now that we have reviewed extensions, let’s dive into the data pipeline portion of the workshop. A pipeline defines a path the data follows in the Collector starting from reception, moving to further processing or modification, and finally exiting the Collector via exporters.
The data pipeline in the OpenTelemetry Collector is made up of receivers, processors, and exporters. We will first start with receivers.
Welcome to the receiver portion of the workshop! This is the starting point of the data pipeline of the OpenTelemetry Collector. Let’s dive in.
A receiver, which can be push or pull based, is how data gets into the Collector. Receivers may support one or more data sources. Generally, a receiver accepts data in a specified format, translates it into the internal format and passes it to processors and exporters defined in the applicable pipelines.
%%{
init:{
"theme":"base",
"themeVariables": {
"primaryColor": "#ffffff",
"clusterBkg": "#eff2fb",
"defaultLinkColor": "#333333"
}
}
}%%
flowchart LR;
style M fill:#e20082,stroke:#333,stroke-width:4px,color:#fff
subgraph Collector
A[OTLP] --> M(Receivers)
B[JAEGER] --> M(Receivers)
C[Prometheus] --> M(Receivers)
end
subgraph Processors
M(Receivers) --> H(Filters, Attributes, etc)
E(Extensions)
end
subgraph Exporters
H(Filters, Attributes, etc) --> S(OTLP)
H(Filters, Attributes, etc) --> T(JAEGER)
H(Filters, Attributes, etc) --> U(Prometheus)
end
The Host Metrics Receiver generates metrics about the host system scraped from various sources. This is intended to be used when the collector is deployed as an agent which is what we will be doing in this workshop.
Let’s update the /etc/otel-contrib/config.yaml file and configure the hostmetrics receiver. Insert the following YAML under the receivers section, taking care to indent by two spaces.
sudo vi /etc/otelcol-contrib/config.yamlreceivers:
hostmetrics:
collection_interval: 10s
scrapers:
# CPU utilization metrics
cpu:
# Disk I/O metrics
disk:
# File System utilization metrics
filesystem:
# Memory utilization metrics
memory:
# Network interface I/O metrics & TCP connection metrics
network:
# CPU load metrics
load:
# Paging/Swap space utilization and I/O metrics
paging:
# Process count metrics
processes:
# Per process CPU, Memory and Disk I/O metrics. Disabled by default.
# process:You will also notice another receiver called prometheus. Prometheus is an open-source toolkit used by the OpenTelemetry Collector. This receiver is used to scrape metrics from the OpenTelemetry Collector itself. These metrics can then be used to monitor the health of the collector.
Let’s modify the prometheus receiver to clearly show that it is for collecting metrics from the collector itself. By changing the name of the receiver from prometheus to prometheus/internal, it is now much clearer as to what that receiver is doing. Update the configuration file to look like this:
prometheus/internal:
config:
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['0.0.0.0:8888']The following screenshot shows an example dashboard of some of the metrics the Prometheus internal receiver collects from the OpenTelemetry Collector. Here, we can see accepted and sent spans, metrics and log records.
The following screenshot is an out-of-the-box (OOTB) dashboard from Splunk Observability Cloud that allows you to easily monitor your Splunk OpenTelemetry Collector install base.
You will notice in the default configuration there are other receivers: otlp, opencensus, jaeger and zipkin. These are used to receive telemetry data from other sources. We will not be covering these receivers in this workshop and they can be left as they are.
We’ve now covered receivers, so let’s now check our configuration changes.
Now that we have reviewed how data gets into the OpenTelemetry Collector through receivers, let’s now take a look at how the Collector processes the received data.
As the /etc/otelcol-contrib/config.yaml is not complete, please do not attempt to restart the collector at this point.
Processors are run on data between being received and being exported. Processors are optional though some are recommended. There are a large number of processors included in the OpenTelemetry contrib Collector.
%%{
init:{
"theme":"base",
"themeVariables": {
"primaryColor": "#ffffff",
"clusterBkg": "#eff2fb",
"defaultLinkColor": "#333333"
}
}
}%%
flowchart LR;
style Processors fill:#e20082,stroke:#333,stroke-width:4px,color:#fff
subgraph Collector
A[OTLP] --> M(Receivers)
B[JAEGER] --> M(Receivers)
C[Prometheus] --> M(Receivers)
end
subgraph Processors
M(Receivers) --> H(Filters, Attributes, etc)
E(Extensions)
end
subgraph Exporters
H(Filters, Attributes, etc) --> S(OTLP)
H(Filters, Attributes, etc) --> T(JAEGER)
H(Filters, Attributes, etc) --> U(Prometheus)
end
By default, only the batch processor is enabled. This processor is used to batch up data before it is exported. This is useful for reducing the number of network calls made to exporters. For this workshop, we will inherit the following defaults which are hard-coded into the Collector:
send_batch_size (default = 8192): Number of spans, metric data points, or log records after which a batch will be sent regardless of the timeout. send_batch_size acts as a trigger and does not affect the size of the batch. If you need to enforce batch size limits sent to the next component in the pipeline see send_batch_max_size.timeout (default = 200ms): Time duration after which a batch will be sent regardless of size. If set to zero, send_batch_size is ignored as data will be sent immediately, subject to only send_batch_max_size.send_batch_max_size (default = 0): The upper limit of the batch size. 0 means no upper limit on the batch size. This property ensures that larger batches are split into smaller units. It must be greater than or equal to send_batch_size.For more information on the Batch processor, see the Batch Processor documentation.
The resourcedetection processor can be used to detect resource information from the host and append or override the resource value in telemetry data with this information.
By default, the hostname is set to the FQDN if possible, otherwise, the hostname provided by the OS is used as a fallback. This logic can be changed from using using the hostname_sources configuration option. To avoid getting the FQDN and use the hostname provided by the OS, we will set the hostname_sources to os.
processors:
batch:
resourcedetection/system:
detectors: [system]
system:
hostname_sources: [os]If the workshop instance is running on an AWS/EC2 instance we can gather the following tags from the EC2 metadata API (this is not available on other platforms).
cloud.provider ("aws")cloud.platform ("aws_ec2")cloud.account.idcloud.regioncloud.availability_zonehost.idhost.image.idhost.namehost.typeWe will create another processor to append these tags to our metrics.
processors:
batch:
resourcedetection/system:
detectors: [system]
system:
hostname_sources: [os]
resourcedetection/ec2:
detectors: [ec2]The attributes processor modifies attributes of a span, log, or metric. This processor also supports the ability to filter and match input data to determine if they should be included or excluded for specified actions.
It takes a list of actions that are performed in the order specified in the config. The supported actions are:
insert: Inserts a new attribute in input data where the key does not already exist.update: Updates an attribute in input data where the key does exist.upsert: Performs insert or update. Inserts a new attribute in input data where the key does not already exist and updates an attribute in input data where the key does exist.delete: Deletes an attribute from the input data.hash: Hashes (SHA1) an existing attribute value.extract: Extracts values using a regular expression rule from the input key to target keys specified in the rule. If a target key already exists, it will be overridden.We are going to create an attributes processor to insert a new attribute to all our host metrics called participant.name with a value of your name e.g. marge_simpson.
Ensure you replace INSERT_YOUR_NAME_HERE with your name and also ensure you do not use spaces in your name.
Later on in the workshop, we will use this attribute to filter our metrics in Splunk Observability Cloud.
processors:
batch:
resourcedetection/system:
detectors: [system]
system:
hostname_sources: [os]
resourcedetection/ec2:
detectors: [ec2]
attributes/conf:
actions:
- key: participant.name
action: insert
value: "INSERT_YOUR_NAME_HERE"That’s processors covered, let’s check our configuration changes.
An exporter, which can be push or pull-based, is how you send data to one or more backends/destinations. Exporters may support one or more data sources.
For this workshop, we will be using the otlphttp exporter. The OpenTelemetry Protocol (OTLP) is a vendor-neutral, standardised protocol for transmitting telemetry data. The OTLP exporter sends data to a server that implements the OTLP protocol. The OTLP exporter supports both gRPC and HTTP/JSON protocols.
%%{
init:{
"theme":"base",
"themeVariables": {
"primaryColor": "#ffffff",
"clusterBkg": "#eff2fb",
"defaultLinkColor": "#333333"
}
}
}%%
flowchart LR;
style Exporters fill:#e20082,stroke:#333,stroke-width:4px,color:#fff
subgraph Collector
A[OTLP] --> M(Receivers)
B[JAEGER] --> M(Receivers)
C[Prometheus] --> M(Receivers)
end
subgraph Processors
M(Receivers) --> H(Filters, Attributes, etc)
E(Extensions)
end
subgraph Exporters
H(Filters, Attributes, etc) --> S(OTLP)
H(Filters, Attributes, etc) --> T(JAEGER)
H(Filters, Attributes, etc) --> U(Prometheus)
end
To send metrics over HTTP to Splunk Observability Cloud, we will need to configure the otlphttp exporter.
Let’s edit our /etc/otelcol-contrib/config.yaml file and configure the otlphttp exporter. Insert the following YAML under the exporters section, taking care to indent by two spaces e.g.
We will also change the verbosity of the logging exporter to prevent the disk from filling up. The default of detailed is very noisy.
exporters:
logging:
verbosity: normal
otlphttp/splunk:Next, we need to define the metrics_endpoint and configure the target URL.
If you are an attendee at a Splunk-hosted workshop, the instance you are using has already been configured with a Realm environment variable. We will reference that environment variable in our configuration file. Otherwise, you will need to create a new environment variable and set the Realm e.g.
export REALM="us1"The URL to use is https://ingest.${env:REALM}.signalfx.com/v2/datapoint/otlp. (Splunk has Realms in key geographical locations around the world for data residency).
The otlphttp exporter can also be configured to send traces and logs by defining a target URL for traces_endpoint and logs_endpoint respectively. Configuring these is outside the scope of this workshop.
exporters:
logging:
verbosity: normal
otlphttp/splunk:
metrics_endpoint: https://ingest.${env:REALM}.signalfx.com/v2/datapoint/otlpBy default, gzip compression is enabled for all endpoints. This can be disabled by setting compression: none in the exporter configuration. We will leave compression enabled for this workshop and accept the default as this is the most efficient way to send data.
To send metrics to Splunk Observability Cloud, we need to use an Access Token. This can be done by creating a new token in the Splunk Observability Cloud UI. For more information on how to create a token, see Create a token. The token needs to be of type INGEST.
If you are an attendee at a Splunk-hosted workshop, the instance you are using has already been configured with an Access Token (which has been set as an environment variable). We will reference that environment variable in our configuration file. Otherwise, you will need to create a new token and set it as an environment variable e.g.
export ACCESS_TOKEN=<replace-with-your-token>The token is defined in the configuration file by inserting X-SF-TOKEN: ${env:ACCESS_TOKEN} under a headers: section:
exporters:
logging:
verbosity: normal
otlphttp/splunk:
metrics_endpoint: https://ingest.${env:REALM}.signalfx.com/v2/datapoint/otlp
headers:
X-SF-TOKEN: ${env:ACCESS_TOKEN}Now that we’ve covered exporters, let’s check our configuration changes:
Of course, you can easily configure the metrics_endpoint to point to any other solution that supports the OTLP protocol.
Next, we need to enable the receivers, processors and exporters we have just configured in the service section of the config.yaml.
The Service section is used to configure what components are enabled in the Collector based on the configuration found in the receivers, processors, exporters, and extensions sections.
If a component is configured, but not defined within the Service section then it is not enabled.
The service section consists of three sub-sections:
In the default configuration, the extension section has been configured to enable health_check, pprof and zpages, which we configured in the Extensions module earlier.
service:
extensions: [health_check, pprof, zpages]So lets configure our Metric Pipeline!
If you recall from the Receivers portion of the workshop, we defined the Host Metrics Receiver to generate metrics about the host system, which are scraped from various sources. To enable the receiver, we must include the hostmetrics receiver in the metrics pipeline.
In the metrics pipeline, add hostmetrics to the metrics receivers section.
service:
pipelines:
traces:
receivers: [otlp, opencensus, jaeger, zipkin]
processors: [batch]
exporters: [debug]
metrics:
receivers: [hostmetrics, otlp, opencensus, prometheus]
processors: [batch]
exporters: [debug]Earlier in the workshop, we also renamed the prometheus receiver to reflect that is was collecting metrics internal to the collector, renaming it to prometheus/internal.
We now need to enable the prometheus/internal receiver under the metrics pipeline. Update the receivers section to include prometheus/internal under the metrics pipeline:
service:
pipelines:
traces:
receivers: [otlp, opencensus, jaeger, zipkin]
processors: [batch]
exporters: [debug]
metrics:
receivers: [hostmetrics, otlp, opencensus, prometheus/internal]
processors: [batch]
exporters: [debug]We also added resourcedetection/system and resourcedetection/ec2 processors so that the collector can capture the instance hostname and AWS/EC2 metadata. We now need to enable these two processors under the metrics pipeline.
Update the processors section to include resourcedetection/system and resourcedetection/ec2 under the metrics pipeline:
service:
pipelines:
traces:
receivers: [otlp, opencensus, jaeger, zipkin]
processors: [batch]
exporters: [debug]
metrics:
receivers: [hostmetrics, otlp, opencensus, prometheus/internal]
processors: [batch, resourcedetection/system, resourcedetection/ec2]
exporters: [debug]Also in the Processors section of this workshop, we added the attributes/conf processor so that the collector will insert a new attribute called participant.name to all the metrics. We now need to enable this under the metrics pipeline.
Update the processors section to include attributes/conf under the metrics pipeline:
service:
pipelines:
traces:
receivers: [otlp, opencensus, jaeger, zipkin]
processors: [batch]
exporters: [debug]
metrics:
receivers: [hostmetrics, otlp, opencensus, prometheus/internal]
processors: [batch, resourcedetection/system, resourcedetection/ec2, attributes/conf]
exporters: [debug]In the Exporters section of the workshop, we configured the otlphttp exporter to send metrics to Splunk Observability Cloud. We now need to enable this under the metrics pipeline.
Update the exporters section to include otlphttp/splunk under the metrics pipeline:
service:
pipelines:
traces:
receivers: [otlp, opencensus, jaeger, zipkin]
processors: [batch]
exporters: [debug]
metrics:
receivers: [hostmetrics, otlp, opencensus, prometheus/internal]
processors: [batch, resourcedetection/system, resourcedetection/ec2, attributes/conf]
exporters: [debug, otlphttp/splunk]It is recommended that you validate your configuration file before restarting the collector. You can do this by pasting the contents of your config.yaml file into otelbin.io.
Now that we have a working configuration, let’s start the collector and then check to see what zPages is reporting.
otelcol-contrib --config=file:/etc/otelcol-contrib/config.yamlOpen up zPages in your browser: http://localhost:55679/debug/pipelinez (change localhost to reflect your own environment).
Now that we have configured the OpenTelemetry Collector to send metrics to Splunk Observability Cloud, let’s take a look at the data in Splunk Observability Cloud. If you have not received an invite to Splunk Observability Cloud, your instructor will provide you with login credentials.
Before that, let’s make things a little more interesting and run a stress test on the instance. This in turn will light up the dashboards.
sudo apt install stress
while true; do stress -c 2 -t 40; stress -d 5 -t 40; stress -m 20 -t 40; doneOnce you are logged into Splunk Observability Cloud, using the left-hand navigation, navigate to Dashboards from the main menu. This will take you to the Teams view. At the top of this view click on All Dashboards :
In the search box, search for OTel Contrib:
If the dashboard does not exist, then your instructor will be able to quickly add it. If you are not attending a Splunk hosted version of this workshop then the Dashboard Group to import can be found at the bottom of this page.
Click on the OTel Contrib Dashboard dashboard to open it, next click in the Participant Name box, at the top of the dashboard, and select the name you configured for participant.name in the config.yaml in the drop-down list or start typing the name to search for it:
You can now see the host metrics for the host upon which you configured the OpenTelemetry Collector.
Building a component for the Open Telemetry Collector requires three key parts:
For this, we will use the example of building a component that works with Jenkins so that we can track important DevOps metrics of our project(s).
The metrics we are looking to measure are:
These indicators were identified Google’s DevOps Research and Assesment (DORA)[^1] team to help show performance of a software development team. The reason for choosing Jenkins CI is that we remain in the same Open Source Software ecosystem which we can serve as the example for the vendor managed CI tools to adopt in future.
There is something to consider when improving level of Observability within your organisation since there are some trade offs that get made.
| Pros | Cons | |
|---|---|---|
| (Auto) Instrumented | Does not require an external API to be monitored in order to observe the system. | Changing instrumentation requires changes to the project. |
| Gives system owners/developers to make changes in their observability. | Requires additional runtime dependancies. | |
| Understands system context and can corrolate captured data with Exemplars. | Can impact performance of the system. | |
| Component | - Changes to data names or semantics can be rolled out independently of the system’s release cycle. | Breaking API changes require a coordinated release between system and collector. |
| Updating/extending data collected is a seemless user facing change. | Captured data semantics can unexpectedly break that does not align with a new system release. | |
| Does not require the supporting teams to have a deep understanding of observability practice. | Strictly external / exposed information can be surfaced from the system. |
The time to finish this section of the workshop can vary depending on experience.
A complete solution can be found here in case you’re stuck or want to follow along with the instructor.
To get started developing the new Jenkins CI receiver, we first need to set up a Golang project. The steps to create your new Golang project is:
${HOME}/go/src/jenkinscireceiver and change into itgo mod init splunk.conf/workshop/example/jenkinscireceivergo.mod which is used to track our direct and indirect dependenciesgo.sum which is the checksum value of the dependencies being imported.The configuration portion of the component is how the user is able to have their inputs over the component, so the values that is used for the configuration need to be:
---
jenkins_server_addr: hostname
jenkins_server_api_port: 8089
interval: 10m
filter_builds_by:
- name: my-awesome-build
status: amber
track:
values:
example.metric.1: yes
example.metric.2: yes
example.metric.3: no
example.metric.4: no---
# Required Values
endpoint: http://my-jenkins-server:8089
auth:
authenticator: basicauth/jenkins
# Optional Values
collection_interval: 10m
metrics:
example.metric.1:
enabled: true
example.metric.2:
enabled: true
example.metric.3:
enabled: true
example.metric.4:
enabled: trueThe bad configuration highlights how doing the opposite of the recommendations of configuration practices impacts the usability of the component. It doesn’t make it clear what field values should be, it includes features that can be pushed to existing processors, and the field naming is not consistent with other components that exist in the collector.
The good configuration keeps the required values simple, reuses field names from other components, and ensures the component focuses on just the interaction between Jenkins and the collector.
The code tab shows how much is required to be added by us and what is already provided for us by shared libraries within the collector. These will be explained in more detail once we get to the business logic. The configuration should start off small and will change once the business logic has started to include additional features that is needed.
In order to implement the code needed for the configuration, we are going to create a new file named config.go with the following content:
package jenkinscireceiver
import (
"go.opentelemetry.io/collector/config/confighttp"
"go.opentelemetry.io/collector/receiver/scraperhelper"
"splunk.conf/workshop/example/jenkinscireceiver/internal/metadata"
)
type Config struct {
// HTTPClientSettings contains all the values
// that are commonly shared across all HTTP interactions
// performed by the collector.
confighttp.HTTPClientSettings `mapstructure:",squash"`
// ScraperControllerSettings will allow us to schedule
// how often to check for updates to builds.
scraperhelper.ScraperControllerSettings `mapstructure:",squash"`
// MetricsBuilderConfig contains all the metrics
// that can be configured.
metadata.MetricsBuilderConfig `mapstructure:",squash"`
}To recap the type of component we will need to capture metrics from Jenkins:
The business use case an extension helps solves for are:
See Extensions Overview for more details.
The business use case a receiver solves for:
This is commonly referred to pull vs push based data collection, and you read more about the details in the Receiver Overview.
The business use case a processor solves for is:
The thing to keep in mind is the data type flowing through a processor needs to forward the same data type to its downstream components. Read through Processor Overview for the details.
The business use case an exporter solves for:
The OpenTelemetry collector does not want to be “backend”, an all-in-one observability suite, but rather keep to the principles that founded OpenTelemetry to begin with; A vendor agnostic Observability for all. To help revisit the details, please read through Exporter Overview.
This is a component type that was missed in the workshop since it is a relatively new addition to the collector, but the best way to think about a connector is that it is like a processor that allows it to be used across different telemetry types and pipelines. Meaning that a connector can accept data as logs, and output metrics, or accept metrics from one pipeline and provide metrics on the data it has observed.
The business case that a connector solves for:
There was a brief overview within the Ninja section as part of the Processor Overview, and be sure what the project for updates for new connector components.
From the component overviews, it is clear that developing a pull-based receiver for Jenkins.
To help define and export the metrics captured by our receiver, we will be using, mdatagen, a tool developed for the collector that turns YAML defined metrics into code.
---
# Type defines the name to reference the component
# in the configuration file
type: jenkins
# Status defines the component type and the stability level
status:
class: receiver
stability:
development: [metrics]
# Attributes are the expected fields reported
# with the exported values.
attributes:
job.name:
description: The name of the associated Jenkins job
type: string
job.status:
description: Shows if the job had passed, or failed
type: string
enum:
- failed
- success
- unknown
# Metrics defines all the pontentially exported values from this receiver.
metrics:
jenkins.jobs.count:
enabled: true
description: Provides a count of the total number of configured jobs
unit: "{Count}"
gauge:
value_type: int
jenkins.job.duration:
enabled: true
description: Show the duration of the job
unit: "s"
gauge:
value_type: int
attributes:
- job.name
- job.status
jenkins.job.commit_delta:
enabled: true
description: The calculation difference of the time job was finished minus commit timestamp
unit: "s"
gauge:
value_type: int
attributes:
- job.name
- job.status// To generate the additional code needed to capture metrics,
// the following command to be run from the shell:
// go generate -x ./...
//go:generate go run github.com/open-telemetry/opentelemetry-collector-contrib/cmd/mdatagen@v0.80.0 metadata.yaml
package jenkinscireceiver
// There is no code defined within this file.Create these files within the project folder before continuing onto the next section.
The Factory is a software design pattern that effectively allows for an object, in this case a jenkinscireceiver, to be created dynamically with the provided configuration. To use a more real-world example, it would be going to a phone store, asking for a phone
that matches your exact description, and then providing it to you.
Run the following command go generate -x ./... , it will create a new folder, jenkinscireceiver/internal/metadata, that contains all code required to export the defined metrics. The required code is:
package jenkinscireceiver
import (
"errors"
"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/config/confighttp"
"go.opentelemetry.io/collector/receiver"
"go.opentelemetry.io/collector/receiver/scraperhelper"
"splunk.conf/workshop/example/jenkinscireceiver/internal/metadata"
)
func NewFactory() receiver.Factory {
return receiver.NewFactory(
metadata.Type,
newDefaultConfig,
receiver.WithMetrics(newMetricsReceiver, metadata.MetricsStability),
)
}
func newMetricsReceiver(_ context.Context, set receiver.CreateSettings, cfg component.Config, consumer consumer.Metrics) (receiver.Metrics, error) {
// Convert the configuration into the expected type
conf, ok := cfg.(*Config)
if !ok {
return nil, errors.New("can not convert config")
}
sc, err := newScraper(conf, set)
if err != nil {
return nil, err
}
return scraperhelper.NewScraperControllerReceiver(
&conf.ScraperControllerSettings,
set,
consumer,
scraperhelper.AddScraper(sc),
)
}package jenkinscireceiver
import (
"go.opentelemetry.io/collector/config/confighttp"
"go.opentelemetry.io/collector/receiver/scraperhelper"
"splunk.conf/workshop/example/jenkinscireceiver/internal/metadata"
)
type Config struct {
// HTTPClientSettings contains all the values
// that are commonly shared across all HTTP interactions
// performed by the collector.
confighttp.HTTPClientSettings `mapstructure:",squash"`
// ScraperControllerSettings will allow us to schedule
// how often to check for updates to builds.
scraperhelper.ScraperControllerSettings `mapstructure:",squash"`
// MetricsBuilderConfig contains all the metrics
// that can be configured.
metadata.MetricsBuilderConfig `mapstructure:",squash"`
}
func newDefaultConfig() component.Config {
return &Config{
ScraperControllerSettings: scraperhelper.NewDefaultScraperControllerSettings(metadata.Type),
HTTPClientSettings: confighttp.NewDefaultHTTPClientSettings(),
MetricsBuilderConfig: metadata.DefaultMetricsBuilderConfig(),
}
}package jenkinscireceiver
type scraper struct {}
func newScraper(cfg *Config, set receiver.CreateSettings) (scraperhelper.Scraper, error) {
// Create a our scraper with our values
s := scraper{
// To be filled in later
}
return scraperhelper.NewScraper(metadata.Type, s.scrape)
}
func (scraper) scrape(ctx context.Context) (pmetric.Metrics, error) {
// To be filled in
return pmetrics.NewMetrics(), nil
}---
dist:
name: otelcol
description: "Conf workshop collector"
output_path: ./dist
version: v0.0.0-experimental
extensions:
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/basicauthextension v0.80.0
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension v0.80.0
receivers:
- gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.80.0
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/jaegerreceiver v0.80.0
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver v0.80.0
- gomod: splunk.conf/workshop/example/jenkinscireceiver v0.0.0
path: ./jenkinscireceiver
processors:
- gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.80.0
exporters:
- gomod: go.opentelemetry.io/collector/exporter/loggingexporter v0.80.0
- gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.80.0
- gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v0.80.0
# This replace is a go directive that allows for redefine
# where to fetch the code to use since the default would be from a remote project.
replaces:
- splunk.conf/workshop/example/jenkinscireceiver => ./jenkinscireceiver├── build-config.yaml
└── jenkinscireceiver
├── go.mod
├── config.go
├── factory.go
├── scraper.go
└── internal
└── metadataOnce you have written these files into the project with the expected contents run, go mod tidy, which will fetch all the remote dependencies and update go.mod and generate the go.sum files.
At this point, we have a custom component that currently does nothing so we need to add in the required logic to capture this data from Jenkins.
From this point, the steps that we need to take are:
The changes will be made to scraper.go.
To be able to connect to the Jenkins server, we will be using the package, “github.com/yosida95/golang-jenkins”, which provides the functionality required to read data from the jenkins server.
Then we are going to utilise some of the helper functions from the, “go.opentelemetry.io/collector/receiver/scraperhelper” , library to create a start function so that we can connect to the Jenkins server once component has finished starting.
package jenkinscireceiver
import (
"context"
jenkins "github.com/yosida95/golang-jenkins"
"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/pdata/pmetric"
"go.opentelemetry.io/collector/receiver"
"go.opentelemetry.io/collector/receiver/scraperhelper"
"splunk.conf/workshop/example/jenkinscireceiver/internal/metadata"
)
type scraper struct {
mb *metadata.MetricsBuilder
client *jenkins.Jenkins
}
func newScraper(cfg *Config, set receiver.CreateSettings) (scraperhelper.Scraper, error) {
s := &scraper{
mb : metadata.NewMetricsBuilder(cfg.MetricsBuilderConfig, set),
}
return scraperhelper.NewScraper(
metadata.Type,
s.scrape,
scraperhelper.WithStart(func(ctx context.Context, h component.Host) error {
client, err := cfg.ToClient(h, set.TelemetrySettings)
if err != nil {
return err
}
// The collector provides a means of injecting authentication
// on our behalf, so this will ignore the libraries approach
// and use the configured http client with authentication.
s.client = jenkins.NewJenkins(nil, cfg.Endpoint)
s.client.SetHTTPClient(client)
return nil
}),
)
}
func (s scraper) scrape(ctx context.Context) (pmetric.Metrics, error) {
// To be filled in
return pmetric.NewMetrics(), nil
}This finishes all the setup code that is required in order to initialise a Jenkins receiver.
From this point on, we will be focuses on the scrape method that has been waiting to be filled in.
This method will be run on each interval that is configured within the configuration (by default, every minute).
The reason we want to capture the number of jobs configured so we can see the growth of our Jenkins server, and measure of many projects have onboarded. To do this we will call the jenkins client to list all jobs, and if it reports an error, return that with no metrics, otherwise, emit the data from the metric builder.
func (s scraper) scrape(ctx context.Context) (pmetric.Metrics, error) {
jobs, err := s.client.GetJobs()
if err != nil {
return pmetric.Metrics{}, err
}
// Recording the timestamp to ensure
// all captured data points within this scrape have the same value.
now := pcommon.NewTimestampFromTime(time.Now())
// Casting to an int64 to match the expected type
s.mb.RecordJenkinsJobsCountDataPoint(now, int64(len(jobs)))
// To be filled in
return s.mb.Emit(), nil
}In the last step, we were able to capture all jobs ands report the number of jobs there was. Within this step, we are going to examine each job and use the report values to capture metrics.
func (s scraper) scrape(ctx context.Context) (pmetric.Metrics, error) {
jobs, err := s.client.GetJobs()
if err != nil {
return pmetric.Metrics{}, err
}
// Recording the timestamp to ensure
// all captured data points within this scrape have the same value.
now := pcommon.NewTimestampFromTime(time.Now())
// Casting to an int64 to match the expected type
s.mb.RecordJenkinsJobsCountDataPoint(now, int64(len(jobs)))
for _, job := range jobs {
// Ensure we have valid results to start off with
var (
build = job.LastCompletedBuild
status = metadata.AttributeJobStatusUnknown
)
// This will check the result of the job, however,
// since the only defined attributes are
// `success`, `failure`, and `unknown`.
// it is assume that anything did not finish
// with a success or failure to be an unknown status.
switch build.Result {
case "aborted", "not_built", "unstable":
status = metadata.AttributeJobStatusUnknown
case "success":
status = metadata.AttributeJobStatusSuccess
case "failure":
status = metadata.AttributeJobStatusFailed
}
s.mb.RecordJenkinsJobDurationDataPoint(
now,
int64(job.LastCompletedBuild.Duration),
job.Name,
status,
)
}
return s.mb.Emit(), nil
}The final step is to calculate how long it took from commit to job completion to help infer our DORA metrics.
func (s scraper) scrape(ctx context.Context) (pmetric.Metrics, error) {
jobs, err := s.client.GetJobs()
if err != nil {
return pmetric.Metrics{}, err
}
// Recording the timestamp to ensure
// all captured data points within this scrape have the same value.
now := pcommon.NewTimestampFromTime(time.Now())
// Casting to an int64 to match the expected type
s.mb.RecordJenkinsJobsCountDataPoint(now, int64(len(jobs)))
for _, job := range jobs {
// Ensure we have valid results to start off with
var (
build = job.LastCompletedBuild
status = metadata.AttributeJobStatusUnknown
)
// Previous step here
// Ensure that the `ChangeSet` has values
// set so there is a valid value for us to reference
if len(build.ChangeSet.Items) == 0 {
continue
}
// Making the assumption that the first changeset
// item is the most recent change.
change := build.ChangeSet.Items[0]
// Record the difference from the build time
// compared against the change timestamp.
s.mb.RecordJenkinsJobCommitDeltaDataPoint(
now,
int64(build.Timestamp-change.Timestamp),
job.Name,
status,
)
}
return s.mb.Emit(), nil
}Once all of these steps have been completed, you now have built a custom Jenkins CI receiver!
There are more than likely features that would be desired from component that you can think of, like:
Please take this time to play around, break it, change things around, or even try to capture logs from the builds.
The goal of this workshop is to help you gain confidence in creating and modifying OpenTelemetry Collector configuration files. You’ll start with a minimal agent.yaml file and progressively build it out to handle several advanced, real-world scenarios.
A key focus of this workshop is learning how to configure the OpenTelemetry Collector to store telemetry data locally, rather than sending it to a third-party vendor backend. This approach not only simplifies debugging and troubleshooting but is also ideal for testing and development environments where you want to avoid sending data to production systems.
To make the most of this workshop, you should have:
Everything in this workshop is designed to run locally, ensuring a hands-on and accessible learning experience. Let’s dive in and start building!
During this workshop, we will cover the following topics:
By the end of this workshop, you’ll be familiar with configuring the OpenTelemetry Collector for a variety of real-world use cases.
vi, vim, nano, or your preferred text editor.jq is required - https://jqlang.org/download/Create a workshop directory: In your environment create a new directory (e.g., advanced-otel-workshop). We will refer to this directory as [WORKSHOP] for the remainder of the workshop.
Download workshop binaries: Change into your [WORKSHOP] directory and download the OpenTelemetry Collector and Load Generator binaries:
curl -L https://github.com/signalfx/splunk-otel-collector/releases/download/v0.120.0/otelcol_linux_amd64 -o otelcol && \
curl -L https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/loadgen/build/loadgen-linux-amd64 -o loadgenUpdate file permissions: Once downloaded, update the file permissions to make both executable:
chmod +x otelcol loadgen && \
./otelcol -v && \
./loadgen --helpcurl -L https://github.com/signalfx/splunk-otel-collector/releases/download/v0.120.0/otelcol_darwin_arm64 -o otelcol && \
curl -L https://github.com/splunk/observability-workshop/raw/refs/heads/main/workshop/ninja/advanced-otel/loadgen/build/loadgen-darwin-arm64 -o loadgenBefore running the binaries on macOS, you need to remove the quarantine attribute that macOS applies to downloaded files. This step ensures they can execute without restrictions.
Run the following command in your terminal:
xattr -dr com.apple.quarantine otelcol && \
xattr -dr com.apple.quarantine loadgenUpdate file permissions: Once downloaded, update the file permissions to make both executable:
chmod +x otelcol loadgen && \
./otelcol -v && \
./loadgen --help[WORKSHOP]
├── otelcol # OpenTelemetry Collector binary
└── loadgen # Load Generator binaryDuring this workshop, you will be using up to four terminal windows simultaneously. To stay organized, consider customizing each terminal or shell with unique names and colors. This will help you quickly identify and switch between them as needed.
We will refer to these terminals as: Agent, Gateway, Spans and Logs.
In the Agent terminal window, change into the [WORKSHOP] directory and create a new subdirectory named 1-agent.
mkdir 1-agent && \
cd 1-agentCreate a file named agent.yaml. This file will define the basic structure of an OpenTelemetry Collector configuration.
Copy and paste the following initial configuration into agent.yaml:
# Extensions
extensions:
health_check: # Health Check Extension
endpoint: 0.0.0.0:13133 # Health Check Endpoint
# Receivers
receivers:
hostmetrics: # Host Metrics Receiver
collection_interval: 3600s # Collection Interval (1hr)
scrapers:
cpu: # CPU Scraper
otlp: # OTLP Receiver
protocols:
http: # Configure HTTP protocol
endpoint: "0.0.0.0:4318" # Endpoint to bind to
# Exporters
exporters:
debug: # Debug Exporter
verbosity: detailed # Detailed verbosity level
# Processors
processors:
memory_limiter: # Limits memory usage
check_interval: 2s # Check interval
limit_mib: 512 # Memory limit in MiB
resourcedetection: # Resource Detection Processor
detectors: [system] # Detect system resources
override: true # Overwrites existing attributes
resource/add_mode: # Resource Processor
attributes:
- action: insert # Action to perform
key: otelcol.service.mode # Key name
value: "agent" # Key value
# Connectors
connectors:
# Service Section - Enabled Pipelines
service:
extensions:
- health_check # Health Check Extension
pipelines:
traces:
receivers:
- otlp # OTLP Receiver
processors:
- memory_limiter # Memory Limiter processor
- resourcedetection # Add system attributes to the data
- resource/add_mode # Add collector mode metadata
exporters:
- debug # Debug Exporter
metrics:
receivers:
- otlp
processors:
- memory_limiter
- resourcedetection
- resource/add_mode
exporters:
- debug
logs:
receivers:
- otlp
processors:
- memory_limiter
- resourcedetection
- resource/add_mode
exporters:
- debugYour directory structure should now look like this:
.
└── agent.yaml # OpenTelemetry Collector configuration fileIn this workshop, we’ll use https://otelbin.io to quickly validate YAML syntax and ensure your OpenTelemetry configurations are accurate. This step helps avoid errors before running tests during the session.
Here’s how to validate your configuration:
Open https://otelbin.io and replace the existing configuration by pasting your YAML into the left pane.
If are on a Mac and not using a Splunk Workshop instance, you can quickly copy the contents of the agent.yaml file to your clipboard by running the following command:
cat agent.yaml | pbcopyAt the top of the page, make sure Splunk OpenTelemetry Collector is selected as the validation target. If you don’t select this option, then you will see warnings in the UI stating Receiver "hostmetrics" is unused. (Line 8).
Once validated, refer to the image representation below to confirm your pipelines are set up correctly.
In most cases, we’ll display only the key pipeline. However, if all three pipelines (Traces, Metrics, and Logs) share the same structure, we’ll note this instead of showing each one individually.
%%{init:{"fontFamily":"monospace"}}%%
graph LR
%% Nodes
REC1( otlp <br>fa:fa-download):::receiver
PRO1(memory_limiter<br>fa:fa-microchip):::processor
PRO2(resourcedetection<br>fa:fa-microchip):::processor
PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
EXP1( debug <br>fa:fa-upload):::exporter
%% Links
subID1:::sub-traces
subgraph " "
subgraph subID1[**Traces/Metrics/Logs**]
direction LR
REC1 --> PRO1
PRO1 --> PRO2
PRO2 --> PRO3
PRO3 --> EXP1
end
end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-traces stroke:#fff,stroke-width:1px, color:#fff,stroke-dasharray: 3 3;For the is workshop we have specifically developed a loadgen tool. loadgen is a flexible load generator for simulating traces and logging activities. It supports base, health, and security traces by default, along with optional logging of random quotes to a file, either in plain text or JSON format.
The output generated by loadgen mimic those produced by an OpenTelemetry instrumentation library, allowing us to test the Collector’s processing logic and offers a simple yet powerful way to mimic real-world scenarios.
You’re ready to start the OpenTelemetry Collector with the newly created agent.yaml. This exercise sets the foundation for understanding how data flows through the OpenTelemetry Collector.
Start the Agent: In the Agent terminal window run the following command:
../otelcol --config=agent.yamlVerify debug output: If everything is configured correctly, the first and last lines of the output will look like:
2025/01/13T12:43:51 settings.go:478: Set config to [agent.yaml]
<snip to the end>
2025-01-13T12:43:51.747+0100 info service@v0.120.0/service.go:261 Everything is ready. Begin running and processing data.Send Test Span: Instead of instrumenting an application, we’ll simulate sending trace data to the OpenTelemetry Collector using the loadgen tool.
In the Spans terminal window, change into the 1-agent directory and run the following command to send a single span:
../loadgen -count 1Sending traces. Use Ctrl-C to stop.
Response: {"partialSuccess":{}}
Base trace sent with traceId: 1aacb1db8a6d510f10e52f154a7fdb90 and spanId: 7837a3a2d3635d9f{"partialSuccess":{}}: Indicates 100% success, as the partialSuccess field is empty. In case of a partial failure, this field will include details about any failed parts.
Verify Debug Output:
In the Agent terminal window check the collector’s debug output:
2025-03-06T10:11:35.174Z info Traces {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 1}
2025-03-06T10:11:35.174Z info ResourceSpans #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.6.1
Resource attributes:
-> service.name: Str(cinema-service)
-> deployment.environment: Str(production)
-> host.name: Str(workshop-instance)
-> os.type: Str(linux)
-> otelcol.service.mode: Str(agent)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope cinema.library 1.0.0
InstrumentationScope attributes:
-> fintest.scope.attribute: Str(Starwars, LOTR)
Span #0
Trace ID : 0ef4daa44a259a7199a948231bc383c0
Parent ID :
ID : e8fdd442c36cbfb1
Name : /movie-validator
Kind : Server
Start time : 2025-03-06 10:11:35.163557 +0000 UTC
End time : 2025-03-06 10:11:36.163557 +0000 UTC
Status code : Ok
Status message : Success
Attributes:
-> user.name: Str(George Lucas)
-> user.phone_number: Str(+1555-867-5309)
-> user.email: Str(george@deathstar.email)
-> user.password: Str(LOTR>StarWars1-2-3)
-> user.visa: Str(4111 1111 1111 1111)
-> user.amex: Str(3782 822463 10005)
-> user.mastercard: Str(5555 5555 5555 4444)
-> payment.amount: Double(86.48)
{"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces"}Stop the agent in the Agent terminal window using Ctrl-C.
To capture more than just debug output on the screen, we also want to generate output during the export phase of the pipeline. For this, we’ll add a File Exporter to write OTLP data to files for comparison.
The difference between the OpenTelemetry debug exporter and the file exporter lies in their purpose and output destination:
| Feature | Debug Exporter | File Exporter |
|---|---|---|
| Output Location | Console/Log | File on disk |
| Purpose | Real-time debugging | Persistent offline analysis |
| Best for | Quick inspection during testing | Temporary storage and sharing |
| Production Use | No | Rare, but possible |
| Persistence | No | Yes |
In summary, the Debug Exporter is great for real-time, in-development troubleshooting, while the File Exporter is better suited for storing telemetry data locally for later use.
In the Agent terminal window ensure the collector is not running then edit the agent.yaml and configure the File Exporter:
Configuring a file exporter: The File Exporter writes telemetry data to files on disk.
file: # File Exporter
path: "./agent.out" # Save path (OTLP/JSON)
append: false # Overwrite the file each timeUpdate the Pipelines Section: Add the file exporter to the traces pipeline only:
pipelines:
traces:
receivers:
- otlp # OTLP Receiver
processors:
- memory_limiter # Memory Limiter processor
- resourcedetection # Add system attributes to the data
- resource/add_mode # Add collector mode metadata
exporters:
- debug # Debug Exporter
- file # File Exporter
metrics:
receivers:
- otlp
processors:
- memory_limiter
- resourcedetection
- resource/add_mode
exporters:
- debug
logs:
receivers:
- otlp
processors:
- memory_limiter
- resourcedetection
- resource/add_mode
exporters:
- debugValidate the agent configuration using https://otelbin.io:
%%{init:{"fontFamily":"monospace"}}%%
graph LR
%% Nodes
REC1( otlp <br>fa:fa-download):::receiver
PRO1(memory_limiter<br>fa:fa-microchip):::processor
PRO2(resourcedetection<br>fa:fa-microchip):::processor
PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
EXP1( debug <br>fa:fa-upload):::exporter
EXP2( file <br>fa:fa-upload):::exporter
%% Links
subID1:::sub-traces
subgraph " "
subgraph subID1[**Traces**]
direction LR
REC1 --> PRO1
PRO1 --> PRO2
PRO2 --> PRO3
PRO3 --> EXP1
PRO3 --> EXP2
end
end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3;Restart your agent: Find your Agent terminal window, and (re)start the agent using the modified configuration:
../otelcol --config=agent.yaml2025-01-13T12:43:51.747+0100 info service@v0.120.0/service.go:261 Everything is ready. Begin running and processing data.Send a Trace: From the Spans terminal window send another span and verify you get the same output on the console as we saw previously:
../loadgen -count 1Verify that the agent.out file is written: Check that a file named agent.out is written in the current directory.
.
├── agent.out # OTLP/Json output created by the File Exporter
└── agent.yamlVerify the span format:
agent.out.agent.out, you can use the cat ./agent.out command. For a more readable formatted view, pipe the output to jq like this: cat ./agent.out | jq:{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"cinema-service"}},{"key":"deployment.environment","value":{"stringValue":"production"}},{"key":"host.name","value":{"stringValue":"workshop-instance"}},{"key":"os.type","value":{"stringValue":"linux"}},{"key":"otelcol.service.mode","value":{"stringValue":"agent"}}]},"scopeSpans":[{"scope":{"name":"cinema.library","version":"1.0.0","attributes":[{"key":"fintest.scope.attribute","value":{"stringValue":"Starwars, LOTR"}}]},"spans":[{"traceId":"d824a28db5aa5f5a3011f19c452e5af0","spanId":"ab4cde146f77eacf","parentSpanId":"","name":"/movie-validator","kind":2,"startTimeUnixNano":"1741256991405300000","endTimeUnixNano":"1741256992405300000","attributes":[{"key":"user.name","value":{"stringValue":"George Lucas"}},{"key":"user.phone_number","value":{"stringValue":"+1555-867-5309"}},{"key":"user.email","value":{"stringValue":"george@deathstar.email"}},{"key":"user.password","value":{"stringValue":"LOTR\u003eStarWars1-2-3"}},{"key":"user.visa","value":{"stringValue":"4111 1111 1111 1111"}},{"key":"user.amex","value":{"stringValue":"3782 822463 10005"}},{"key":"user.mastercard","value":{"stringValue":"5555 5555 5555 4444"}},{"key":"payment.amount","value":{"doubleValue":56.24}}],"status":{"message":"Success","code":1}}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.6.1"}]}{
"resourceSpans": [
{
"resource": {
"attributes": [
{
"key": "service.name",
"value": {
"stringValue": "cinema-service"
}
},
{
"key": "deployment.environment",
"value": {
"stringValue": "production"
}
},
{
"key": "host.name",
"value": {
"stringValue": "RCASTLEY-M-YQRY.local"
}
},
{
"key": "os.type",
"value": {
"stringValue": "darwin"
}
},
{
"key": "otelcol.service.mode",
"value": {
"stringValue": "agent"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "cinema.library",
"version": "1.0.0",
"attributes": [
{
"key": "fintest.scope.attribute",
"value": {
"stringValue": "Starwars, LOTR"
}
}
]
},
"spans": [
{
"traceId": "d824a28db5aa5f5a3011f19c452e5af0",
"spanId": "ab4cde146f77eacf",
"parentSpanId": "",
"name": "/movie-validator",
"kind": 2,
"startTimeUnixNano": "1741256991405300000",
"endTimeUnixNano": "1741256992405300000",
"attributes": [
{
"key": "user.name",
"value": {
"stringValue": "George Lucas"
}
},
{
"key": "user.phone_number",
"value": {
"stringValue": "+1555-867-5309"
}
},
{
"key": "user.email",
"value": {
"stringValue": "george@deathstar.email"
}
},
{
"key": "user.password",
"value": {
"stringValue": "LOTR>StarWars1-2-3"
}
},
{
"key": "user.visa",
"value": {
"stringValue": "4111 1111 1111 1111"
}
},
{
"key": "user.amex",
"value": {
"stringValue": "3782 822463 10005"
}
},
{
"key": "user.mastercard",
"value": {
"stringValue": "5555 5555 5555 4444"
}
},
{
"key": "payment.amount",
"value": {
"doubleValue": 56.24
}
}
],
"status": {
"message": "Success",
"code": 1
}
}
]
}
],
"schemaUrl": "https://opentelemetry.io/schemas/1.6.1"
}
]
}Stop the agent in the Agent terminal window using Ctrl-C.
The OpenTelemetry Gateway is designed to receive, process, and export telemetry data. It acts as an intermediary between telemetry sources (e.g. applications, services) and backends (e.g., observability platforms like Prometheus, Jaeger, or Splunk Observability Cloud).
The gateway is useful because it centralizes telemetry data collection, enabling features like data filtering, transformation, and routing to multiple destinations. It also reduces the load on individual services by offloading telemetry processing and ensures consistent data formats across distributed systems. This makes it easier to manage, scale, and analyze telemetry data in complex environments.
[WORKSHOP] directory and create a new subdirectory named 2-gateway.Change ALL terminal windows to the [WORKSHOP]/2-gateway directory.
agent.yaml from the 1-agent directory into 2-gateway.gateway.yaml and add the following initial configuration:########################### This section holds all the
## Configuration section ## configurations that can be
########################### used in this OpenTelemetry Collector
extensions: # List of extensions
health_check: # Health check extension
endpoint: 0.0.0.0:14133 # Custom port to avoid conflicts
receivers:
otlp: # OTLP receiver
protocols:
http: # HTTP protocol
endpoint: "0.0.0.0:5318" # Custom port to avoid conflicts
include_metadata: true # Required for token pass-through
exporters: # List of exporters
debug: # Debug exporter
verbosity: detailed # Enable detailed debug output
file/traces: # Exporter Type/Name
path: "./gateway-traces.out" # Path for OTLP JSON output
append: false # Overwrite the file each time
file/metrics: # Exporter Type/Name
path: "./gateway-metrics.out" # Path for OTLP JSON output
append: false # Overwrite the file each time
file/logs: # Exporter Type/Name
path: "./gateway-logs.out" # Path for OTLP JSON output
append: false # Overwrite the file each time
connectors:
processors: # List of processors
memory_limiter: # Limits memory usage
check_interval: 2s # Memory check interval
limit_mib: 512 # Memory limit in MiB
batch: # Batches data before exporting
metadata_keys: # Groups data by token
- X-SF-Token
resource/add_mode: # Adds metadata
attributes:
- action: upsert # Inserts or updates a key
key: otelcol.service.mode # Key name
value: "gateway" # Key value
###########################
### Activation Section ###
###########################
service: # Service configuration
telemetry:
metrics:
level: none # Disable metrics
extensions: [health_check] # Enabled extensions
pipelines: # Configured pipelines
traces: # Traces pipeline
receivers:
- otlp # OTLP receiver
processors: # Processors for traces
- memory_limiter
- resource/add_mode
- batch
exporters:
- debug # Debug exporter
- file/traces
metrics: # Metrics pipeline
receivers:
- otlp # OTLP receiver
processors: # Processors for metrics
- memory_limiter
- resource/add_mode
- batch
exporters:
- debug # Debug exporter
- file/metrics
logs: # Logs pipeline
receivers:
- otlp # OTLP receiver
processors: # Processors for logs
- memory_limiter
- resource/add_mode
- batch
exporters:
- debug # Debug exporter
- file/logsWhen the gateway is started it will generate three files: gateway-traces.out, gateway-metrics.out, and gateway-logs.out. These files will eventually contain the telemetry data received by the gateway.
.
├── agent.yaml
└── gateway.yamlThe configuration for the gateway does not need any additional configuration changes to function. This has been done to save time and focus on the core concepts of the Gateway.
Validate the gateway configuration using otelbin.io. For reference, the logs: section of your pipelines will look similar to this:
%%{init:{"fontFamily":"monospace"}}%%
graph LR
%% Nodes
REC1( otlp <br>fa:fa-download):::receiver
PRO1(memory_limiter<br>fa:fa-microchip):::processor
PRO2(resource<br>fa:fa-microchip<br>add_mode):::processor
PRO3(batch<br>fa:fa-microchip):::processor
EXP1( file <br>fa:fa-upload<br>logs):::exporter
EXP2( debug <br>fa:fa-upload):::exporter
%% Links
subID1:::sub-logs
subgraph " "
subgraph subID1[**Logs**]
direction LR
REC1 --> PRO1
PRO1 --> PRO2
PRO2 --> PRO3
PRO3 --> EXP2
PRO3 --> EXP1
end
end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-logs stroke:#34d399,stroke-width:1px, color:#34d399,stroke-dasharray: 3 3;Start the Gateway: In the Gateway terminal window, run the following command to start the gateway:
../otelcol --config=gateway.yamlIf everything is configured correctly, the first and last lines of the output should look like:
2025/01/15 15:33:53 settings.go:478: Set config to [gateway.yaml]
<snip to the end>
2025-01-13T12:43:51.747+0100 info service@v0.120.0/service.go:261 Everything is ready. Begin running and processing data.Next, we will configure the agent to send data to the newly created gateway.
Add the otlphttp exporter: The OTLP/HTTP Exporter is used to send data from the agent to the gateway using the OTLP/HTTP protocol.
gateway-logs.out, gateway-metrics.out, and gateway-traces.out are present in the directory.agent.yaml file in your editor.otlphttp exporter configuration to the exporters: section: otlphttp: # Exporter Type
endpoint: "http://localhost:5318" # Gateway OTLP endpointAdd a Batch Processor configuration: The Batch Processor will accept spans, metrics, or logs and place them into batches. Batching helps better compress the data and reduce the number of outgoing connections required to transmit the data. It is highly recommended configuring the batch processor on every collector.
batch processor configuration to the processors: section: batch: # Processor TypeUpdate the pipelines:
hostmetrics to the metrics pipeline. The HostMetrics Receiver will generate host CPU metrics once per hour with the current configuration.batch processor (after the resource/add_mode processor) to the traces, metrics, and logs pipelines.otlphttp exporter to the traces, metrics, and logs pipelines. pipelines:
traces:
receivers:
- otlp # OTLP Receiver
processors:
- memory_limiter # Memory Limiter processor
- resourcedetection # Add system attributes to the data
- resource/add_mode # Add collector mode metadata
- batch # Batch processor
exporters:
- debug # Debug Exporter
- file # File Exporter
- otlphttp # OTLP/HTTP Exporter
metrics:
receivers:
- otlp
- hostmetrics # Host Metrics Receiver
processors:
- memory_limiter
- resourcedetection
- resource/add_mode
- batch
exporters:
- debug
- otlphttp
logs:
receivers:
- otlp
processors:
- memory_limiter
- resourcedetection
- resource/add_mode
- batch
exporters:
- debug
- otlphttpValidate the agent configuration using otelbin.io. For reference, the metrics: section of your pipelines will look similar to this:
%%{init:{"fontFamily":"monospace"}}%%
graph LR
%% Nodes
REC1( otlp <br>fa:fa-download):::receiver
REC2(hostmetrics<br>fa:fa-download):::receiver
PRO1(memory_limiter<br>fa:fa-microchip):::processor
PRO2(resourcedetection<br>fa:fa-microchip):::processor
PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
PRO4(batch<br>fa:fa-microchip):::processor
EXP1(otlphttp<br>fa:fa-upload):::exporter
EXP2( debug <br>fa:fa-upload):::exporter
%% Links
subID1:::sub-metrics
subgraph " "
subgraph subID1[**Metrics**]
direction LR
REC1 --> PRO1
REC2 --> PRO1
PRO1 --> PRO2
PRO2 --> PRO3
PRO3 --> PRO4
PRO4 --> EXP2
PRO4 --> EXP1
end
end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-metrics stroke:#38bdf8,stroke-width:1px, color:#38bdf8,stroke-dasharray: 3 3;Start the Agent: In the Agent terminal window start the agent with the updated configuration:
../otelcol --config=agent.yamlVerify CPU Metrics:
agent starts, it immediately starts sending CPU metrics.agent and the gateway will display this activity in their debug output. The output should resemble the following snippet:<snip>
NumberDataPoints #37
Data point attributes:
-> cpu: Str(cpu0)
-> state: Str(system)
StartTimestamp: 2024-12-09 14:18:28 +0000 UTC
Timestamp: 2025-01-15 15:27:51.319526 +0000 UTC
Value: 9637.660000At this stage, the agent continues to collect CPU metrics once per hour or upon each restart and sends them to the gateway.
The gateway processes these metrics and exports them to a file named ./gateway-metrics.out. This file stores the exported metrics as part of the pipeline service.
Verify Data arrived at Gateway: To confirm that CPU metrics, specifically for cpu0, have successfully reached the gateway, we’ll inspect the gateway-metrics.out file using the jq command.
The following command filters and extracts the system.cpu.time metric, focusing on cpu0. It displays the metric’s state (e.g., user, system, idle, interrupt) along with the corresponding values.
Run the command below in the Tests terminal to check the system.cpu.time metric:
jq '.resourceMetrics[].scopeMetrics[].metrics[] | select(.name == "system.cpu.time") | .sum.dataPoints[] | select(.attributes[0].value.stringValue == "cpu0") | {cpu: .attributes[0].value.stringValue, state: .attributes[1].value.stringValue, value: .asDouble}' gateway-metrics.out{
"cpu": "cpu0",
"state": "user",
"value": 123407.02
}
{
"cpu": "cpu0",
"state": "system",
"value": 64866.6
}
{
"cpu": "cpu0",
"state": "idle",
"value": 216427.87
}
{
"cpu": "cpu0",
"state": "interrupt",
"value": 0
}Send a Test Trace:
agent and gateway are still running.agent and gateway debug logs:../loadgen -count 52025-03-06T11:49:00.456Z info Traces {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 1}
2025-03-06T11:49:00.456Z info ResourceSpans #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.6.1
Resource attributes:
-> service.name: Str(cinema-service)
-> deployment.environment: Str(production)
-> host.name: Str(workshop-instance)
-> os.type: Str(linux)
-> otelcol.service.mode: Str(agent)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope cinema.library 1.0.0
InstrumentationScope attributes:
-> fintest.scope.attribute: Str(Starwars, LOTR)
Span #0
Trace ID : 97fb4e5b13400b5689e3306da7cff077
Parent ID :
ID : 413358465e5b4f15
Name : /movie-validator
Kind : Server
Start time : 2025-03-06 11:49:00.431915 +0000 UTC
End time : 2025-03-06 11:49:01.431915 +0000 UTC
Status code : Ok
Status message : Success
Attributes:
-> user.name: Str(George Lucas)
-> user.phone_number: Str(+1555-867-5309)
-> user.email: Str(george@deathstar.email)
-> user.password: Str(LOTR>StarWars1-2-3)
-> user.visa: Str(4111 1111 1111 1111)
-> user.amex: Str(3782 822463 10005)
-> user.mastercard: Str(5555 5555 5555 4444)
-> payment.amount: Double(87.01)
{"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces"}Verify the Gateway has handled the spans: Once the gateway processes incoming spans, it writes the trace data to a file named gateway-traces.out. To confirm that the spans have been successfully handled, you can inspect this file.
Using the jq command, you can extract and display key details about each span, such as its spanId and its position in the file. Also, we can extract the attributes that the Hostmetrics Receiver added to the spans.
jq -c '.resourceSpans[] as $resource | $resource.scopeSpans[].spans[] | "Span \(input_line_number) found with spanId \(.spanId), hostname \($resource.resource.attributes[] | select(.key == "host.name") | .value.stringValue), os \($resource.resource.attributes[] | select(.key == "os.type") | .value.stringValue)"' ./gateway-traces.out"Span 1 found with spanId d71fe6316276f97d, hostname workshop-instance, os linux"
"Span 2 found with spanId e8d19489232f8c2a, hostname workshop-instance, os linux"
"Span 3 found with spanId 9dfaf22857a6bd05, hostname workshop-instance, os linux"
"Span 4 found with spanId c7f544a4b5fef5fc, hostname workshop-instance, os linux"
"Span 5 found with spanId 30bb49261315969d, hostname workshop-instance, os linux"Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.
The FileLog Receiver in the OpenTelemetry Collector is used to ingest logs from files.
It monitors specified files for new log entries and streams those logs into the Collector for further processing or exporting. It is also useful for testing and development purposes.
For this part of the workshop, the loadgen will generate logs using random quotes:
lotrQuotes := []string{
"One does not simply walk into Mordor.",
"Even the smallest person can change the course of the future.",
"All we have to decide is what to do with the time that is given us.",
"There is some good in this world, and it's worth fighting for.",
}
starWarsQuotes := []string{
"Do or do not, there is no try.",
"The Force will be with you. Always.",
"I find your lack of faith disturbing.",
"In my experience, there is no such thing as luck.",
}The FileLog receiver in the agent will read these log lines and send them to the gateway.
[WORKSHOP] directory and create a new subdirectory named 3-filelog.*.yaml from 2-gateway into 3-filelog.Change ALL terminal windows to the [WORKSHOP]/3-filelog directory.
Start the loadgen and this will begin writing lines to a file named quotes.log:
../loadgen -logsWriting logs to quotes.log. Press Ctrl+C to stop..
├── agent.yaml
├── gateway.yaml
└── quotes.yamlIn the Agent terminal window edit the agent.yaml and configure the FileLog receiver.
Configure FileLog Receiver: The filelog receiver reads log data from a file and includes custom resource attributes in the log data:
filelog/quotes: # Receiver Type/Name
include: ./quotes.log # The file to read log data from
include_file_path: true # Include file path in the log data
include_file_name: false # Exclude file name from the log data
resource: # Add custom resource attributes
com.splunk.source: ./quotes.log # Source of the log data
com.splunk.sourcetype: quotes # Source type of the log dataUpdate logs pipeline: Add thefilelog/quotes receiver to the logs pipeline only:
logs:
receivers:
- otlp
- filelog/quotes # Filelog Receiver
processors:
- memory_limiter
- resourcedetection
- resource/add_mode
- batch
exporters:
- debug
- otlphttpValidate Configuration: Paste the updated agent.yaml into otelbin.io. For reference, the logs: section of your pipelines will look similar to this:
%%{init:{"fontFamily":"monospace"}}%%
graph LR
%% Nodes
REC1( otlp <br>fa:fa-download):::receiver
REC2(filelog<br>fa:fa-download<br>quotes):::receiver
PRO1(memory_limiter<br>fa:fa-microchip):::processor
PRO2(resourcedetection<br>fa:fa-microchip):::processor
PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
PRO4(batch<br>fa:fa-microchip):::processor
EXP1( debug <br>fa:fa-upload):::exporter
EXP2(otlphttp<br>fa:fa-upload):::exporter
%% Links
subID1:::sub-logs
subgraph " "
subgraph subID1[**Logs**]
direction LR
REC1 --> PRO1
REC2 --> PRO1
PRO1 --> PRO2
PRO2 --> PRO3
PRO3 --> PRO4
PRO4 --> EXP1
PRO4 --> EXP2
end
end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-logs stroke:#34d399,stroke-width:1px, color:#34d399,stroke-dasharray: 3 3;Start the Gateway: In your Gateway terminal window start the gateway.
Start the Agent: In your Agent terminal window start the agent.
A continuous stream of log data from the quotes.log will be in the agent and gateway debug logs:
Timestamp: 1970-01-01 00:00:00 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Str(2025-03-06 15:18:32 [ERROR] - There is some good in this world, and it's worth fighting for. LOTR)
Attributes:
-> log.file.path: Str(quotes.log)
Trace ID:
Span ID:
Flags: 0
LogRecord #1Stop the loadgen: In the Logs terminal window, stop the loadgen using Ctrl-C.
Verify the gateway: Check if the gateway has written a ./gateway-logs.out file.
At this point, your directory structure will appear as follows:
.
├── agent.out
├── agent.yaml
├── gateway-logs.out # Output from the logs pipeline
├── gateway-metrics.out # Output from the metrics pipeline
├── gateway-traces.out # Output from the traces pipeline
├── gateway.yaml
└── quotes.log # File containing Random log linesExamine a log line: In gateway-logs.out compare a log line with the snippet below. Verify that the log entry includes the same attributes as we have seen in metrics and traces data previously:
{"resourceLogs":[{"resource":{"attributes":[{"key":"com.splunk.source","value":{"stringValue":"./quotes.log"}},{"key":"com.splunk.sourcetype","value":{"stringValue":"quotes"}},{"key":"host.name","value":{"stringValue":"workshop-instance"}},{"key":"os.type","value":{"stringValue":"linux"}},{"key":"otelcol.service.mode","value":{"stringValue":"gateway"}}]},"scopeLogs":[{"scope":{},"logRecords":[{"observedTimeUnixNano":"1741274312475540000","body":{"stringValue":"2025-03-06 15:18:32 [DEBUG] - All we have to decide is what to do with the time that is given us. LOTR"},"attributes":[{"key":"log.file.path","value":{"stringValue":"quotes.log"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1741274312475560000","body":{"stringValue":"2025-03-06 15:18:32 [DEBUG] - Your focus determines your reality. SW"},"attributes":[{"key":"log.file.path","value":{"stringValue":"quotes.log"}}],"traceId":"","spanId":""}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.6.1"}]}{
"resourceLogs": [
{
"resource": {
"attributes": [
{
"key": "com.splunk.source",
"value": {
"stringValue": "./quotes.log"
}
},
{
"key": "com.splunk.sourcetype",
"value": {
"stringValue": "quotes"
}
},
{
"key": "host.name",
"value": {
"stringValue": "workshop-instance"
}
},
{
"key": "os.type",
"value": {
"stringValue": "linux"
}
},
{
"key": "otelcol.service.mode",
"value": {
"stringValue": "gateway"
}
}
]
},
"scopeLogs": [
{
"scope": {},
"logRecords": [
{
"observedTimeUnixNano": "1741274312475540000",
"body": {
"stringValue": "2025-03-06 15:18:32 [DEBUG] - All we have to decide is what to do with the time that is given us. LOTR"
},
"attributes": [
{
"key": "log.file.path",
"value": {
"stringValue": "quotes.log"
}
}
],
"traceId": "",
"spanId": ""
},
{
"observedTimeUnixNano": "1741274312475560000",
"body": {
"stringValue": "2025-03-06 15:18:32 [DEBUG] - Your focus determines your reality. SW"
},
"attributes": [
{
"key": "log.file.path",
"value": {
"stringValue": "quotes.log"
}
}
],
"traceId": "",
"spanId": ""
}
]
}
],
"schemaUrl": "https://opentelemetry.io/schemas/1.6.1"
}
]
}You may also have noticed that every log line contains empty placeholders for "traceId":"" and "spanId":"".
The FileLog receiver will populate these fields only if they are not already present in the log line.
For example, if the log line is generated by an application instrumented with an OpenTelemetry instrumentation library, these fields will already be included and will not be overwritten.
Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.
The OpenTelemetry Collector’s FileStorage Extension enhances the resilience of your telemetry pipeline by providing reliable checkpointing, managing retries, and handling temporary failures effectively.
With this extension enabled, the OpenTelemetry Collector can store intermediate states on disk, preventing data loss during network disruptions and allowing it to resume operations seamlessly.
This solution will work for metrics as long as the connection downtime is brief—up to 15 minutes. If the downtime exceeds this, Splunk Observability Cloud will drop data due to datapoints being out of order.
For logs, there are plans to implement a more enterprise-ready solution in one of the upcoming Splunk OpenTelemetry Collector releases.
[WORKSHOP] directory, create a new subdirectory named 4-resilience.*.yaml from the 3-filelog directory into 4-resilience.Change ALL terminal windows to the [WORKSHOP]/4-resilience directory.
Your updated directory structure will now look like this:
.
├── agent.yaml
└── gateway.yamlIn this exercise, we will update the extensions: section of the agent.yaml file. This section is part of the OpenTelemetry configuration YAML and defines optional components that enhance or modify the OpenTelemetry Collector’s behavior.
While these components do not process telemetry data directly, they provide valuable capabilities and services to improve the Collector’s functionality.
Update the agent.yaml: In the Agent terminal window, add the file_storage extension and name it checkpoint:
file_storage/checkpoint: # Extension Type/Name
directory: "./checkpoint-dir" # Define directory
create_directory: true # Create directory
timeout: 1s # Timeout for file operations
compaction: # Compaction settings
on_start: true # Start compaction at Collector startup
# Define compaction directory
directory: "./checkpoint-dir/tmp"
max_transaction_size: 65536 # Max. size limit before compaction occursAdd file_storage to existing otlphttp exporter: Modify the otlphttp: exporter to configure retry and queuing mechanisms, ensuring data is retained and resent if failures occur:
otlphttp:
endpoint: "http://localhost:5318"
retry_on_failure:
enabled: true # Enable retry on failure
sending_queue: #
enabled: true # Enable sending queue
num_consumers: 10 # No. of consumers
queue_size: 10000 # Max. queue size
storage: file_storage/checkpoint # File storage extensionUpdate the services section: Add the file_storage/checkpoint extension to the existing extensions: section. This will cause the extension to be enabled:
service:
extensions:
- health_check
- file_storage/checkpoint # Enabled extensions for this collectorUpdate the metrics pipeline: For this exercise we are going to remove the hostmetrics receiver from the Metric pipeline to reduce debug and log noise:
metrics:
receivers:
- otlp
# - hostmetrics # Hostmetrics ReceiverValidate the agent configuration using otelbin.io. For reference, the metrics: section of your pipelines will look similar to this:
%%{init:{"fontFamily":"monospace"}}%%
graph LR
%% Nodes
REC1( otlp <br>fa:fa-download):::receiver
PRO1(memory_limiter<br>fa:fa-microchip):::processor
PRO2(resourcedetection<br>fa:fa-microchip):::processor
PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
PRO4(batch<br>fa:fa-microchip):::processor
EXP1( debug <br>fa:fa-upload):::exporter
EXP2(otlphttp<br>fa:fa-upload):::exporter
%% Links
subID1:::sub-metrics
subgraph " "
subgraph subID1[**Metrics**]
direction LR
REC1 --> PRO1
PRO1 --> PRO2
PRO2 --> PRO3
PRO3 --> PRO4
PRO4 --> EXP1
PRO4 --> EXP2
end
end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-metrics stroke:#38bdf8,stroke-width:1px, color:#38bdf8,stroke-dasharray: 3 3;Next, we will configure our environment to be ready for testing the File Storage configuration.
Start the Gateway: In the Gateway terminal window navigate to the [WORKSHOP]/4-resilience directory and run:
../otelcol --config=gateway.yamlStart the Agent: In the Agent terminal window navigate to the [WORKSHOP]/4-resilience directory and run:
../otelcol --config=agent.yamlSend five test spans: In the Spans terminal window navigate to the [WORKSHOP]/4-resilience directory and run:
../loadgen -count 5Both the agent and gateway should display debug logs, and the gateway should create a ./gateway-traces.out file.
If everything functions correctly, we can proceed with testing system resilience.
To assess the Agent’s resilience, we’ll simulate a temporary gateway outage and observe how the agent handles it:
Summary:
agent.agent to enter retry mode.agent will recover traces from its persistent queue and forward them successfully. Without the persistent queue, these traces would have been lost permanently.Simulate a network failure: In the Gateway terminal stop the gateway with Ctrl-C and wait until the gateway console shows that it has stopped:
2025-01-28T13:24:32.785+0100 info service@v0.120.0/service.go:309 Shutdown complete.Send traces: In the Spans terminal window send five more traces using the loadgen.
Notice that the agent’s retry mechanism is activated as it continuously attempts to resend the data. In the agent’s console output, you will see repeated messages similar to the following:
2025-01-28T14:22:47.020+0100 info internal/retry_sender.go:126 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "traces", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"http://localhost:5318/v1/traces\": dial tcp 127.0.0.1:5318: connect: connection refused", "interval": "9.471474933s"}Stop the Agent: In the Agent terminal window, use Ctrl-C to stop the agent. Wait until the agent’s console confirms it has stopped:
2025-01-28T14:40:28.702+0100 info extensions/extensions.go:66 Stopping extensions...
2025-01-28T14:40:28.702+0100 info service@v0.120.0/service.go:309 Shutdown complete.Stopping the agent will halt its retry attempts and prevent any future retry activity.
If the agent runs for too long without successfully delivering data, it may begin dropping traces, depending on the retry configuration, to conserve memory. By stopping the agent, any metrics, traces, or logs currently stored in memory are lost before being dropped, ensuring they remain available for recovery.
This step is essential for clearly observing the recovery process when the agent is restarted.
In this exercise, we’ll test how the OpenTelemetry Collector recovers from a network outage by restarting the Gateway. When the gateway becomes available again, the agent will resume sending data from its last checkpointed state, ensuring no data loss.
Restart the Gateway: In the Gateway terminal window run:
../otelcol --config=gateway.yamlRestart the Agent: In the Agent terminal window run:
../otelcol --config=agent.yamlAfter the agent is up and running, the File_Storage extension will detect buffered data in the checkpoint folder.
It will start to dequeue the stored spans from the last checkpoint folder, ensuring no data is lost.
Verify the Agent Debug output
Note that the Agent Debug Screen does NOT change and still shows the following line indicating no new data is being exported.
2025-02-07T13:40:12.195+0100 info service@v0.120.0/service.go:253 Everything is ready. Begin running and processing data.Watch the Gateway Debug output
You should see from the gateway debug screen, it has started receiving the previously missed traces without requiring any additional action on your part.
2025-02-07T12:44:32.651+0100 info service@v0.120.0/service.go:253 Everything is ready. Begin running and processing data.
2025-02-07T12:47:46.721+0100 info Traces {"kind": "exporter", "data_type": "traces", "name": "debug", "resource spans": 4, "spans": 4}
2025-02-07T12:47:46.721+0100 info ResourceSpans #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.6.1
Resource attributes:Check the gateway-traces.out file
Using jq, count the number of traces in the recreated gateway-traces.out. It should match the number you send when the gateway was down.
jq '.resourceSpans | length | "\(.) resourceSpans found"' gateway-traces.outStop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.
This exercise demonstrated how to enhance the resilience of the OpenTelemetry Collector by configuring the file_storage extension, enabling retry mechanisms for the otlp exporter, and using a file-backed queue for temporary data storage.
By implementing file-based checkpointing and queue persistence, you ensure the telemetry pipeline can gracefully recover from temporary interruptions, making it a more robust and reliable for production environments.
In this section, we will explore how to use the Filter Processor to selectively drop spans based on certain conditions.
Specifically, we will drop traces based on the span name, which is commonly used to filter out unwanted spans such as health checks or internal communication traces. In this case, we will be filtering out spans whose name is "/_healthz", typically associated with health check requests and usually are quite “noisy”.
[WORKSHOP] directory, create a new subdirectory named 5-dropping-spans.*.yaml from the 4-resilience directory into 5-dropping-spans.Change ALL terminal windows to the [WORKSHOP]/5-dropping-spans directory.
Your updated directory structure will now look like this:
.
├── agent.yaml
└── gateway.yamlNext, we will configure the filter processor and the respective pipelines.
Switch to your Gateway terminal window and open the gateway.yaml file. Update the processors section with the following configuration:
Add a filter processor:
Configure the gateway to exclude spans with the name /_healthz. The error_mode: ignore directive ensures that any errors encountered during filtering are ignored, allowing the pipeline to continue running smoothly. The traces section defines the filtering rules, specifically targeting spans named /_healthz for exclusion.
filter/health: # Defines a filter processor
error_mode: ignore # Ignore errors
traces: # Filtering rules for traces
span: # Exclude spans named "/_healthz"
- 'name == "/_healthz"'Add the filter processor to the traces pipeline:
Include the filter/health processor in the traces pipeline. For optimal performance, place the filter as early as possible—right after the memory_limiter and before the batch processor. Here’s how the configuration should look:
traces:
receivers:
- otlp
processors:
- memory_limiter
- filter/health # Filters data based on rules
- resource/add_mode
- batch
exporters:
- debug
- file/tracesThis setup ensures that health check-related spans (/_healthz) are filtered out early in the pipeline, reducing unnecessary noise in your telemetry data.
Validate the agent configuration using otelbin.io. For reference, the traces: section of your pipelines will look similar to this:
%%{init:{"fontFamily":"monospace"}}%%
graph LR
%% Nodes
REC1( otlp <br>fa:fa-download):::receiver
PRO1(memory_limiter<br>fa:fa-microchip):::processor
PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
PRO4(filter<br>fa:fa-microchip<br>health):::processor
PRO5(batch<br>fa:fa-microchip):::processor
EXP1( debug <br>fa:fa-upload):::exporter
EXP2(  file  <br>fa:fa-upload<br>traces):::exporter
%% Links
subID1:::sub-traces
subgraph " "
subgraph subID1[**Traces**]
direction LR
REC1 --> PRO1
PRO1 --> PRO4
PRO4 --> PRO3
PRO3 --> PRO5
PRO5 --> EXP1
PRO5 --> EXP2
end
end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3;To test your configuration, you’ll need to generate some trace data that includes a span named "/_healthz".
Start the Gateway: In your Gateway terminal window start the gateway.
./otelcol --config ./gateway.yamlStart the Agent: In your Agent terminal window start the agent.
./otelcol --config ./agent.yamlStart the Loadgen: In the Spans terminal window run the loadgen with the flag to also send healthz spans along with base spans:
../loadgen -health -count 5Verify agent.out: Using jq confirm the name of the spans received by the agent:
jq -c '.resourceSpans[].scopeSpans[].spans[] | "Span \(input_line_number) found with name \(.name)"' ./agent.out"Span 1 found with name /movie-validator"
"Span 2 found with name /_healthz"
"Span 3 found with name /movie-validator"
"Span 4 found with name /_healthz"
"Span 5 found with name /movie-validator"
"Span 6 found with name /_healthz"
"Span 7 found with name /movie-validator"
"Span 8 found with name /_healthz"
"Span 9 found with name /movie-validator"
"Span 10 found with name /_healthz"Check the Gateway Debug output: Using jq confirm the name of the spans received by the gateway:
jq -c '.resourceSpans[].scopeSpans[].spans[] | "Span \(input_line_number) found with name \(.name)"' ./gateway-traces.outThe gateway-metrics.out file will not contain any spans named /_healthz.
"Span 1 found with name /movie-validator"
"Span 2 found with name /movie-validator"
"Span 3 found with name /movie-validator"
"Span 4 found with name /movie-validator"
"Span 5 found with name /movie-validator"When using the Filter processor make sure you understand the look of your incoming data and test the configuration thoroughly. In general, use as specific a configuration as possible to lower the risk of the wrong data being dropped.
You can further extend this configuration to filter out spans based on different attributes, tags, or other criteria, making the OpenTelemetry Collector more customizable and efficient for your observability needs.
Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.
In this section, you’ll learn how to configure the OpenTelemetry Collector to remove specific tags and redact sensitive data from telemetry spans. This is crucial for protecting sensitive information such as credit card numbers, personal data, or other security-related details that must be anonymized before being processed or exported.
We’ll walk through configuring key processors in the OpenTelemetry Collector, including:
[WORKSHOP] directory, create a new subdirectory named 6-sensitive-data.*.yaml from the 5-dropping-spans directory into 6-sensitive-data.Change ALL terminal windows to the [WORKSHOP]/6-sensitive-data directory.
Your updated directory structure will now look like this:
.
├── agent.yaml
└── gateway.yamlIn this step, we’ll modify agent.yaml to include the attributes and redaction processors. These processors will help ensure that sensitive data within span attributes is properly handled before being logged or exported.
Previously, you may have noticed that some span attributes displayed in the console contained personal and sensitive data. We’ll now configure the necessary processors to filter out and redact this information effectively.
<snip>
Attributes:
-> user.name: Str(George Lucas)
-> user.phone_number: Str(+1555-867-5309)
-> user.email: Str(george@deathstar.email)
-> user.account_password: Str(LOTR>StarWars1-2-3)
-> user.visa: Str(4111 1111 1111 1111)
-> user.amex: Str(3782 822463 10005)
-> user.mastercard: Str(5555 5555 5555 4444)
{"kind": "exporter", "data_type": "traces", "name": "debug"}Switch to your Agent terminal window and open the agent.yaml file in your editor. We’ll add two processors to enhance the security and privacy of your telemetry data: the Attributes Processor and the Redaction Processor.
Add an attributes Processor: The Attributes Processor allows you to modify span attributes (tags) by updating, deleting, or hashing their values. This is particularly useful for obfuscating sensitive information before it is exported.
In this step, we’ll:
user.phone_number attribute to a static value ("UNKNOWN NUMBER").user.email attribute to ensure the original email is not exposed.user.password attribute to remove it entirely from the span. attributes/update:
actions: # Actions
- key: user.phone_number # Target key
action: update # Update action
value: "UNKNOWN NUMBER" # New value
- key: user.email # Target key
action: hash # Hash the email value
- key: user.password # Target key
action: delete # Delete the passwordAdd a redaction Processor: The The Redaction Processor detects and redacts sensitive data in span attributes based on predefined patterns, such as credit card numbers or other personally identifiable information (PII).
In this step:
We set allow_all_keys: true to ensure all attributes are processed (if set to false, only explicitly allowed keys are retained).
We define blocked_values with regular expressions to detect and redact Visa and MasterCard credit card numbers.
The summary: debug option logs detailed information about the redaction process for debugging purposes.
redaction/redact:
allow_all_keys: true # If false, only allowed keys will be retained
blocked_values: # List of regex patterns to block
- '\b4[0-9]{3}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b' # Visa
- '\b5[1-5][0-9]{2}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b' # MasterCard
summary: debug # Show debug details about redactionUpdate the traces Pipeline: Integrate both processors into the traces pipeline. Make sure that you comment out the redaction processor at first (we will enable it later in a separate exercise):
traces:
receivers:
- otlp
processors:
- memory_limiter
- attributes/update # Update, hash, and remove attributes
#- redaction/redact # Redact sensitive fields using regex
- resourcedetection
- resource/add_mode
- batch
exporters:
- debug
- file
- otlphttpValidate the agent configuration using otelbin.io. For reference, the traces: section of your pipelines will look similar to this:
%%{init:{"fontFamily":"monospace"}}%%
graph LR
%% Nodes
REC1( otlp <br>fa:fa-download):::receiver
PRO1(memory_limiter<br>fa:fa-microchip):::processor
PRO2(resourcedetection<br>fa:fa-microchip):::processor
PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
PRO5(batch<br>fa:fa-microchip):::processor
PRO6(attributes<br>fa:fa-microchip<br>update):::processor
EXP1(otlphttp<br>fa:fa-upload):::exporter
EXP2(  debug  <br>fa:fa-upload):::exporter
EXP3(file<br>fa:fa-upload):::exporter
%% Links
subID1:::sub-traces
subgraph " "
subgraph subID1[**Traces**]
direction LR
REC1 --> PRO1
PRO1 --> PRO6
PRO6 --> PRO2
PRO2 --> PRO3
PRO3 --> PRO5
PRO5 --> EXP2
PRO5 --> EXP3
PRO5 --> EXP1
end
end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3;In this exercise, we will delete the user.account_password, update the user.phone_number attribute and hash the user.email in the span data before it is exported by the agent.
Start the Gateway: In your Gateway terminal window start the gateway.
../otelcol --config=gateway.yamlStart the Agent: In your Agent terminal window start the agent.
../otelcol --config=agent.yamlStart the Load Generator: In the Spans terminal window start the loadgen:
../loadgen -count 1Check the debug output: For both the agent and gateway debug output, confirm that user.account_password has been removed, and both user.phone_number & user.email have been updated.
-> user.name: Str(George Lucas)
-> user.phone_number: Str(UNKNOWN NUMBER)
-> user.email: Str(62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287)
-> payment.amount: Double(51.71)
-> user.visa: Str(4111 1111 1111 1111)
-> user.amex: Str(3782 822463 10005)
-> user.mastercard: Str(5555 5555 5555 4444) -> user.name: Str(George Lucas)
-> user.phone_number: Str(+1555-867-5309)
-> user.email: Str(george@deathstar.email)
-> user.password: Str(LOTR>StarWars1-2-3)
-> user.visa: Str(4111 1111 1111 1111)
-> user.amex: Str(3782 822463 10005)
-> user.mastercard: Str(5555 5555 5555 4444)
-> payment.amount: Double(95.22)Check file output: Using jq validate that user.account_password has been removed, and user.phone_number & user.email have been updated in gateway-taces.out:
jq '.resourceSpans[].scopeSpans[].spans[].attributes[] | select(.key == "user.password" or .key == "user.phone_number" or .key == "user.email") | {key: .key, value: .value.stringValue}' ./gateway-traces.outNotice that the user.account_password has been removed, and the user.phone_number & user.email have been updated:
{
"key": "user.phone_number",
"value": "UNKNOWN NUMBER"
}
{
"key": "user.email",
"value": "62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287"
}Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.
The redaction processor gives precise control over which attributes and values are permitted or removed from telemetry data.
In this exercise, we will redact the Prepare the terminals: Delete the Start the Gateway: In your Gateway terminal window start the Enable the Start the Agent: In your Agent terminal window start the Start the Load Generator: In the Spans terminal window start the Check the debug output: For both the By including Check file output: Using Notice that These are just a few examples of how Stop the user.visa & user.mastercard values in the span data before it is exported by the agent.
Exercise
*.out files and clear the screen.gateway.../otelcol --config=gateway.yamlredaction/redact processor: In the Agent terminal window, edit agent.yaml and remove the # we inserted in the previous exercise. traces:
receivers:
- otlp
processors:
- memory_limiter
- attributes/update # Update, hash, and remove attributes
- redaction/redact # Redact sensitive fields using regex
- resourcedetection
- resource/add_mode
- batch
exporters:
- debug
- file
- otlphttpagent.../otelcol --config=agent.yamlloadgen:../loadgen -count 1agent and gateway confirm the values for user.visa & user.mastercard have been updated. Notice user.amex attribute value was NOT redacted because a matching regex pattern was not added to blocked_values
-> user.name: Str(George Lucas)
-> user.phone_number: Str(UNKNOWN NUMBER)
-> user.email: Str(62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287)
-> payment.amount: Double(69.71)
-> user.visa: Str(****)
-> user.amex: Str(3782 822463 10005)
-> user.mastercard: Str(****)
-> redaction.masked.keys: Str(user.mastercard,user.visa)
-> redaction.masked.count: Int(2) -> user.name: Str(George Lucas)
-> user.phone_number: Str(+1555-867-5309)
-> user.email: Str(george@deathstar.email)
-> user.password: Str(LOTR>StarWars1-2-3)
-> user.visa: Str(4111 1111 1111 1111)
-> user.amex: Str(3782 822463 10005)
-> user.mastercard: Str(5555 5555 5555 4444)
-> payment.amount: Double(65.54)
Note
summary:debug in the redaction processor, the debug output will include summary information about which matching keys values were redacted, along with the count of values that were masked. -> redaction.masked.keys: Str(user.mastercard,user.visa)
-> redaction.masked.count: Int(2)jq verify that user.visa & user.mastercard have been updated in the gateway-traces.out.
jq '.resourceSpans[].scopeSpans[].spans[].attributes[] | select(.key == "user.visa" or .key == "user.mastercard" or .key == "user.amex") | {key: .key, value: .value.stringValue}' ./gateway-traces.outuser.amex has not been redacted because a matching regex pattern was not added to blocked_values:{
"key": "user.visa",
"value": "****"
}
{
"key": "user.amex",
"value": "3782 822463 10005"
}
{
"key": "user.mastercard",
"value": "****"
}attributes and redaction processors can be configured to protect sensitive data.
Important
agent and the gateway processes by pressing Ctrl-C in their respective terminals.
The Transform Processor lets you modify telemetry data—logs, metrics, and traces—as it flows through the pipeline. Using the OpenTelemetry Transformation Language (OTTL), you can filter, enrich, and transform data on the fly without touching your application code.
In this exercise we’ll update agent.yaml to include a Transform Processor that will:
You may have noticed that in previous logs, fields like SeverityText and SeverityNumber were undefined (this is typical of the filelog receiver). However, the severity is embedded within the log body:
<snip>
SeverityText:
SeverityNumber: Unspecified(0)
Body: Str(2025-01-31 15:49:29 [WARN] - Do or do not, there is no try.)
</snip>Logs often contain structured data encoded as JSON within the log body. Extracting these fields into attributes allows for better indexing, filtering, and querying. Instead of manually parsing JSON in downstream systems, OTTL enables automatic transformation at the telemetry pipeline level.
[WORKSHOP] directory, create a new subdirectory named 7-transform-data.*.yaml from the 6-sensitve-data directory into 7-transform-data.Change ALL terminal windows to the [WORKSHOP]/7-transform-data directory.
Your updated directory structure will now look like this:
.
├── agent.yaml
└── gateway.yamlAdd a transform processor: Switch to your Agent terminal window and edit the agent.yaml and add the following transform processor:
transform/logs: # Processor Type/Name
log_statements: # Log Processing Statements
- context: resource # Log Context
statements: # List of attribute keys to keep
- keep_keys(attributes, ["com.splunk.sourcetype", "host.name", "otelcol.service.mode"])By using the -context: resource key we are targeting the resourceLog attributes of logs.
This configuration ensures that only the relevant resource attributes (com.splunk.sourcetype, host.name, otelcol.service.mode) are retained, improving log efficiency and reducing unnecessary metadata.
Adding a Context Block for Log Severity Mapping: To properly set the severity_text and severity_number fields of a log record, we add a log context block within log_statements. This configuration extracts the level value from the log body, maps it to severity_text, and assigns the corresponding severity_number based on the log level:
- context: log # Log Context
statements: # Transform Statements Array
- set(cache, ParseJSON(body)) where IsMatch(body, "^\\{") # Parse JSON log body into a cache object
- flatten(cache, "") # Flatten nested JSON structure
- merge_maps(attributes, cache, "upsert") # Merge cache into attributes, updating existing keys
- set(severity_text, attributes["level"]) # Set severity_text from the "level" attribute
- set(severity_number, 1) where severity_text == "TRACE" # Map severity_text to severity_number
- set(severity_number, 5) where severity_text == "DEBUG"
- set(severity_number, 9) where severity_text == "INFO"
- set(severity_number, 13) where severity_text == "WARN"
- set(severity_number, 17) where severity_text == "ERROR"
- set(severity_number, 21) where severity_text == "FATAL"The merge_maps function is used to combine two maps (dictionaries) into one. In this case, it merges the cache object (containing parsed JSON data from the log body) into the attributes map.
attributes: The target map where the data will be merged.cache: The source map containing the parsed JSON data."upsert": This mode ensures that if a key already exists in the attributes map, its value will be updated with the value from cache. If the key does not exist, it will be inserted.This step is crucial because it ensures that all relevant fields from the log body (e.g., level, message, etc.) are added to the attributes map, making them available for further processing or exporting.
Summary of Key Transformations:
You should have a single transform processor containing two context blocks: one for resource and one for log.
This configuration ensures that log severity is correctly extracted, standardized, and structured for efficient processing.
This method of mapping all JSON fields to top-level attributes should only be used for testing and debugging OTTL. It will result in high cardinality in a production scenario.
Update the logs pipeline: Add the transform/logs: processor into the logs: pipeline:
logs:
receivers:
- otlp
- filelog/quotes
processors:
- memory_limiter
- resourcedetection
- resource/add_mode
- transform/logs # Transform logs processor
- batch
exporters:
- debug
- otlphttpValidate the agent configuration using https://otelbin.io. For reference, the logs: section of your pipelines will look similar to this:
%%{init:{"fontFamily":"monospace"}}%%
graph LR
%% Nodes
REC1( otlp <br>fa:fa-download):::receiver
REC2(filelog<br>fa:fa-download<br>quotes):::receiver
PRO1(memory_limiter<br>fa:fa-microchip):::processor
PRO2(resourcedetection<br>fa:fa-microchip):::processor
PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
PRO4(transform<br>fa:fa-microchip<br>logs):::processor
PRO5(batch<br>fa:fa-microchip):::processor
EXP1(otlphttp<br>fa:fa-upload):::exporter
EXP2(  debug  <br>fa:fa-upload):::exporter
%% Links
subID1:::sub-logs
subgraph " "
subgraph subID1[**Logs**]
direction LR
REC1 --> PRO1
REC2 --> PRO1
PRO1 --> PRO2
PRO2 --> PRO3
PRO3 --> PRO4
PRO4 --> PRO5
PRO5 --> EXP2
PRO5 --> EXP1
end
end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-logs stroke:#34d399,stroke-width:1px, color:#34d399,stroke-dasharray: 3 3;Start the Gateway: In the Gateway terminal run:
../otelcol --config=gateway.yamlStart the Agent: In the Agent terminal run:
../otelcol --config=agent.yamlStart the Load Generator: Open the Logs terminal window and run the loadgen.
To ensure the logs are structured in JSON format, include the -json flag when starting the script.
../loadgen -logs -json -count 5The loadgen will write 5 log lines to ./quotes.log in JSON format.
This test verifies that the com.splunk/source and os.type metadata have been removed from the log resource attributes before being exported by the agent. Additionally, the test ensures that:
SeverityText and SeverityNumber are set on the LogRecord.attributes.This ensures proper metadata filtering, severity mapping, and structured log enrichment before export.
Check the debug output: For both the agent and gateway confirm that com.splunk/source and os.type have been removed:
Resource attributes:
-> com.splunk.sourcetype: Str(quotes)
-> host.name: Str(workshop-instance)
-> otelcol.service.mode: Str(agent)Resource attributes:
-> com.splunk.source: Str(./quotes.log)
-> com.splunk.sourcetype: Str(quotes)
-> host.name: Str(workshop-instance)
-> os.type: Str(linux)
-> otelcol.service.mode: Str(agent)For both the agent and gateway confirm that SeverityText and SeverityNumber in the LogRecord is now defined with the severity level from the log body. Confirm that the JSON fields from the body can be accessed as top-level log Attributes:
<snip>
SeverityText: WARN
SeverityNumber: Warn(13)
Body: Str({"level":"WARN","message":"Your focus determines your reality.","movie":"SW","timestamp":"2025-03-07 11:17:26"})
Attributes:
-> log.file.path: Str(quotes.log)
-> level: Str(WARN)
-> message: Str(Your focus determines your reality.)
-> movie: Str(SW)
-> timestamp: Str(2025-03-07 11:17:26)
</snip><snip>
SeverityText:
SeverityNumber: Unspecified(0)
Body: Str({"level":"WARN","message":"Your focus determines your reality.","movie":"SW","timestamp":"2025-03-07 11:17:26"})
Attributes:
-> log.file.path: Str(quotes.log)
</snip>Check file output: In the new gateway-logs.out file verify the data has been transformed:
jq '[.resourceLogs[].scopeLogs[].logRecords[] | {severityText, severityNumber, body: .body.stringValue}]' gateway-logs.out[
{
"severityText": "DEBUG",
"severityNumber": 5,
"body": "{\"level\":\"DEBUG\",\"message\":\"All we have to decide is what to do with the time that is given us.\",\"movie\":\"LOTR\",\"timestamp\":\"2025-03-07 11:56:29\"}"
},
{
"severityText": "WARN",
"severityNumber": 13,
"body": "{\"level\":\"WARN\",\"message\":\"The Force will be with you. Always.\",\"movie\":\"SW\",\"timestamp\":\"2025-03-07 11:56:29\"}"
},
{
"severityText": "ERROR",
"severityNumber": 17,
"body": "{\"level\":\"ERROR\",\"message\":\"One does not simply walk into Mordor.\",\"movie\":\"LOTR\",\"timestamp\":\"2025-03-07 11:56:29\"}"
},
{
"severityText": "DEBUG",
"severityNumber": 5,
"body": "{\"level\":\"DEBUG\",\"message\":\"Do or do not, there is no try.\",\"movie\":\"SW\",\"timestamp\":\"2025-03-07 11:56:29\"}"
}
]
[
{
"severityText": "ERROR",
"severityNumber": 17,
"body": "{\"level\":\"ERROR\",\"message\":\"There is some good in this world, and it's worth fighting for.\",\"movie\":\"LOTR\",\"timestamp\":\"2025-03-07 11:56:29\"}"
}
]Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.
The Routing Connector in OpenTelemetry is a powerful feature that allows you to direct data (traces, metrics, or logs) to different pipelines based on specific criteria. This is especially useful in scenarios where you want to apply different processing or exporting logic to subsets of your telemetry data.
For example, you might want to send production data to one exporter while directing test or development data to another. Similarly, you could route certain spans based on their attributes, such as service name, environment, or span name, to apply custom processing or storage logic.
[WORKSHOP] directory, create a new subdirectory named 8-routing-data.*.yaml from the 7-transform-data directory into 8-routing-data.[WORKSHOP]/8-routing-data directory.Your updated directory structure will now look like this:
.
├── agent.yaml
└── gateway.yamlNext, we will configure the routing connector and the respective pipelines.
In this exercise, you will configure the Routing Connector in the gateway.yaml file. This setup enables the gateway to route traces based on the deployment.environment attribute in the spans you send. By implementing this, you can process and handle traces differently depending on their attributes.
In OpenTelemetry configuration files, connectors have their own dedicated section, similar to receivers and processors.
Add the routing connector:
In the Gateway terminal window edit gateway.yaml and add the following below the connectors: section:
routing:
default_pipelines: [traces/standard] # Default pipeline if no rule matches
error_mode: ignore # Ignore errors in routing
table: # Define routing rules
# Routes spans to a target pipeline if the resourceSpan attribute matches the rule
- statement: route() where attributes["deployment.environment"] == "security-applications"
pipelines: [traces/security] # Target pipeline The rules above apply to traces, but this approach also applies to metrics and logs, allowing them to be routed based on attributes in resourceMetrics or resourceLogs.
Configure file: exporters: The routing connector requires separate targets for routing. Add two file exporters, file/traces/security and file/traces/standard, to ensure data is directed correctly.
file/traces/standard: # Exporter for regular traces
path: "./gateway-traces-standard.out" # Path for saving trace data
append: false # Overwrite the file each time
file/traces/security: # Exporter for security traces
path: "./gateway-traces-security.out" # Path for saving trace data
append: false # Overwrite the file each time With the routing configuration complete, the next step is to configure the pipelines to apply these routing rules.
Update the traces pipeline to use routing:
To enable routing, update the original traces: pipeline by using routing as the only exporter. This ensures all span data is sent through the routing connector for evaluation.
Remove all processors and replace it with an empty array ([]). These are now defined in the traces/standard and traces/security pipelines.
pipelines:
traces: # Original traces pipeline
receivers:
- otlp # OTLP Receiver
processors: []
exporters:
- routing # Routing ConnectorAdd both the standard and security traces pipelines:
Configure the Security pipeline: This pipeline will handle all spans that match the routing rule for security.
This uses routing as its receiver. Place it below the existing traces: pipeline:
traces/security: # New Security Traces/Spans Pipeline
receivers:
- routing # Receive data from the routing connector
processors:
- memory_limiter # Memory Limiter Processor
- resource/add_mode # Adds collector mode metadata
- batch
exporters:
- debug # Debug Exporter
- file/traces/security # File Exporter for spans matching ruleAdd the Standard pipeline: This pipeline processes all spans that do not match the routing rule.
This pipeline is also using routing as its receiver. Add this below the traces/security one:
traces/standard: # Default pipeline for unmatched spans
receivers:
- routing # Receive data from the routing connector
processors:
- memory_limiter # Memory Limiter Processor
- resource/add_mode # Adds collector mode metadata
- batch
exporters:
- debug # Debug exporter
- file/traces/standard # File exporter for unmatched spansValidate the agent configuration using otelbin.io. For reference, the traces: section of your pipelines will look similar to this:
%%{init:{"fontFamily":"monospace"}}%%
graph LR
%% Nodes
REC1( otlp <br>fa:fa-download):::receiver
PRO1(memory_limiter<br>fa:fa-microchip):::processor
PRO2(memory_limiter<br>fa:fa-microchip):::processor
PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
PRO4(resource<br>fa:fa-microchip<br>add_mode):::processor
PRO5(batch<br>fa:fa-microchip):::processor
PRO6(batch<br>fa:fa-microchip):::processor
EXP1(  debug  <br>fa:fa-upload):::exporter
EXP2(  file  <br>fa:fa-upload<br>traces):::exporter
EXP3(  debug  <br>fa:fa-upload):::exporter
EXP4(  file  <br>fa:fa-upload<br>traces):::exporter
ROUTE1( routing <br>fa:fa-route):::con-export
ROUTE2( routing <br>fa:fa-route):::con-receive
ROUTE3( routing <br>fa:fa-route):::con-receive
%% Links
subID1:::sub-traces
subID2:::sub-traces
subID3:::sub-traces
subgraph " "
direction LR
subgraph subID1[**Traces**]
REC1 --> ROUTE1
end
subgraph subID2[**Traces/standard**]
ROUTE1 --> ROUTE2
ROUTE2 --> PRO1
PRO1 --> PRO3
PRO3 --> PRO5
PRO5 --> EXP1
PRO5 --> EXP2
end
subgraph subID3[**Traces/security**]
ROUTE1 --> ROUTE3
ROUTE3 --> PRO2
PRO2 --> PRO4
PRO4 --> PRO6
PRO6 --> EXP3
PRO6 --> EXP4
end
end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3;In this section, we will test the routing rule configured for the Gateway. The expected result is that thespan generated by the loadgen will be sent to the gateway-traces-security.out file.
Start the Gateway: In your Gateway terminal window start the gateway.
../otelcol --config gateway.yamlStart the Agent: In your Agent terminal window start the agent.
../otelcol --config agent.yamlSend a Regular Span: In the Spans terminal window send a regular span using the loadgen:
../loadgen -count 1Both the agent and gateway will display debug information. The gateway will also generate a new gateway-traces-standard.out file, as this is now the designated destination for regular spans.
If you check gateway-traces-standard.out, it will contain the span sent by loadgen. You will also see an empty gateway-traces-security.out file, as the routing configuration creates output files immediately, even if no matching spans have been processed yet.
Send a Security Span: In the Spans terminal window send a security span using the security flag:
../loadgen -security -count 1Again, both the agent and gateway should display debug information, including the span you just sent. This time, the gateway will write a line to the gateway-traces-security.out file, which is designated for spans where the deployment.environment resource attribute matches "security-applications".
The gateway-traces-standard.out should be unchanged.
jq -c '.resourceSpans[] as $resource | $resource.scopeSpans[].spans[] | {spanId: .spanId, deploymentEnvironment: ($resource.resource.attributes[] | select(.key == "deployment.environment") | .value.stringValue)}' gateway-traces-security.out{"spanId":"cb799e92e26d5782","deploymentEnvironment":"security-applications"}You can repeat this scenario multiple times, and each trace will be written to its corresponding output file.
Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.
In this section, we successfully tested the routing connector in the gateway by sending different spans and verifying their destinations.
Regular spans were correctly routed to gateway-traces-standard.out, confirming that spans without a matching deployment.environment attribute follow the default pipeline.
Security-related spans were routed to gateway-traces-security.out, demonstrating that the routing rule based on "deployment.environment": "security-applications" works as expected.
By inspecting the output files, we confirmed that the OpenTelemetry Collector correctly evaluates span attributes and routes them to the appropriate destinations. This validates that routing rules can effectively separate and direct telemetry data for different use cases.
You can now extend this approach by defining additional routing rules to further categorize spans, metrics, and logs based on different attributes.
In this section, we’ll explore how to use the Count Connector to extract attribute values from logs and convert them into meaningful metrics.
Specifically, we’ll use the Count Connector to track the number of “Star Wars” and “Lord of the Rings” quotes appearing in our logs, turning them into measurable data points.
[WORKSHOP] directory, create a new subdirectory named 9-sum-count.8-routing-data directory into 9-sum-count.*.out and *.log files.[WORKSHOP]/9-sum-count directory..
├── agent.yaml
└── gateway.yamlfilelog/quotes receiver in the agent.yaml and add a poll_interval attribute: filelog/quotes: # Receiver Type/Name
poll_interval: 10s # Only read every ten seconds The reason for the delay is that the Count Connector in the OpenTelemetry Collector counts logs only within each processing interval. This means that every time the data is read, the count resets to zero for the next interval. With the default Filelog reciever interval of 200ms it reads every line the loadgen writes, giving us counts of 1. With this interval we make sure we have multiple entries to count.
The Collector can maintain a running count for each read interval by omitting conditions, as shown below. However, it’s best practice to let your backend handle running counts since it can track them over a longer time period.
Include the Count Connector in the connector’s section of your configuration and define the metrics counters we want to use:
connectors:
count:
logs:
logs.full.count:
description: "Running count of all logs read in interval"
logs.sw.count:
description: "StarWarsCount"
conditions:
- attributes["movie"] == "SW"
logs.lotr.count:
description: "LOTRCount"
conditions:
- attributes["movie"] == "LOTR"
logs.error.count:
description: "ErrorCount"
conditions:
- attributes["level"] == "ERROR"Explanation of the Metrics Counters
logs.full.count: Tracks the total number of logs processed during each read interval.logs.sw.count Counts logs that contain a quote from a Star Wars movie.logs.lotr.count: Counts logs that contain a quote from a Lord of the Rings movie.logs.error.count: Represents a real-world scenario by counting logs with a severity level of ERROR for the read interval.Configure the Count Connector in the pipelines
In the pipeline configuration below, the connector exporter is added to the logs section, while the connector receiver is added to the metrics section.
pipelines:
traces:
receivers:
- otlp
processors:
- memory_limiter
- attributes/update # Update, hash, and remove attributes
- redaction/redact # Redact sensitive fields using regex
- resourcedetection
- resource/add_mode
- batch
exporters:
- debug
- file
- otlphttp
metrics:
receivers:
- count # Count Connector - Receiver
- otlp
#- hostmetrics # Host Metrics Receiver
processors:
- memory_limiter
- resourcedetection
- resource/add_mode
- batch
exporters:
- debug
- otlphttp
logs:
receivers:
- otlp
- filelog/quotes
processors:
- memory_limiter
- resourcedetection
- resource/add_mode
- transform/logs # Transform logs processor
- batch
exporters:
- count # Count Connector - Exporter
- debug
- otlphttpWe count logs based on their attributes. If your log data is stored in the log body instead of attributes, you’ll need to use a Transform processor in your pipeline to extract key/value pairs and add them as attributes.
In this workshop, we’ve already added merge_maps(attributes, cache, "upsert") in the 07-transform section. This ensures that all relevant data is included in the log attributes for processing.
When selecting fields to create attributes from, be mindful—adding all fields indiscriminately is generally not ideal for production environments. Instead, choose only the fields that are truly necessary to avoid unnecessary data clutter.
logs and metrics: sections of your pipelines will look like this:%%{init:{"fontFamily":"monospace"}}%%
graph LR
%% Nodes
REC1(otlp<br>fa:fa-download):::receiver
REC2(filelog<br>fa:fa-download<br>quotes):::receiver
REC3(otlp<br>fa:fa-download):::receiver
PRO1(memory_limiter<br>fa:fa-microchip):::processor
PRO2(memory_limiter<br>fa:fa-microchip):::processor
PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
PRO4(resource<br>fa:fa-microchip<br>add_mode):::processor
PRO5(batch<br>fa:fa-microchip):::processor
PRO6(batch<br>fa:fa-microchip):::processor
PRO7(resourcedetection<br>fa:fa-microchip):::processor
PRO8(resourcedetection<br>fa:fa-microchip):::processor
PRO9(transfrom<br>fa:fa-microchip<br>logs):::processor
EXP1(  debug  <br>fa:fa-upload):::exporter
EXP2(  otlphttp  <br>fa:fa-upload):::exporter
EXP3(  debug  <br>fa:fa-upload):::exporter
EXP4(  otlphttp  <br>fa:fa-upload):::exporter
ROUTE1( count <br>fa:fa-route):::con-export
ROUTE2( count <br>fa:fa-route):::con-receive
%% Links
subID1:::sub-logs
subID2:::sub-metrics
subgraph " "
direction LR
subgraph subID1[**Logs**]
direction LR
REC1 --> PRO1
REC2 --> PRO1
PRO1 --> PRO7
PRO7 --> PRO3
PRO3 --> PRO9
PRO9 --> PRO5
PRO5 --> ROUTE1
PRO5 --> EXP1
PRO5 --> EXP2
end
subgraph subID2[**Metrics**]
direction LR
ROUTE1 --> ROUTE2
ROUTE2 --> PRO2
REC3 --> PRO2
PRO2 --> PRO8
PRO8 --> PRO4
PRO4 --> PRO6
PRO6 --> EXP3
PRO6 --> EXP4
end
end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-logs stroke:#34d399,stroke-width:1px, color:#34d399,stroke-dasharray: 3 3;
classDef sub-metrics stroke:#38bdf8,stroke-width:1px, color:#38bdf8,stroke-dasharray: 3 3;Start the Gateway:
In the Gateway terminal window navigate to the [WORKSHOP]/9-sum-count directory and run:
../otelcol --config=gateway.yamlStart the Agent:
In the Agent terminal window navigate to the [WORKSHOP]/9-sum-count directory and run:
../otelcol --config=agent.yamlSend 12 Logs lines with the Loadgen:
In the Spans terminal window navigate to the [WORKSHOP]/9-sum-count directory.
Send 12 log lines, they should be read in two intervals. Do this with the following loadgen command:
../loadgen -logs -json -count 12Both the agent and gateway will display debug information, showing they are processing data. Wait until the loadgen completes.
Verify metrics have been generated
As the logs are processed, the Agent generates metrics and forwards them to the Gateway, which then writes them to gateway-metrics.out.
To check if the metrics logs.full.count, logs.sw.count, logs.lotr.count, and logs.error.count are present in the output, run the following jq query:
jq '.resourceMetrics[].scopeMetrics[].metrics[]
| select(.name == "logs.full.count" or .name == "logs.sw.count" or .name == "logs.lotr.count" or .name == "logs.error.count")
| {name: .name, value: (.sum.dataPoints[0].asInt // "-")}' gateway-metrics.out{
"name": "logs.sw.count",
"value": "2"
}
{
"name": "logs.lotr.count",
"value": "2"
}
{
"name": "logs.full.count",
"value": "4"
}
{
"name": "logs.error.count",
"value": "2"
}
{
"name": "logs.error.count",
"value": "1"
}
{
"name": "logs.sw.count",
"value": "2"
}
{
"name": "logs.lotr.count",
"value": "6"
}
{
"name": "logs.full.count",
"value": "8"
}Note: the logs.full.count normally is equal to logs.sw.count + logs.lotr.count, while the logs.error.count will be a random number.
Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.
In this section, we’ll explore how the Sum Connector can extract values from spans and convert them into metrics.
We’ll specifically use the credit card charges from our base spans and leverage the Sum Connector to retrieve the total charges as a metric.
The connector can be used to collect (sum) attribute values from spans, span events, metrics, data points, and log records. It captures each individual value, transforms it into a metric, and passes it along. However, it’s the backend’s job to use these metrics and their attributes for calculations and further processing.
Switch to your Agent terminal window and open the agent.yaml file in your editor.
sum:
spans:
user.card-charge:
source_attribute: payment.amount
conditions:
- attributes["payment.amount"] != "NULL"
attributes:
- key: user.name
In the example above, we check for the payment.amount attribute in spans. If it has a valid value, the Sum connector generates a metric called user.card-charge and includes the user.name as an attribute. This enables the backend to track and display a user’s total charges over an extended period, such as a billing cycle.
In the pipeline configuration below, the connector exporter is added to the traces section, while the connector receiver is added to the metrics section.
pipelines:
traces:
receivers:
- otlp
processors:
- memory_limiter
- attributes/update # Update, hash, and remove attributes
- redaction/redact # Redact sensitive fields using regex
- resourcedetection
- resource/add_mode
- batch
exporters:
- debug
- file
- otlphttp
- sum # Sum Connector - Receiver
metrics:
receivers:
- sum # Sum Connector -
- count # Count Connector - Receiver
- otlp
#- hostmetrics # Host Metrics Receiver
processors:
- memory_limiter
- resourcedetection
- resource/add_mode
- batch
exporters:
- debug
- otlphttp
logs:
receivers:
- otlp
- filelog/quotes
processors:
- memory_limiter
- resourcedetection
- resource/add_mode
- transform/logs # Transform logs processor
- batch
exporters:
- count # Count Connector - Exporter
- debug
- otlphttptraces and metrics: sections of your pipelines will look like this:%%{init:{"fontFamily":"monospace"}}%%
graph LR
%% Nodes
REC1(otlp<br>fa:fa-download<br> ):::receiver
REC3(otlp<br>fa:fa-download<br> ):::receiver
PRO1(memory_limiter<br>fa:fa-microchip<br> ):::processor
PRO2(memory_limiter<br>fa:fa-microchip<br> ):::processor
PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
PRO4(resource<br>fa:fa-microchip<br>add_mode):::processor
PRO5(batch<br>fa:fa-microchip<br> ):::processor
PRO6(batch<br>fa:fa-microchip<br> ):::processor
PRO7(resourcedetection<br>fa:fa-microchip<br> ):::processor
PRO8(resourcedetection<br>fa:fa-microchip<br>):::processor
PROA(attributes<br>fa:fa-microchip<br>redact):::processor
PROB(redaction<br>fa:fa-microchip<br>update):::processor
EXP1(  debug  <br>fa:fa-upload<br> ):::exporter
EXP2(  file  <br>fa:fa-upload<br> ):::exporter
EXP3(  debug  <br>fa:fa-upload<br> ):::exporter
EXP4(  otlphttp  <br>fa:fa-upload<br> ):::exporter
EXP5(  otlphttp  <br>fa:fa-upload<br> ):::exporter
ROUTE1( sum <br>fa:fa-route<br> ):::con-export
ROUTE2( count <br>fa:fa-route<br> ):::con-receive
ROUTE3( sum <br>fa:fa-route<br> ):::con-receive
%% Links
subID1:::sub-traces
subID2:::sub-metrics
subgraph " "
direction LR
subgraph subID1[**Traces**]
direction LR
REC1 --> PRO1
PRO1 --> PROA
PROA --> PROB
PROB --> PRO7
PRO7 --> PRO3
PRO3 --> PRO5
PRO5 --> EXP1
PRO5 --> EXP2
PRO5 --> EXP5
PRO5 --> ROUTE1
end
subgraph subID2[**Metrics**]
direction LR
ROUTE1 --> ROUTE3
ROUTE3 --> PRO2
ROUTE2 --> PRO2
REC3 --> PRO2
PRO2 --> PRO8
PRO8 --> PRO4
PRO4 --> PRO6
PRO6 --> EXP3
PRO6 --> EXP4
end
end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-logs stroke:#34d399,stroke-width:1px, color:#34d399,stroke-dasharray: 3 3;
classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3;
classDef sub-metrics stroke:#38bdf8,stroke-width:1px, color:#38bdf8,stroke-dasharray: 3 3;Start the Gateway:
In the Gateway terminal window navigate to the [WORKSHOP]/9-sum-count directory and run:
../otelcol --config=gateway.yamlStart the Agent:
In the Agent terminal window navigate to the [WORKSHOP]/9-sum-count directory and run:
../otelcol --config=agent.yamlStart the Loadgen:
In the Spans terminal window navigate to the [WORKSHOP]/9-sum-count directory. Send 8 spans with the following loadgen command:
../loadgen -count 8Both the agent and gateway will display debug information, showing they are processing data. Wait until the loadgen completes.
Verify that metrics
While processing the span, the Agent, has generated metrics and passed them on to the Gateway. The Gateway has written them into gateway-metrics.out.
To confirm the presence of user.card-chargeand verify each one has an attribute user.name in the metrics output, run the following jq query:
jq -r '.resourceMetrics[].scopeMetrics[].metrics[] | select(.name == "user.card-charge") | .sum.dataPoints[] | "\(.attributes[] | select(.key == "user.name").value.stringValue)\t\(.asDouble)"' gateway-metrics.out | while IFS=$'\t' read -r name charge; do
printf "%-20s %s\n" "$name" "$charge"
done George Lucas 67.49
Frodo Baggins 87.14
Thorin Oakenshield 90.98
Luke Skywalker 51.37
Luke Skywalker 65.56
Thorin Oakenshield 67.5
Thorin Oakenshield 66.66
Peter Jackson 94.39Stop the agent and the gateway processes by pressing Ctrl-C in their respective terminals.
