Open or create your second terminal window and name it Gateway. Navigate to the first exercise directory [WORKSHOP]/1-agent-gateway then check the contents of the gateway.yaml file.

This file outlines the core structure of the OpenTelemetry Collector as deployed in Gateway mode.

Gateway: In the Gateway terminal window, run the following command to start the Gateway:

../otelcol --config=gateway.yaml

If everything is configured correctly, the collector will start and state Everything is ready. Begin running and processing data. in the output, similar to the following:

2025-06-09T09:22:11.944+0100    info    service@v0.126.0/service.go:289 Everything is ready. Begin running and processing data. {"resource": {}}

Once the Gateway is running, it will listen for incoming data on port 5318 and export the received data to the following files:

• gateway-traces.out
• gateway-metrics.out
• gateway-logs.out

Start the Agent: In the Agent terminal window start the agent with the agent configuration:

../otelcol --config=agent.yaml

Verify CPU Metrics:

1. Check that when the Agent starts, it immediately starts sending CPU metrics.
2. Both the Agent and the Gateway will display this activity in their debug output. The output should resemble the following snippet:

<snip>
NumberDataPoints #31
Data point attributes:
     -> cpu: Str(cpu3)
     -> state: Str(wait)
StartTimestamp: 2025-07-07 16:49:42 +0000 UTC
Timestamp: 2025-07-09 09:36:21.190226459 +0000 UTC
Value: 77.380000
        {"resource": {}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics"}

At this stage, the Agent continues to collect CPU metrics once per hour or upon each restart and sends them to the gateway. The Gateway processes these metrics and exports them to a file named gateway-metrics.out. This file stores the exported metrics as part of the pipeline service.

Verify Data arrived at Gateway: To confirm that CPU metrics, specifically for cpu0, have successfully reached the gateway, we’ll inspect the gateway-metrics.out file using the jq command.

The following command filters and extracts the system.cpu.time metric, focusing on cpu0. It displays the metric’s state (e.g., user, system, idle, interrupt) along with the corresponding values.

Open or create your third terminal window and name it Tests. Run the command below in the Tests terminal to check the system.cpu.time metric:

jq '.resourceMetrics[].scopeMetrics[].metrics[] | select(.name == "system.cpu.time") | .sum.dataPoints[] | select(.attributes[0].value.stringValue == "cpu0") | {cpu: .attributes[0].value.stringValue, state: .attributes[1].value.stringValue, value: .asDouble}' gateway-metrics.out

{
  "cpu": "cpu0",
  "state": "user",
  "value": 123407.02
}
{
  "cpu": "cpu0",
  "state": "system",
  "value": 64866.6
}
{
  "cpu": "cpu0",
  "state": "idle",
  "value": 216427.87
}
{
  "cpu": "cpu0",
  "state": "interrupt",
  "value": 0
}

Your directory structure will look like this:

.
├── agent.yaml
└── gateway.yaml

Update the agent.yaml: In the Agent terminal window, add the file_storage extension under the existing health_check extension:

  file_storage/checkpoint:             # Extension Type/Name
    directory: "./checkpoint-dir"      # Define directory
    create_directory: true             # Create directory
    timeout: 1s                        # Timeout for file operations
    compaction:                        # Compaction settings
      on_start: true                   # Start compaction at Collector startup
      # Define compaction directory
      directory: "./checkpoint-dir/tmp"
      max_transaction_size: 65536      # Max. size limit before compaction occurs

Add file_storage to the exporter: Modify the otlphttp exporter to configure retry and queuing mechanisms, ensuring data is retained and resent if failures occur. Add the following under the endpoint: "http://localhost:5318" and make sure the indentation matches endpoint:

    retry_on_failure:
      enabled: true                    # Enable retry on failure
    sending_queue:                     # 
      enabled: true                    # Enable sending queue
      num_consumers: 10                # No. of consumers
      queue_size: 10000                # Max. queue size
      storage: file_storage/checkpoint # File storage extension

Update the services section: Add the file_storage/checkpoint extension to the existing extensions: section and the configuration needs to look like this:

service:
  extensions:
  - health_check
  - file_storage/checkpoint            # Enabled extensions for this collector

Update the metrics pipeline: For this exercise we are going to comment out the hostmetrics receiver from the Metric pipeline to reduce debug and log noise, again the configuration needs to look like this:

    metrics:
      receivers:
      # - hostmetrics                    # Hostmetric reciever (cpu only)
      - otlp

Start the Gateway: In the Gateway terminal window run:

../otelcol --config=gateway.yaml

Start the Agent: In the Agent terminal window run:

../otelcol --config=agent.yaml

Send five test spans: In the Loadgen terminal window run:

../loadgen -count 5

Both the Agent and Gateway should display debug logs, and the Gateway should create a ./gateway-traces.out file.

If everything functions correctly, we can proceed with testing system resilience.

Simulate a network failure: In the Gateway terminal stop the Gateway with Ctrl-C and wait until the gateway console shows that it has stopped. The Agent will continue running, but it will not be able to send data to the gateway. The output in the Gateway terminal should look similar to this:

2025-07-09T10:22:37.941Z        info    service@v0.126.0/service.go:345 Shutdown complete.      {"resource": {}}

Send traces: In the Loadgen terminal window send five more traces using the loadgen.

../loadgen -count 5

Notice that the agent’s retry mechanism is activated as it continuously attempts to resend the data. In the agent’s console output, you will see repeated messages similar to the following:

2025-01-28T14:22:47.020+0100  info  internal/retry_sender.go:126  Exporting failed. Will retry the request after interval.  {"kind": "exporter", "data_type": "traces", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"http://localhost:5318/v1/traces\": dial tcp 127.0.0.1:5318: connect: connection refused", "interval": "9.471474933s"}

Stop the Agent: In the Agent terminal window, use Ctrl-C to stop the agent. Wait until the agent’s console confirms it has stopped:

2025-07-09T10:25:59.344Z        info    service@v0.126.0/service.go:345 Shutdown complete.      {"resource": {}}

Restart the Gateway: In the Gateway terminal window run:

../otelcol --config=gateway.yaml

Restart the Agent: In the Agent terminal window run:

../otelcol --config=agent.yaml

After the Agent is up and running, the File_Storage extension will detect buffered data in the checkpoint folder. It will start to dequeue the stored spans from the last checkpoint folder, ensuring no data is lost.

Verify the Agent Debug output: Note that the Agent debug output does NOT change and still shows the following line indicating no new data is being exported:

2025-07-11T08:31:58.176Z        info    service@v0.126.0/service.go:289 Everything is ready. Begin running and processing data.   {"resource": {}}

Watch the Gateway Debug output
You should see from the Gateway debug screen, it has started receiving the previously missed traces without requiring any additional action on your part e.g.:

Attributes:
   -> user.name: Str(Luke Skywalker)
   -> user.phone_number: Str(+1555-867-5309)
   -> user.email: Str(george@deathstar.email)
   -> user.password: Str(LOTR>StarWars1-2-3)
   -> user.visa: Str(4111 1111 1111 1111)
   -> user.amex: Str(3782 822463 10005)
   -> user.mastercard: Str(5555 5555 5555 4444)
   -> payment.amount: Double(75.75)
      {"resource": {}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces"}

Check the gateway-traces.out file: Using jq, count the number of traces in the recreated gateway-traces.out. It should match the number you send when the Gateway was down.

jq '.resourceSpans | length | "\(.) resourceSpans found"' gateway-traces.out

"5 resourceSpans found"

Switch to your Gateway terminal window and open the gateway.yaml file. Update the processors section with the following configuration:

1. Add a filter processor:
   Configure the gateway to exclude spans with the name /_healthz. The error_mode: ignore directive ensures that any errors encountered during filtering are ignored, allowing the pipeline to continue running smoothly. The traces section defines the filtering rules, specifically targeting spans named /_healthz for exclusion.

     filter/health:                       # Defines a filter processor
       error_mode: ignore                 # Ignore errors
       traces:                            # Filtering rules for traces
         span:                            # Exclude spans named "/_healthz"
          - 'name == "/_healthz"'

2. Add the filter processor to the traces pipeline:
   Include the filter/health processor in the traces pipeline. For optimal performance, place the filter as early as possible—right after the memory_limiter and before the batch processor. Here’s how the configuration should look:

   traces:
     receivers:
       - otlp
     processors:
       - memory_limiter
       - filter/health             # Filters data based on rules
       - resource/add_mode
       - batch
     exporters:
       - debug
       - file/traces

This setup ensures that health check related spans (/_healthz) are filtered out early in the pipeline, reducing unnecessary noise in your telemetry data.

Start the Gateway: In your Gateway terminal window start the Gateway.

../otelcol --config ./gateway.yaml

Start the Agent: In your Agent terminal window start the Agent.

../otelcol --config ./agent.yaml

Start the Loadgen: In the Loadgen terminal window, execute the following command to start the load generator with health check spans enabled:

../loadgen -health -count 5

The debug output in the Agent terminal will show _healthz spans:

InstrumentationScope healthz 1.0.0
Span #0
   Trace ID       : 0cce8759b5921c8f40b346b2f6e2f4b6
   Parent ID      :
   ID             : bc32bd0e4ddcb174
   Name           : /_healthz
   Kind           : Server
   Start time     : 2025-07-11 08:47:50.938703979 +0000 UTC
   End time       : 2025-07-11 08:47:51.938704091 +0000 UTC
   Status code    : Ok
   Status message : Success

They will not be present in the Gateway debug as they are dropped by the filter processor that was configured earlier.

Verify agent.out: Using jq, in the Test terminal, confirm the name of the spans received by the Agent:

jq -c '.resourceSpans[].scopeSpans[].spans[] | "Span \(input_line_number) found with name \(.name)"' ./agent.out

"Span 1 found with name /movie-validator"
"Span 2 found with name /_healthz"
"Span 3 found with name /movie-validator"
"Span 4 found with name /_healthz"
"Span 5 found with name /movie-validator"
"Span 6 found with name /_healthz"
"Span 7 found with name /movie-validator"
"Span 8 found with name /_healthz"
"Span 9 found with name /movie-validator"
"Span 10 found with name /_healthz"

Check the Gateway Debug output: Using jq confirm the name of the spans received by the Gateway:

jq -c '.resourceSpans[].scopeSpans[].spans[] | "Span \(input_line_number) found with name \(.name)"' ./gateway-traces.out

"Span 1 found with name /movie-validator"
"Span 2 found with name /movie-validator"
"Span 3 found with name /movie-validator"
"Span 4 found with name /movie-validator"
"Span 5 found with name /movie-validator"

Switch to your Agent terminal window and open the agent.yaml file in your editor. We’ll add two processors to enhance the security and privacy of your telemetry data.

1. Add an attributes Processor: The Attributes Processor allows you to modify span attributes (tags) by updating, deleting, or hashing their values. This is particularly useful for obfuscating sensitive information before it is exported.

In this step, we’ll:

1. Update the user.phone_number attribute to a static value ("UNKNOWN NUMBER").
2. Hash the user.email attribute to ensure the original email is not exposed.
3. Delete the user.password attribute to remove it entirely from the span.

  attributes/update:
    actions:                           # Actions
      - key: user.phone_number         # Target key
        action: update                 # Update action
        value: "UNKNOWN NUMBER"        # New value
      - key: user.email                # Target key
        action: hash                   # Hash the email value
      - key: user.password             # Target key
        action: delete                 # Delete the password

2. Add a redaction Processor: The Redaction Processor detects and redacts sensitive data in span attributes based on predefined patterns, such as credit card numbers or other personally identifiable information (PII).

In this step:

• We set allow_all_keys: true to ensure all attributes are processed (if set to false, only explicitly allowed keys are retained).

• We define blocked_values with regular expressions to detect and redact Visa and MasterCard credit card numbers.

• The summary: debug option logs detailed information about the redaction process for debugging purposes.

  redaction/redact:
    allow_all_keys: true               # If false, only allowed keys will be retained
    blocked_values:                    # List of regex patterns to block
      - '\b4[0-9]{3}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b'       # Visa
      - '\b5[1-5][0-9]{2}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b'  # MasterCard
    summary: debug                     # Show debug details about redaction

Update the traces Pipeline: Integrate both processors into the traces pipeline. Make sure that you comment out the redaction processor at first (we will enable it later in a separate exercise). Your configuration should look like this:

    traces:
      receivers:
      - otlp
      processors:
      - memory_limiter
      - attributes/update              # Update, hash, and remove attributes
      #- redaction/redact               # Redact sensitive fields using regex
      - resourcedetection
      - resource/add_mode
      - batch
      exporters:
      - debug
      - file
      - otlphttp

Start the Gateway: In your Gateway terminal window start the Gateway.

../otelcol --config=gateway.yaml

Start the Agent: In your Agent terminal window start the Agent.

../otelcol --config=agent.yaml

Start the Load Generator: In the Loadgen terminal window start the loadgen:

../loadgen -count 1

Check the debug output: For both the Agent and Gateway confirm that user.account_password has been removed, and both user.phone_number & user.email have been updated:

   -> user.name: Str(George Lucas)
   -> user.phone_number: Str(UNKNOWN NUMBER)
   -> user.email: Str(62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287)
   -> payment.amount: Double(51.71)
   -> user.visa: Str(4111 1111 1111 1111)
   -> user.amex: Str(3782 822463 10005)
   -> user.mastercard: Str(5555 5555 5555 4444)

    -> user.name: Str(George Lucas)
    -> user.phone_number: Str(+1555-867-5309)
    -> user.email: Str(george@deathstar.email)
    -> user.password: Str(LOTR>StarWars1-2-3)
    -> user.visa: Str(4111 1111 1111 1111)
    -> user.amex: Str(3782 822463 10005)
    -> user.mastercard: Str(5555 5555 5555 4444)
    -> payment.amount: Double(95.22)

Check file output: Using jq validate that user.account_password has been removed, and user.phone_number & user.email have been updated in gateway-taces.out:

jq '.resourceSpans[].scopeSpans[].spans[].attributes[] | select(.key == "user.password" or .key == "user.phone_number" or .key == "user.email") | {key: .key, value: .value.stringValue}' ./gateway-traces.out

{
  "key": "user.phone_number",
  "value": "UNKNOWN NUMBER"
}
{
  "key": "user.email",
  "value": "62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287"
}

Start the Gateway: In your Gateway terminal window start the Gateway.

../otelcol --config=gateway.yaml

Enable the redaction/redact processor: In the Agent terminal window, edit agent.yaml and remove the # we inserted in the previous exercise.

    traces:
      receivers:
      - otlp
      processors:
      - memory_limiter
      - attributes/update              # Update, hash, and remove attributes
      - redaction/redact               # Redact sensitive fields using regex
      - resourcedetection
      - resource/add_mode
      - batch
      exporters:
      - debug
      - file
      - otlphttp

Start the Agent: In your Agent terminal window start the Agent.

../otelcol --config=agent.yaml

Start the Load Generator: In the Loadgen terminal window start the loadgen:

../loadgen -count 1

Check the debug output: For both the Agent and Gateway confirm the values for user.visa & user.mastercard have been updated. Notice user.amex attribute value was NOT redacted because a matching regex pattern was not added to blocked_values

   -> user.name: Str(George Lucas)
   -> user.phone_number: Str(UNKNOWN NUMBER)
   -> user.email: Str(62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287)
   -> payment.amount: Double(69.71)
   -> user.visa: Str(****)
   -> user.amex: Str(3782 822463 10005)
   -> user.mastercard: Str(****)
   -> redaction.masked.keys: Str(user.mastercard,user.visa)
   -> redaction.masked.count: Int(2)

    -> user.name: Str(George Lucas)
    -> user.phone_number: Str(+1555-867-5309)
    -> user.email: Str(george@deathstar.email)
    -> user.password: Str(LOTR>StarWars1-2-3)
    -> user.visa: Str(4111 1111 1111 1111)
    -> user.amex: Str(3782 822463 10005)
    -> user.mastercard: Str(5555 5555 5555 4444)
    -> payment.amount: Double(65.54)

     -> redaction.masked.keys: Str(user.mastercard,user.visa)
     -> redaction.masked.count: Int(2)

Check file output: Using jq verify that user.visa & user.mastercard have been updated in the gateway-traces.out.

jq '.resourceSpans[].scopeSpans[].spans[].attributes[] | select(.key == "user.visa" or .key == "user.mastercard" or .key == "user.amex") | {key: .key, value: .value.stringValue}' ./gateway-traces.out

{
  "key": "user.visa",
  "value": "****"
}
{
  "key": "user.amex",
  "value": "3782 822463 10005"
}
{
  "key": "user.mastercard",
  "value": "****"
}

These are just a couple of examples of how attributes and redaction processors can be configured to protect sensitive data.

Add a transform processor: Switch to your Gateway terminal window and edit the gateway.yaml and add the following transform processor:

  transform/logs:                      # Processor Type/Name
    log_statements:                    # Log Processing Statements
      - context: resource              # Log Context
        statements:                    # List of attribute keys to keep
          - keep_keys(attributes, ["com.splunk.sourcetype", "host.name", "otelcol.service.mode"])

By using the -context: resource key we are targeting the resourceLog attributes of logs.

This configuration ensures that only the relevant resource attributes (com.splunk.sourcetype, host.name, otelcol.service.mode) are retained, improving log efficiency and reducing unnecessary metadata.

Adding a Context Block for Log Severity Mapping: To properly set the severity_text and severity_number fields of a log record, we add a log context block within log_statements. This configuration extracts the level value from the log body, maps it to severity_text, and assigns the corresponding severity_number based on the log level:

      - context: log                   # Log Context
        statements:                    # Transform Statements Array
          - set(cache, ParseJSON(body)) where IsMatch(body, "^\\{")  # Parse JSON log body into a cache object
          - flatten(cache, "")                                        # Flatten nested JSON structure
          - merge_maps(attributes, cache, "upsert")                   # Merge cache into attributes, updating existing keys
          - set(severity_text, attributes["level"])                   # Set severity_text from the "level" attribute
          - set(severity_number, 1) where severity_text == "TRACE"    # Map severity_text to severity_number
          - set(severity_number, 5) where severity_text == "DEBUG"
          - set(severity_number, 9) where severity_text == "INFO"
          - set(severity_number, 13) where severity_text == "WARN"
          - set(severity_number, 17) where severity_text == "ERROR"
          - set(severity_number, 21) where severity_text == "FATAL"

The merge_maps function is used to combine two maps (dictionaries) into one. In this case, it merges the cache object (containing parsed JSON data from the log body) into the attributes map.

• Parameters:
  • attributes: The target map where the data will be merged.
  • cache: The source map containing the parsed JSON data.
  • "upsert": This mode ensures that if a key already exists in the attributes map, its value will be updated with the value from cache. If the key does not exist, it will be inserted.

This step is crucial because it ensures that all relevant fields from the log body (e.g., level, message, etc.) are added to the attributes map, making them available for further processing or exporting.

Summary of Key Transformations:

• Parse JSON: Extracts structured data from the log body.
• Flatten JSON: Converts nested JSON objects into a flat structure.
• Merge Attributes: Integrates extracted data into log attributes.
• Map Severity Text: Assigns severity_text from the log’s level attribute.
• Assign Severity Numbers: Converts severity levels into standardized numerical values.

This configuration ensures that log severity is correctly extracted, standardized, and structured for efficient processing.

Update the logs pipeline: Add the transform/logs: processor into the logs: pipeline so your configuration looks like this:

    logs:                         # Logs pipeline
      receivers:
      - otlp                      # OTLP receiver
      processors:                 # Processors for logs
      - memory_limiter
      - resource/add_mode
      - transform/logs
      - batch
      exporters:
      - debug                     # Debug exporter
      - file/logs

Start the Gateway: In the Gateway terminal run:

../otelcol --config=gateway.yaml

Start the Agent: In the Agent terminal run:

../otelcol --config=agent.yaml

Start the Load Generator: In the Loadgen terminal window, execute the following command to start the load generator with JSON enabled:

../loadgen -logs -json -count 5

The loadgen will write 5 log lines to ./quotes.log in JSON format.

Check the debug output: For both the Agent and Gateway confirm that com.splunk/source and os.type have been removed:

Resource attributes:
   -> com.splunk.sourcetype: Str(quotes)
   -> host.name: Str(workshop-instance)
   -> otelcol.service.mode: Str(agent)

Resource attributes:
   -> com.splunk.source: Str(./quotes.log)
   -> com.splunk.sourcetype: Str(quotes)
   -> host.name: Str(workshop-instance)
   -> os.type: Str(linux)
   -> otelcol.service.mode: Str(agent)

For both the Agent and Gateway confirm that SeverityText and SeverityNumber in the LogRecord is now defined with the severity level from the log body. Confirm that the JSON fields from the body can be accessed as top-level log Attributes:

<snip>
SeverityText: WARN
SeverityNumber: Warn(13)
Body: Str({"level":"WARN","message":"Your focus determines your reality.","movie":"SW","timestamp":"2025-03-07 11:17:26"})
Attributes:
     -> log.file.path: Str(quotes.log)
     -> level: Str(WARN)
     -> message: Str(Your focus determines your reality.)
     -> movie: Str(SW)
     -> timestamp: Str(2025-03-07 11:17:26)
</snip>

<snip>
SeverityText:
SeverityNumber: Unspecified(0)
Body: Str({"level":"WARN","message":"Your focus determines your reality.","movie":"SW","timestamp":"2025-03-07 11:17:26"})
Attributes:
     -> log.file.path: Str(quotes.log)
</snip>

Check file output: In the new gateway-logs.out file verify the data has been transformed:

jq '[.resourceLogs[].scopeLogs[].logRecords[] | {severityText, severityNumber, body: .body.stringValue}]' gateway-logs.out

[
  {
    "severityText": "DEBUG",
    "severityNumber": 5,
    "body": "{\"level\":\"DEBUG\",\"message\":\"All we have to decide is what to do with the time that is given us.\",\"movie\":\"LOTR\",\"timestamp\":\"2025-03-07 11:56:29\"}"
  },
  {
    "severityText": "WARN",
    "severityNumber": 13,
    "body": "{\"level\":\"WARN\",\"message\":\"The Force will be with you. Always.\",\"movie\":\"SW\",\"timestamp\":\"2025-03-07 11:56:29\"}"
  },
  {
    "severityText": "ERROR",
    "severityNumber": 17,
    "body": "{\"level\":\"ERROR\",\"message\":\"One does not simply walk into Mordor.\",\"movie\":\"LOTR\",\"timestamp\":\"2025-03-07 11:56:29\"}"
  },
  {
    "severityText": "DEBUG",
    "severityNumber": 5,
    "body": "{\"level\":\"DEBUG\",\"message\":\"Do or do not, there is no try.\",\"movie\":\"SW\",\"timestamp\":\"2025-03-07 11:56:29\"}"
  }
]
[
  {
    "severityText": "ERROR",
    "severityNumber": 17,
    "body": "{\"level\":\"ERROR\",\"message\":\"There is some good in this world, and it's worth fighting for.\",\"movie\":\"LOTR\",\"timestamp\":\"2025-03-07 11:56:29\"}"
  }
]

Add new file exporters: The routing connector requires different targets for routing. Create two new file exporters, file/traces/route1-regular and file/traces/route2-security, to ensure data is directed correctly in the exporters section of the gateway.yaml:

  file/traces/route1-regular:                     # Exporter for regular traces
    path: "./gateway-traces-route1-regular.out"   # Path for saving trace data
    append: false                                 # Overwrite the file each time
  file/traces/route2-security:                    # Exporter for security traces
    path: "./gateway-traces-route2-security.out"  # Path for saving trace data
    append: false                                 # Overwrite the file each time 

Enable Routing by adding the routing connector. In OpenTelemetry configuration files, connectors have their own dedicated section, similar to receivers and processors.

In the Gateway terminal window, edit gateway.yaml and find and uncomment the #connectors: section. Then, add the following below the connectors: section:

  routing:
    default_pipelines: [traces/route1-regular]  # Default pipeline if no rule matches
    error_mode: ignore                          # Ignore errors in routing
    table:                                      # Define routing rules
      # Routes spans to a target pipeline if the resourceSpan attribute matches the rule
      - statement: route() where attributes["deployment.environment"] == "security-applications"
        pipelines: [traces/route2-security]     # Security target pipeline 

Update the original traces pipeline to use routing:

1. To enable routing, update the original traces pipeline to use routing as the only exporter. This ensures all span data is sent through the Routing Connector for evaluation and then onwards to connected pipelines. Also, remove all processors and replace it with an empty array ([]) as this will now behandeld in the traces/route1-regular and traces/route2-security pipelines, allowing for custom behaviour for each route. Your traces: configuration should look like this:

   traces:                       # Traces pipeline
     receivers:
     - otlp                      # OTLP receiver
     processors: []              # Processors for traces
     exporters:
     - routing

Add both the route1-regular and route2-security traces pipelines below the existing traces pipeline:

1. Configure Route1-regular pipeline: This pipeline will handle all spans that have no match in the routing table in the connector. Notice this uses routing as its only receiver and will recieve data thought its connection from the original traces pipeline.

       traces/route1-regular:         # Default pipeline for unmatched spans
         receivers: 
         - routing                    # Receive data from the routing connector
         processors:
         - memory_limiter             # Memory Limiter Processor
         - resource/add_mode          # Adds collector mode metadata
         - batch
         exporters:
         - debug                      # Debug Exporter 
         - file/traces/route1-regular # File Exporter for unmatched spans 

2. Add the route2-security pipeline: This pipeline processes all spans that do match our rule "[deployment.environment"] == "security-applications" in the the routing rule. This pipeline is also using routing as its receiver. Add this pipline below the traces/route1-regular one.

       traces/route2-security:         # Default pipeline for unmatched spans
         receivers: 
         - routing                     # Receive data from the routing connector
         processors:
         - memory_limiter              # Memory Limiter Processor
         - resource/add_mode           # Adds collector mode metadata
         - batch
         exporters:
         - debug                       # Debug exporter
         - file/traces/route2-security # File exporter for unmatched spans

In this section, we will test the routing rule configured for the Gateway. The expected result is that a span generated by the loadgen that match the "[deployment.environment"] == "security-applications" rule will be sent to the gateway-traces-route2-security.out file.

Start the Gateway: In your Gateway terminal window start the Gateway.

../otelcol --config gateway.yaml

Start the Agent: In your Agent terminal window start the Agent.

../otelcol --config agent.yaml

Send a Regular Span: In the Loadgen terminal window send a regular span using the loadgen:

../loadgen -count 1

Both the Agent and Gateway will display debug information. The gateway will also generate a new gateway-traces-route1-regular.out file, as this is now the designated destination for regular spans.

Send a Security Span: In the Loadgen terminal window send a security span using the security flag:

../loadgen -security -count 1

Again, both the Agent and Gateway should display debug information, including the span you just sent. This time, the Gateway will write a line to the gateway-traces-route2-security.out file, which is designated for spans where the deployment.environment resource attribute matches "security-applications".

jq -c '.resourceSpans[] as $resource | $resource.scopeSpans[].spans[] | {spanId: .spanId, deploymentEnvironment: ($resource.resource.attributes[] | select(.key == "deployment.environment") | .value.stringValue)}' gateway-traces-route2-security.out

{"spanId":"cb799e92e26d5782","deploymentEnvironment":"security-applications"}

You can repeat this scenario multiple times, and each trace will be written to its corresponding output file.