8. Routing Data

10 minutes  

The Routing Connector in OpenTelemetry is a powerful feature that allows you to direct data (traces, metrics, or logs) to different pipelines based on specific criteria. This is especially useful in scenarios where you want to apply different processing or exporting logic to subsets of your telemetry data.

For example, you might want to send production data to one exporter while directing test or development data to another. Similarly, you could route certain spans based on their attributes, such as service name, environment, or span name, to apply custom processing or storage logic.

Exercise
  • Inside the [WORKSHOP] directory, create a new subdirectory named 8-routing.
  • Next, copy all contents from the 7-transform-data directory into 8-routing.
  • After copying, remove any *.out and *.log files.
  • Change all terminal windows to the [WORKSHOP]/8-routing directory.

Your updated directory structure will now look like this:

WORKSHOP
β”œβ”€β”€ 1-agent
β”œβ”€β”€ 2-gateway
β”œβ”€β”€ 3-filelog
β”œβ”€β”€ 4-resilience
β”œβ”€β”€ 5-dropping-spans
β”œβ”€β”€ 6-sensitive-data
β”œβ”€β”€ 7-transform-data
β”œβ”€β”€ 8-routing
β”‚   β”œβ”€β”€β”€checkpoint-dir
β”‚   β”œβ”€β”€ agent.yaml
β”‚   β”œβ”€β”€ health.json
β”‚   β”œβ”€β”€ gateway.yaml
β”‚   β”œβ”€β”€ log-gen.sh (or .ps1)
β”‚Β Β  └── trace.json
└── otelcol

Next, we will configure the routing connector and the respective pipelines.

Last Modified Feb 7, 2025

Subsections of 8. Routing Data

8.1 Configure the Routing Connector

In this exercise, you will configure the routing connector in the gateway.yaml file. This setup enables the Gateway to route traces based on the deployment.environment attribute in the spans you send. By implementing this, you can process and handle traces differently depending on their attributes.

Exercise

In OpenTelemetry configuration files, connectors have their own dedicated section, similar to receivers and processors.

Add the routing connector: In the Gateway terminal window edit gateway.yaml and add the following below the receivers: section and above the processors: section:

connectors:
  routing:
    default_pipelines: [traces/standard] # Default pipeline if no rule matches
    error_mode: ignore                   # Ignore errors in routing
    table:                               # Define routing rules
      # Routes spans to a target pipeline if the resourceSpan attribute matches the rule
      - statement: route() where attributes["deployment.environment"] == "security_applications"
        pipelines: [traces/security]     # Target pipeline 

The rules above apply to traces, but this approach also applies to metrics and logs, allowing them to be routed based on attributes in resourceMetrics or resourceLogs.

Configure file: exporters: The routing connector requires separate targets for routing. Add two file exporters, file/traces/security and file/traces/standard, to ensure data is directed correctly.

  file/traces/standard:                    # Exporter for regular traces
    path: "./gateway-traces-standard.out"  # Path for saving trace data
    append: false                          # Overwrite the file each time
  file/traces/security:                    # Exporter for security traces
    path: "./gateway-traces-security.out"  # Path for saving trace data
    append: false                          # Overwrite the file each time 

With the routing configuration complete, the next step is to configure the pipelines to apply these routing rules.

Last Modified Feb 17, 2025

8.2 Configuring the Pipelines

Exercise

Add both the standard and security traces pipelines:

  1. Add the Standard pipeline: This pipeline processes all spans that do not match the routing rule.
    This pipeline is using routing as its receiver. Place it below the existing traces: pipeline, keeping its configuration unchanged for now:

        traces/standard:                # Default pipeline for unmatched spans
          receivers: 
          - routing                     # Receive data from the routing connector
          processors:
          - memory_limiter              # Limits memory usage
          - resource/add_mode           # Adds collector mode metadata
          exporters:
          - debug                       # Debug exporter
          - file/traces/standard        # File exporter for unmatched spans
  2. Configure the Security pipeline: This pipeline will handle all spans that match the routing rule.
    This also uses routing as its receiver. Add this below the Standard one:

        traces/security:                # New Security Traces/Spans Pipeline       
          receivers: 
          - routing                     # Routing Connector, Only receives data from Connector
          processors:
          - memory_limiter              # Memory Limiter Processor
          - resource/add_mode           # Adds collector mode metadata
          exporters:
          - debug                       # Debug Exporter 
          - file/traces/security        # File Exporter for spans matching rule

Update the traces pipeline to use routing:

  1. To enable routing, update the original traces: pipeline by using routing as the only exporter.
    This ensures all span data is sent through the routing connector for evaluation.

  2. Remove all processors and replace it with an empty array []. These are now defined in the traces/standard and traces/security pipelines.

      pipelines:
        traces:                           # Original traces pipeline
          receivers: 
          - otlp                          # OTLP Receiver
          processors: []
          exporters: 
          - routing                       # Routing Connector
Note

By excluding the batch processor, spans are written immediately instead of waiting for multiple spans to accumulate before processing. This improves responsiveness, making the workshop run faster and allowing you to see results sooner.

Validate the agent configuration using otelbin.io. For reference, the traces: section of your pipelines will look similar to this:

%%{init:{"fontFamily":"monospace"}}%%
graph LR
    %% Nodes
      REC1(&nbsp;&nbsp;&nbsp;otlp&nbsp;&nbsp;&nbsp;<br>fa:fa-download):::receiver
      PRO1(memory_limiter<br>fa:fa-microchip):::processor
      PRO2(memory_limiter<br>fa:fa-microchip):::processor
      PRO3(resource<br>fa:fa-microchip):::processor
      PRO4(resource<br>fa:fa-microchip):::processor
      EXP1(&nbsp;&ensp;debug&nbsp;&ensp;<br>fa:fa-upload):::exporter
      EXP2(&emsp;&emsp;file&emsp;&emsp;<br>fa:fa-upload):::exporter
      EXP3(&nbsp;&ensp;debug&nbsp;&ensp;<br>fa:fa-upload):::exporter
      EXP4(&emsp;&emsp;file&emsp;&emsp;<br>fa:fa-upload):::exporter
      ROUTE1(&nbsp;routing&nbsp;<br>fa:fa-route):::con-export
      ROUTE2(&nbsp;routing&nbsp;<br>fa:fa-route):::con-receive
      ROUTE3(&nbsp;routing&nbsp;<br>fa:fa-route):::con-receive
    %% Links
    subID1:::sub-traces
    subID2:::sub-traces
    subID3:::sub-traces
    subgraph " "
    direction LR
      subgraph subID1[**Traces**]
      REC1 --> ROUTE1
      end
      subgraph subID2[**Traces/standard**]
      ROUTE1 --> ROUTE2
      ROUTE2 --> PRO1
      PRO1 --> PRO3
      PRO3 --> EXP1
      PRO3 --> EXP2
      end
      subgraph subID3[**Traces/security**]
      ROUTE1 --> ROUTE3
      ROUTE3 --> PRO2
      PRO2 --> PRO4
      PRO4 --> EXP3
      PRO4 --> EXP4
      end
    end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3;

Lets’ test our configuration!

Last Modified Feb 17, 2025

8.3 Setup Environment

In this section, we will test the routing rule configured for the Gateway. The expected result is that thespan from the security.json file will be sent to the gateway-traces-security.out file.

Exercise

Start the Gateway: In the Gateway terminal window navigate to the [WORKSHOP]/8-routing directory and run:

../otelcol --config=gateway.yaml

Start the Agent: In the Agent terminal window navigate to the [WORKSHOP]/8-routing directory and run:

../otelcol --config=agent.yaml

Create new security trace: In the Tests terminal window navigate to the [WORKSHOP]/8-routing directory.

The following JSON contains attributes which will trigger the routing rule. Copy the content from the tab below and save into a file named security.json.

{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"password_check"}},{"key":"deployment.environment","value":{"stringValue":"security_applications"}}]},"scopeSpans":[{"scope":{"name":"my.library","version":"1.0.0","attributes":[{"key":"my.scope.attribute","value":{"stringValue":"some scope attribute"}}]},"spans":[{"traceId":"5B8EFFF798038103D269B633813FC60C","spanId":"EEE19B7EC3C1B174","parentSpanId":"EEE19B7EC3C1B173","name":"I'm a server span","startTimeUnixNano":"1544712660000000000","endTimeUnixNano":"1544712661000000000","kind":2,"attributes":[{"keytest":"my.span.attr","value":{"stringValue":"some value"}}]}]}]}]}
Last Modified Feb 8, 2025

8.4 Test Routing Connector

Exercise

Send a Regular Span:

  1. Locate the Test terminal and navigate to the [WORKSHOP]/8-routing directory.
  2. Send a regular span using the trace.json file to confirm proper communication.

Both the Agent and Gateway should display debug information, including the span you just sent. The gateway will also generate a new gateway-traces-standard.out file, as this is now the designated destination for regular spans.

Tip

If you check gateway-traces-standard.out, it should contain the span sent using the cURL command. You will also see an empty gateway-traces-security.out file, as the routing configuration creates output files immediately, even if no matching spans have been processed yet.

Send a Security Span:

  1. Ensure both the Agent and Gateway are running.
  2. Send a security span using the security.json file to test the gateway’s routing rule.

Again, both the Agent and Gateway should display debug information, including the span you just sent. This time, the Gateway will write a line to the gateway-traces-security.out file, which is designated for spans where the deployment.environment resource attribute matches "security_applications". The gateway-traces-standard.out should be unchanged.

Tip

If you verify the ./gateway-traces-security.out it should only contain the spans from the "security_applications" deployment.environment.

You can repeat this scenario multiple times, and each trace will be written to its corresponding output file.

Conclusion

In this section, we successfully tested the routing connector in the gateway by sending different spans and verifying their destinations.

  • Regular spans were correctly routed to gateway-traces-standard.out, confirming that spans without a matching deployment.environment attribute follow the default pipeline.

  • Security-related spans from security.json were routed to gateway-traces-security.out, demonstrating that the routing rule based on "deployment.environment": "security_applications" works as expected.

By inspecting the output files, we confirmed that the OpenTelemetry Collector correctly evaluates span attributes and routes them to the appropriate destinations. This validates that routing rules can effectively separate and direct telemetry data for different use cases.

You can now extend this approach by defining additional routing rules to further categorize spans, metrics, and logs based on different attributes.

Stop the Agent, Gateway and the log-gen script in their respective terminals.