6. Redacting Sensitive Data

10 minutes

In this section, you’ll learn how to configure the OpenTelemetry Collector to remove specific tags and redact sensitive data from telemetry spans. This is crucial for protecting sensitive information such as credit card numbers, personal data, or other security-related details that must be anonymized before being processed or exported.

We’ll walk through configuring key processors in the OpenTelemetry Collector, including:

Attributes Processor: Modifies or removes specific span attributes.
Redaction Processor: Ensures sensitive data is sanitized before being stored or transmitted.

Exercise

Inside the [WORKSHOP] directory, create a new subdirectory named 6-sensitive-data.
Next, copy all contents from the 5-dropping-spans directory into 6-sensitive-data.
After copying, remove any *.out and *.log files.
Change all terminal windows to the [WORKSHOP]/6-sensitive-data directory.

Your updated directory structure will now look like this:

WORKSHOP
├── 1-agent
├── 2-gateway
├── 3-filelog
├── 4-resilience
├── 5-dropping-spans
├── 6-sensitive-data
│   ├───checkpoint-dir
│   ├── agent.yaml
│   ├── gateway.yaml
│   ├── health.json
│   ├── log-gen.sh (or .ps1)
│   └── trace.json
└── otelcol

6.1 Configuration

In this step, we’ll modify agent.yaml to include the attributes and redaction processors. These processors will help ensure that sensitive data within span attributes is properly handled before being logged or exported.

Previously, you may have noticed that some span attributes displayed in the console contained personal and sensitive data. We’ll now configure the necessary processors to filter out and redact this information effectively.

<snip>
Attributes:
     -> user.name: Str(George Lucas)
     -> user.phone_number: Str(+1555-867-5309)
     -> user.email: Str(george@deathstar.email)
     -> user.account_password: Str(LOTR>StarWars1-2-3)
     -> user.visa: Str(4111 1111 1111 1111)
     -> user.amex: Str(3782 822463 10005)
     -> user.mastercard: Str(5555 5555 5555 4444)
  {"kind": "exporter", "data_type": "traces", "name": "debug"}

Exercise

Switch to your Agent terminal window. Navigate to the [WORKSHOP]/6-sensitive-data directory and open the agent.yaml file in your editor.

Add an attributes Processor: This processor allows you to update, delete, or hash specific attributes (tags) within spans.
We’ll update the user.phone_number, hash the user.email, and delete the user.account_password:

  attributes/update:               # Processor Type/Name
    actions:                       # List of actions
      - key: user.phone_number     # Target key
        action: update             # Replace value with "UNKNOWN NUMBER"
        value: "UNKNOWN NUMBER"
      - key: user.email            # Hash the email value
        action: hash               
      - key: user.account_password # Remove the password
        action: delete

Add a redaction Processor: This processor will detect and redact sensitive data values based on predefined patterns. We’ll block credit card numbers using regular expressions.

  redaction/redact:               # Processor Type/Name
    allow_all_keys: true          # If false, only allowed keys will be retained
    blocked_values:               # List of regex patterns to hash
      - '\b4[0-9]{3}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b'  # Visa card
      - '\b5[1-5][0-9]{2}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b'  # MasterCard
    summary: debug  # Show debug details about redaction

Update the traces Pipeline: Integrate both processors into the traces pipeline. Make sure that you comment out the redaction processor at first: (We will enable it later)

    traces:
      receivers:
      - otlp                      # OTLP Receiver
      processors:
      - memory_limiter            # Manage memory usage
      - attributes/update         # Update, hash, and remove attributes
      #- redaction/redact          # Redact sensitive fields using regex
      - resourcedetection         # Add system attributes
      - resource/add_mode         # Add metadata about collector mode
      - batch                     # Batch Processor, groups data before send
      exporters:
      - debug                     # Debug Exporter
      - otlphttp                  # OTLP/HTTP EXporter used by Splunk O11Y

Validate the agent configuration using otelbin.io. For reference, the traces: section of your pipelines will look similar to this:

%%{init:{"fontFamily":"monospace"}}%%
graph LR
    %% Nodes
      REC1(&nbsp;&nbsp;otlp&nbsp;&nbsp;<br>fa:fa-download):::receiver
      PRO1(memory_limiter<br>fa:fa-microchip):::processor
      PRO2(resourcedetection<br>fa:fa-microchip):::processor
      PRO3(resource<br>fa:fa-microchip<br>add_mode):::processor
      PRO5(batch<br>fa:fa-microchip):::processor
      PRO6(attributes<br>fa:fa-microchip<br>update):::processor
      EXP1(otlphttp<br>fa:fa-upload):::exporter
      EXP2(&ensp;&ensp;debug&ensp;&ensp;<br>fa:fa-upload):::exporter
    %% Links
    subID1:::sub-traces
    subgraph " "
      subgraph subID1[**Traces**]
      direction LR
      REC1 --> PRO1
      PRO1 --> PRO6
      PRO6 --> PRO2
      PRO2 --> PRO3
      PRO3 --> PRO5
      PRO5 --> EXP2
      PRO5 --> EXP1
      end
    end
classDef receiver,exporter fill:#8b5cf6,stroke:#333,stroke-width:1px,color:#fff;
classDef processor fill:#6366f1,stroke:#333,stroke-width:1px,color:#fff;
classDef con-receive,con-export fill:#45c175,stroke:#333,stroke-width:1px,color:#fff;
classDef sub-traces stroke:#fbbf24,stroke-width:1px, color:#fbbf24,stroke-dasharray: 3 3;

6.2 Test Attribute Processor

In this exercise, we will delete the user.account_password, update the user.phone_number attribute and hash the user.email in the span data before it is exported by the Agent.

Exercise

Start the Gateway: In the Gateway terminal window navigate to the [WORKSHOP]/6-sensitive-data directory and run:

../otelcol --config=gateway.yaml

Start the Agent: In the Agent terminal window navigate to the [WORKSHOP]/6-sensitive-data directory and run:

../otelcol --config=agent.yaml

Send a span:

In the Test terminal window change into the 6-sensitive-data directory.
Send the span containing sensitive data by running the curl command to send trace.json.

curl -X POST -i http://localhost:4318/v1/traces -H "Content-Type: application/json" -d "@trace.json"

Check the debug output: For both the Agent and Gateway debug output, confirm that user.account_password has been removed, and both user.phone_number & user.email have been updated.

     -> user.name: Str(George Lucas)
     -> user.phone_number: Str(UNKNOWN NUMBER)
     -> user.email: Str. (62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287)
     -> user.mastercard: Str(5555 5555 5555 4444)
     -> user.visa: Str(4111 1111 1111 1111)
     -> user.amex: Str(3782 822463 10005)

      -> user.name: Str(George Lucas)
      -> user.phone_number: Str(+1555-867-5309)
      -> user.email: Str(george@deathstar.email)
      -> user.account_password: Str(LOTR>StarWars1-2-3)
      -> user.mastercard: Str(5555 5555 5555 4444)
      -> user.visa: Str(4111 1111 1111 1111)
      -> user.amex: Str(3782 822463 10005)

Check file output: In the new gateway-traces.out file confirm that user.account_password has been removed, and user.phone_number & user.email have been updated:

"attributes": [
              {
                "key": "user.name",
                "value": {
                  "stringValue": "George Lucas"
                }
              },
              {
                "key": "user.phone_number",
                "value": {
                  "stringValue": "UNKNOWN NUMBER"
                }
              },
              {
                "key": "user.email",
                "value": {
                  "stringValue": "62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287"
                }
              },
              {
                "key": "user.mastercard",
                "value": {
                  "stringValue": "5555 5555 5555 4444"
                }
              },
              {
                "key": "user.visa",
                "value": {
                  "stringValue": "4111 1111 1111 1111"
                }
              },
              {
                "key": "user.amex",
                "value": {
                  "stringValue": "3782 822463 10005"
                }
              } 
            ]

"attributes": [
              {
                "key": "user.name",
                "value": {
                  "stringValue": "George Lucas"
                }
              },
              {
                "key": "user.phone_number",
                "value": {
                  "stringValue": "+1555-867-5309"
                }
              },
              {
                "key": "user.email",
                "value": {
                  "stringValue": "george@deathstar.email"
                }
              },
              {
                "key": "user.account_password",
                "value": {
                  "stringValue": "LOTR>StarWars1-2-3"
                }
              },
              {
                "key": "user.mastercard",
                "value": {
                  "stringValue": "5555 5555 5555 4444"
                }
              },
              {
                "key": "user.visa",
                "value": {
                  "stringValue": "4111 1111 1111 1111"
                }
              },
              {
                "key": "user.amex",
                "value": {
                  "stringValue": "3782 822463 10005"
                }
              } 
            ]

Stop the Agent and Gateway using Ctrl-C.

6.3 Test Redaction Processor

The redaction processor gives precise control over which attributes and values are permitted or removed from telemetry data.

Earlier we configured the agent collector to:

Block sensitive data: Any values (in this case Credit card numbers) matching the provided regex patterns (Visa and MasterCard) are automatically detected and redacted.

This is achieved using the redaction processor you added earlier, where we define regex patterns to filter out unwanted data:

  redaction/redact:               # Processor Type/Name
    allow_all_keys: true          # False removes all key unless in allow list 
    blocked_values:               # List of regex to check and hash
        # Visa card regex.  - Please note the '' around the regex
      - '\b4[0-9]{3}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b'
        # MasterCard card regex - Please note the '' around the regex
      - '\b5[1-5][0-9]{2}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b' 
    summary: debug  # Show detailed debug information about the redaction

Test the Redaction Processor

In this exercise, we will redact the user.visa & user.mastercard values in the span data before it is exported by the Agent.

Exercise

Prepare the terminals: Delete the *.out files and clear the screen.

Enable the redaction/redact processor: Edit agent.yaml and remove the # we inserted in the previous exercise.

Start the Gateway: In the Gateway terminal window navigate to the [WORKSHOP]/6-sensitive-data directory and run:

../otelcol --config=gateway.yaml

Start the Agent: In the Agent terminal window navigate to the [WORKSHOP]/6-sensitive-data directory and run:

../otelcol --config=agent.yaml

Send a span: Run the curl command and in the Test terminal window to send trace.json.

curl -X POST -i http://localhost:4318/v1/traces -H "Content-Type: application/json" -d "@trace.json"

Check the debug output: For both the Agent and Gateway confirm the values for user.visa & user.mastercard have been updated. Notice user.amex attribute value was NOT redacted because a matching regex pattern was not added to blocked_values

     -> user.name: Str(George Lucas)
     -> user.phone_number: Str(UNKNOWN NUMBER)
     -> user.email: Str. (62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287)
     -> user.mastercard: Str(****)
     -> user.visa: Str(****)
     -> user.amex: Str(3782 822463 10005)
     -> redaction.masked.keys: Str(user.mastercard,user.visa)
     -> redaction.masked.count: Int(2)

      -> user.name: Str(George Lucas)
      -> user.phone_number: Str(+1555-867-5309)
      -> user.email: Str(george@deathstar.email)
      -> user.account_password: Str(LOTR>StarWars1-2-3)
      -> user.mastercard: Str(5555 5555 5555 4444)
      -> user.visa: Str(4111 1111 1111 1111)
      -> user.amex: Str(3782 822463 10005)

Tip

By including summary:debug in the redaction processor, the debug output will include summary information about which matching keys values were redacted, along with the count of values that were masked.

     -> redaction.masked.keys: Str(user.mastercard,user.visa)
     -> redaction.masked.count: Int(2)

Check file output: In the newly created gateway-traces.out file to verify confirm that user.visa & user.mastercard have been updated.

"attributes": [
              {
                "key": "user.name",
                "value": {
                  "stringValue": "George Lucas"
                }
              },
              {
                "key": "user.phone_number",
                "value": {
                  "stringValue": "UNKNOWN NUMBER"
                }
              },
              {
                "key": "user.email",
                "value": {
                  "stringValue": "62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287"
                }
              },
              {
                "key": "user.mastercard",
                "value": {
                  "stringValue": "****"
                }
              },
              {
                "key":"user.visa",
                "value":{
                  "stringValue":"****"
                  }
               },
              {
                "key":"user.amex",
                "value":{
                  "stringValue":"3782 822463 10005"
                  }
               }
            ]

"attributes": [
              {
                "key": "user.name",
                "value": {
                  "stringValue": "George Lucas"
                }
              },
              {
                "key": "user.phone_number",
                "value": {
                  "stringValue": "+1555-867-5309"
                }
              },
              {
                "key": "user.email",
                "value": {
                  "stringValue": "george@deathstar.email"
                }
              },
              {
                "key": "user.account_password",
                "value": {
                  "stringValue": "LOTR>StarWars1-2-3"
                }
              },
              {
                "key": "user.mastercard",
                "value": {
                  "stringValue": "5555 5555 5555 4444"
                }
              },  
              {
                "key": "user.visa",
                "value": {
                  "stringValue": "4111 1111 1111 1111"
                }
              },
              {
                "key":"user.amex",
                "value":{
                  "stringValue":"3782 822463 10005"
                  }
               }
            ]

These are just a few examples of how attributes and redaction processors can be configured to protect sensitive data.

Stop the Agent and Gateway using Ctrl-C.

6. Redacting Sensitive Data

Subsections of 6. Sensitive Data

6.1 Configuration

6.2 Test Attribute Processor

6.3 Test Redaction Processor

Test the Redaction Processor