4. Redacting Sensitive Data

10 minutes

In this section, you’ll learn how to configure the OpenTelemetry Collector to remove specific tags and redact sensitive data from telemetry spans. This is crucial for protecting sensitive information such as credit card numbers, personal data, or other security-related details that must be anonymized before being processed or exported.

We’ll walk through configuring key processors in the OpenTelemetry Collector, including:

Attributes Processor: Modifies or removes specific span attributes.
Redaction Processor: Ensures sensitive data is sanitized before being stored or transmitted.

Exercise

Important

Change ALL terminal windows to the 4-sensitive-data directory and run the clear command.

Copy *.yaml from the 3-dropping-spans directory into 4-sensitive-data. Your updated directory structure will now look like this:

.
├── agent.yaml
└── gateway.yaml

4.1 Configuration

In this step, we’ll modify agent.yaml to include the attributes and redaction processors. These processors will help ensure that sensitive data within span attributes is properly handled before being logged or exported.

Previously, you may have noticed that some span attributes displayed in the console contained personal and sensitive data. We’ll now configure the necessary processors to filter out and redact this information effectively.

Attributes:
     -> user.name: Str(George Lucas)
     -> user.phone_number: Str(+1555-867-5309)
     -> user.email: Str(george@deathstar.email)
     -> user.account_password: Str(LOTR>StarWars1-2-3)
     -> user.visa: Str(4111 1111 1111 1111)
     -> user.amex: Str(3782 822463 10005)
     -> user.mastercard: Str(5555 5555 5555 4444)
  {"kind": "exporter", "data_type": "traces", "name": "debug"}

Exercise

Switch to your Agent terminal window and open the agent.yaml file in your editor. We’ll add two processors to enhance the security and privacy of your telemetry data.

1. Add an attributes Processor: The Attributes Processor allows you to modify span attributes (tags) by updating, deleting, or hashing their values. This is particularly useful for obfuscating sensitive information before it is exported.

In this step, we’ll:

Update the user.phone_number attribute to a static value ("UNKNOWN NUMBER").
Hash the user.email attribute to ensure the original email is not exposed.
Delete the user.password attribute to remove it entirely from the span.

  attributes/update:
    actions:                           # Actions
      - key: user.phone_number         # Target key
        action: update                 # Update action
        value: "UNKNOWN NUMBER"        # New value
      - key: user.email                # Target key
        action: hash                   # Hash the email value
      - key: user.password             # Target key
        action: delete                 # Delete the password

2. Add a redaction Processor: The Redaction Processor detects and redacts sensitive data in span attributes based on predefined patterns, such as credit card numbers or other personally identifiable information (PII).

In this step:

We set allow_all_keys: true to ensure all attributes are processed (if set to false, only explicitly allowed keys are retained).
We define blocked_values with regular expressions to detect and redact Visa and MasterCard credit card numbers.
The summary: debug option logs detailed information about the redaction process for debugging purposes.

  redaction/redact:
    allow_all_keys: true               # If false, only allowed keys will be retained
    blocked_values:                    # List of regex patterns to block
      - '\b4[0-9]{3}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b'       # Visa
      - '\b5[1-5][0-9]{2}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}\b'  # MasterCard
    summary: debug                     # Show debug details about redaction

Update the traces Pipeline: Integrate both processors into the traces pipeline. Make sure that you comment out the redaction processor at first (we will enable it later in a separate exercise). Your configuration should look like this:

    traces:
      receivers:
      - otlp
      processors:
      - memory_limiter
      - attributes/update              # Update, hash, and remove attributes
      #- redaction/redact               # Redact sensitive fields using regex
      - resourcedetection
      - resource/add_mode
      - batch
      exporters:
      - debug
      - file
      - otlphttp

4.2 Test Attribute Processor

In this exercise, we will delete the user.account_password, update the user.phone_number attribute and hash the user.email in the span data before it is exported by the Agent.

Exercise

Start the Gateway: In your Gateway terminal window start the Gateway.

../otelcol --config=gateway.yaml

Start the Agent: In your Agent terminal window start the Agent.

../otelcol --config=agent.yaml

Start the Load Generator: In the Loadgen terminal window start the loadgen:

../loadgen -count 1

Check the debug output: For both the Agent and Gateway confirm that user.account_password has been removed, and both user.phone_number & user.email have been updated:

   -> user.name: Str(George Lucas)
   -> user.phone_number: Str(UNKNOWN NUMBER)
   -> user.email: Str(62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287)
   -> payment.amount: Double(51.71)
   -> user.visa: Str(4111 1111 1111 1111)
   -> user.amex: Str(3782 822463 10005)
   -> user.mastercard: Str(5555 5555 5555 4444)

    -> user.name: Str(George Lucas)
    -> user.phone_number: Str(+1555-867-5309)
    -> user.email: Str(george@deathstar.email)
    -> user.password: Str(LOTR>StarWars1-2-3)
    -> user.visa: Str(4111 1111 1111 1111)
    -> user.amex: Str(3782 822463 10005)
    -> user.mastercard: Str(5555 5555 5555 4444)
    -> payment.amount: Double(95.22)

Check file output: Using jq validate that user.account_password has been removed, and user.phone_number & user.email have been updated in gateway-taces.out:

jq '.resourceSpans[].scopeSpans[].spans[].attributes[] | select(.key == "user.password" or .key == "user.phone_number" or .key == "user.email") | {key: .key, value: .value.stringValue}' ./gateway-traces.out

Notice that the user.account_password has been removed, and the user.phone_number & user.email have been updated:

{
  "key": "user.phone_number",
  "value": "UNKNOWN NUMBER"
}
{
  "key": "user.email",
  "value": "62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287"
}

Important

Stop the Agent and the Gateway processes by pressing Ctrl-C in their respective terminals.

4.3 Test Redaction Processor

The redaction processor gives precise control over which attributes and values are permitted or removed from telemetry data.

In this exercise, we will redact the user.visa & user.mastercard values in the span data before it is exported by the Agent.

Exercise

Start the Gateway: In your Gateway terminal window start the Gateway.

../otelcol --config=gateway.yaml

Enable the redaction/redact processor: In the Agent terminal window, edit agent.yaml and remove the # we inserted in the previous exercise.

    traces:
      receivers:
      - otlp
      processors:
      - memory_limiter
      - attributes/update              # Update, hash, and remove attributes
      - redaction/redact               # Redact sensitive fields using regex
      - resourcedetection
      - resource/add_mode
      - batch
      exporters:
      - debug
      - file
      - otlphttp

Start the Agent: In your Agent terminal window start the Agent.

../otelcol --config=agent.yaml

Start the Load Generator: In the Loadgen terminal window start the loadgen:

../loadgen -count 1

Check the debug output: For both the Agent and Gateway confirm the values for user.visa & user.mastercard have been updated. Notice user.amex attribute value was NOT redacted because a matching regex pattern was not added to blocked_values

   -> user.name: Str(George Lucas)
   -> user.phone_number: Str(UNKNOWN NUMBER)
   -> user.email: Str(62d5e03d8fd5808e77aee5ebbd90cf7627a470ae0be9ffd10e8025a4ad0e1287)
   -> payment.amount: Double(69.71)
   -> user.visa: Str(****)
   -> user.amex: Str(3782 822463 10005)
   -> user.mastercard: Str(****)
   -> redaction.masked.keys: Str(user.mastercard,user.visa)
   -> redaction.masked.count: Int(2)

    -> user.name: Str(George Lucas)
    -> user.phone_number: Str(+1555-867-5309)
    -> user.email: Str(george@deathstar.email)
    -> user.password: Str(LOTR>StarWars1-2-3)
    -> user.visa: Str(4111 1111 1111 1111)
    -> user.amex: Str(3782 822463 10005)
    -> user.mastercard: Str(5555 5555 5555 4444)
    -> payment.amount: Double(65.54)

Note

By including summary:debug in the redaction processor, the debug output will include summary information about which matching key values were redacted, along with the count of values that were masked.

     -> redaction.masked.keys: Str(user.mastercard,user.visa)
     -> redaction.masked.count: Int(2)

Check file output: Using jq verify that user.visa & user.mastercard have been updated in the gateway-traces.out.

jq '.resourceSpans[].scopeSpans[].spans[].attributes[] | select(.key == "user.visa" or .key == "user.mastercard" or .key == "user.amex") | {key: .key, value: .value.stringValue}' ./gateway-traces.out

Notice that user.amex has not been redacted because a matching regex pattern was not added to blocked_values:

{
  "key": "user.visa",
  "value": "****"
}
{
  "key": "user.amex",
  "value": "3782 822463 10005"
}
{
  "key": "user.mastercard",
  "value": "****"
}

These are just a couple of examples of how attributes and redaction processors can be configured to protect sensitive data.

Important

Stop the Agent and the Gateway processes by pressing Ctrl-C in their respective terminals.

4. Redacting Sensitive Data

Subsections of 4. Sensitive Data

4.1 Configuration

4.2 Test Attribute Processor

4.3 Test Redaction Processor