Palo Alto Network traffic logs: Generate metrics from logs

Use case

Generate metrics with dimensions from Palo Alto Network traffic logs, and route the metrics and the original logs to two different destinations.

Template details

Template description

This is a sample pipeline that generates metrics from Palo Alto Network traffic logs. It takes data that has the pan:traffic source type and then does the following:

1. Generates the following four metrics: bytes_out, bytes_in, packets_out, packets_in. Each metric includes the log type and associated app as dimensions.
2. Sends the metrics to the destination denoted by $metrics_destination.
3. Sends the original unchanged logs to the destination denoted by $destination.

See the following documentation for more information about generating logs into metrics:

• Generate logs into metrics using Ingest Processor

• Metric types

Template outline

This template pipeline has the following stages:

1. Zipps values form the log with keys to produce JSON
2. Extracts time and some other fields from the log which are required for metrics (values and dimensions)
3. Splits the log into two streams: one for metrics and one for logs: - Use logs_to_metrics function to generate metrics for bytes_in - Use logs_to_metrics function to generate metrics for bytes_out
4. Sends the unmodified logs to the destination denoted by $destination (this can be S3, Splunk, or any other destination)

Configuration options

This pipeline uses the following configuration options:

• Add more dimensions to the metrics
• Add more metrics
• Route logs to cheaper storage - Amazon S3

Configuration example scenarios

Scenario 1: Add more dimensions to metrics

You can extract dimensions from the log using the following code: | eval _app = _result.app and added them to the logs_to_metrics function. To add more dimensions to the metrics, you can add additional values to the logs_to_metrics parameter called dimensions.

Perform the following steps to add more dimensions:

1. Extract new variable from the _raw
2. Add the new variable to the logs_to_metrics function as a dimension.
3. Remove newly extracted fields from the log before sending it to the destination.
4. Save the changes.

Scenario 2: Add more metrics

You can extract metric values from the log using the following code: | eval _bytes_out_num = tonumber(_result.bytes_out). Use a separate thru command for each metric to copy the event to separate branch, and routes it to the metrics destination. To add more metrics, follow the same approach used for the two metrics already implemented in the pipelines.

See the Metric types for more information.

Perform the following steps to add more metrics:

1. Extract new metric:

   | eval _my_new_metric = tonumber(_result.some_field) | thru [ | logs_to_metrics name="new_metric" metrictype="gauge" value=_bytes_out_num time=_time dimensions={"aType": _type, "app": _app} | into $metrics_destination ]

2. Remove temporary fields from the log before sending it to the destination (by using fields - _my_new_metric).

3. Save the changes.