Palo Alto Network traffic logs: Generate metrics from logs¶

Use case¶

Generate metrics with dimensions from Palo Alto Network traffic logs, and route the metrics and the original logs to two different destinations.

Template details¶

Template description¶

This is a sample pipeline that generates metrics from Palo Alto Network traffic logs. It takes data that has the pan:traffic source type and then does the following: 1. Generates the following four metrics: bytes_out, bytes_in, packets_out, packets_in. Each metric includes the log type and associated app as dimensions. 2. Sends the metrics to the destination denoted by $metrics_destination. 3. Sends the original unchanged logs to the destination denoted by $destination.

See the follwoing documentation for more information about generating logs into metrics:

Generate logs into metrics using Ingest Processor

Metrics types

Template outline¶

This template pipeline has the following stages: 1. zipping values form the log with keys to produce JSON 2. extract time and some other fields from the log which are required for metrics (values and dimensions) 3. split the log into two streams: one for metrics and one for logs - use logs_to_metrics function to generate metrics for bytes_in - use logs_to_metrics function to generate metrics for bytes_out 4. send the unmodified logs to the destination denoted by $destination (this can be S3, Splunk, or any other destination)

Configuration options¶

This pipeline uses the following configuration options: - add more dimensions to the metrics - add more metrics - route logs to cheaper storage - Amazon S3

Configuration example scenarios¶

Scenario 1: Add more dimensions to metrics¶

You can extract dimensions from the log using the following code: | eval _app = _result.app and added them to the logs_to_metrics function. To add more dimensions to the metrics, you can add additional values to the logs_to_metrics parameter called dimensions.

Perform the following steps to add more dimensions:

Extract new variable from the _raw
Add the new variable to the logs_to_metrics function as a dimension.
Remove newly extracted fields from the log before sending it to the destination.
Save the changes.

Scenario 2: Add more metrics¶

You can extract metric values from the log using the following code: | eval _bytes_out_num = tonumber(_result.bytes_out). Use a separate thru command for each metric to copy the event to separate branch, and routes it to the metrics destination. To add more metrics, follow the same approach used for the two metrics already implemented in the pipelines.

See the Metric types for more information.

Perofrm the following steps to add more metrics:

Extract new metric: | eval _my_new_metric = tonumber(_result.some_field) | thru [ | logs_to_metrics name="new_metric" metrictype="gauge" value=_bytes_out_num time=_time dimensions={"aType": _type, "app": _app} | into $metrics_destination ]
Remove temporary fields from the log before sending it to the destination (by using like fields - _my_new_metric )
Save the changes.