Skip to content

AWS VPC Flow logs: Filter unwanted logs and generate metrics from logs

Disclaimer

By using SPL2 templates for data processing (the “templates”), you understand and agree that templates are provided “as is”. Splunk disclaims any and all warranties, express or implied, including without limitation the implied warranties of merchantability, fitness for a particular purpose and warranties arising out of course of dealing or usage of trade or by statute or in law. Splunk specifically does not warrant that templates will meet your requirements, the operation or output of templates will be error-free, accurate, reliable, complete or uninterrupted.

Use case

Generate metrics with dimensions from AWS VPC Flow logs, and then route the metrics with dimensions to destination.

Template details

Compatibility

This template is compatible with Splunk Add-on for Amazon Web Services v7.10.0.

Template description

This is a sample pipeline that generates metrics and dimensions from AWS VPC Flow logs.

  1. Extracts the log_type of VPC Flow logs. For example: default,custom and select_all.
  2. Generates the following metrics: bytes, packets. Extract different dimensions based on log_type.
  3. Sends the metric logs along with dimensions to the destination denoted by $destination.

Supported sourcetype

This template partitions data by sourcetype aws:cloudwatchlogs:vpcflow:metric which means that events having this pipeline processes events having this exact sourcetype. Events not marching this sourcetype are passed through and the pipeline is not applied.

Template outline

Functions

See the following table for functions description.

Function name Description
zip_with_keys This custom function transforms the _raw field into a multi-value field by splitting it on spaces and constructs a JSON array by assigning keys to each value, storing the result in the _results. field.
extract_fields_default Extracts the fields using zip_with_keys and flattens for default log format.
extract_fields_select_all Extracts the fields using zip_with_keys and flatten for select_all log format.
extract_fields_custom Extracts the fields using zip_with_keys and flattens for custom log format.
filter_events This custom function drops the events with log_status equals to “NODATA” or “SKIPDATA” or “log-status”.
extract_log_type Extracts the log_type field from the raw event in order to identify the log format (Default, Custom or Select_all).
convert_logs_to_metric This custom function converts the normal field to a metric field.
apply_transformations This custom functions applies transformations (mentioned in the template description) over the data.

Pipeline

This template contains the following pipeline stages:

  • Extracts log_type. For example, default,custom and select_all.
  • Branch based on log_type.
  • For each log_type:
    • zips values from the log with keys to produce JSON.
    • drops the events with log_status equals to NODATA,SKIPDATA or log-status.
    • extracts log_type specific dimensions.
    • extracts the following metrics: bytes and packets.
  • sends the metrics logs along with dimensions to the destination denoted by $destination. This can be S3, Splunk, or any other destination.