AWS VPC Flow logs: Filter unwanted logs and generate metrics from logs¶
Disclaimer: BY USING SPL2 TEMPLATES FOR DATA PROCESSING (THE “TEMPLATES”), YOU UNDERSTAND AND AGREE THAT TEMPLATES ARE PROVIDED “AS IS”. SPLUNK DISCLAIMS ANY AND ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND WARRANTIES ARISING OUT OF COURSE OF DEALING OR USAGE OF TRADE OR BY STATUTE OR IN LAW. SPLUNK SPECIFICALLY DOES NOT WARRANT THAT TEMPLATES WILL MEET YOUR REQUIREMENTS, THE OPERATION OR OUTPUT OF TEMPLATES WILL BE ERROR-FREE, ACCURATE, RELIABLE, COMPLETE OR UNINTERRUPTED.
Use case¶
Generate metrics with dimensions from AWS VPC Flow logs, and then route the metrics with dimensions to destination.
Version¶
0.0.1
Template details¶
Compatibility¶
This template is compatible with Splunk Add-on for Amazon Web Services v7.10.0
Template description¶
This is a sample pipeline that generates metrics and dimensions from AWS VPC Flow logs.
- Extract the
log_type
of VPC Flow logs. For example:default
,custom
andselect_all
. - Generates the following metrics:
bytes
,packets
. Extract different dimensions based onlog_type
. - Sends the metric logs along with dimensions to the destination denoted by
$destination
.
Supported sourcetype¶
This template partitions data by sourcetype aws:cloudwatchlogs:vpcflow:metric
which means that events having this pipeline processes events having this exact sourcetype. Events not marching this sourcetype are passed through and the pipeline is not applied.
Template outline¶
Functions¶
See the following table for functions description.
Function name | Description |
---|---|
zip_with_keys |
This custom function transforms the _raw field into a multivalue field by splitting it on spaces and then constructs a JSON array by assigning keys to each value, storing the result in the _results. field. |
extract_fields_default |
Extract the fields using zip_with_keys and flattens for default log format. |
extract_fields_select_all |
Extract the fields using zip_with_keys and flatten for select_all log format. |
extract_fields_custom |
Extract the fields using zip_with_keys and flattens for custom log format. |
filter_events |
This custom function drops the events with log_status equals to “NODATA” or “SKIPDATA” or “log-status”. |
extract_log_type |
Extract the log_type field from the raw event in order to identify the log format (Default, Custom or Select_all). |
convert_logs_to_metric |
This custom function converts the normal field to a metric field. |
apply_transformations |
This custom functions applies transformations (mentioned in the template description) over the data. |
Pipeline¶
This template contains the following pipeline stages:
- Extract log_type. (For example,
default
,custom
andselect_all
) - Branch based on log_type.
- For each log_type:
- zipping values from the log with keys to produce JSON.
- drops the events with log_status equals to
NODATA
,SKIPDATA
orlog-status
. - extract log_type specific dimensions.
- extract these two metrics:
bytes
andpackets
.
- send the metrics logs along with dimensions to the destination denoted by
$destination
. This can be S3, Splunk, or any other destination.