AWS CloudTrail Logs: Reduce log size¶
Disclaimer
By using SPL2 templates for data processing (the “templates”), you understand and agree that templates are provided “as is”. Splunk disclaims any and all warranties, express or implied, including without limitation the implied warranties of merchantability, fitness for a particular purpose and warranties arising out of course of dealing or usage of trade or by statute or in law. Splunk specifically does not warrant that templates will meet your requirements, the operation or output of templates will be error-free, accurate, reliable, complete or uninterrupted.
Use case¶
Reduce the size of AWS CloudTrail logs by removing unnecessary fields while preserving compatibility with the Splunk Common Information Model (CIM) and Security Detections.
Template details¶
Compatibility¶
This template is compatible with Splunk Add-on for Amazon Web Services v7.10.0.
Template description¶
This pipeline reduces the size of AWS CloudTrail events while preserving compatibility with the Splunk Common Information Model (CIM) and Security Detections. The pipeline processes CloudTrail events and removes numerous unnecessary fields that are commonly unused by Splunk. The template also creates two data streams:
- Full events: Cloned events with all original data are sent to a low priority destination (such as an S3 bucket for archival purposes)
- Reduced events: Events with unnecessary fields removed are sent to the primary destination for active monitoring and analysis
Supported sourcetypes¶
This template processes AWS CloudTrail events - of sourcetype aws:cloudtrail. Events not matching the partition conditions are passed through without processing.
Template outline¶
The template consists of a custom function followed by a pipeline that uses this function.
Functions¶
The following table shows the custom function used in this template.
| Function name | Description |
|---|---|
cleanup |
This function cleans up fields that are frequently left as null or empty after the reduction. |
*_common |
These functions remove fields that are present in different types of events. |
shared_group_* |
Same as above, but with big chunks specific to few events. |
r_* |
All of the below “r_” functions remove unused by Splunk fields that are present in “” type of event |
Pipeline¶
This pipeline has the following stages:
- Clone and archive: Creates a copy of the original event and sends it to the low priority destination
- Determine event type: Based on the field “eventName” the pipeline determines what function should be used for fields reduction
- Field removal: The selected function removes the fields specific to that event type, and/or some common fields that are present
- Cleanup: A cleanup function is run on the event, removing empty/null fields
- Route processed events: Sends the reduced events to the primary destination
Configuration instructions¶
The r_* functions remove a comprehensive list of fields that are typically not used in Splunk analytics or security detections. These include:
- Edge device details and insight information
- Verbose request parameters for various AWS services
- Detailed response elements with redundant metadata
- Empty response elements and additional event data
- Session and authentication details that don’t impact security analysis
If you need to retain specific fields that are being removed, you can modify the json_delete function call in the relevant r_* functions or *_common functions and remove those field names from the deletion list.
To delete additional fields, one can add additonal elements in json_delete call in appropriate r_*. Or if the fields need to be removed from all incoming events, we can modify cleanup function and add/modify json_delete call there.