UNIX and Linux bandwidth logs: Reduce log size and convert to TSV format¶

Disclaimer

By using SPL2 templates for data processing (the “templates”), you understand and agree that templates are provided “as is”. Splunk disclaims any and all warranties, express or implied, including without limitation the implied warranties of merchantability, fitness for a particular purpose and warranties arising out of course of dealing or usage of trade or by statute or in law. Splunk specifically does not warrant that templates will meet your requirements, the operation or output of templates will be error-free, accurate, reliable, complete or uninterrupted.

Use case¶

Reduce the size of Unix and Linux bandwidth logs by removing unnecessary fields and converting logs into a tab-separated values (TSV) format while maintaining compatibility with the Splunk Common Information Model (CIM).

Version 0.2.1¶

Version 0.2.1 of the UNIX and Linux bandwidth logs: Reduce log size and convert to TSV format template was released on November 20, 2024.

Template details¶

Compatibility¶

This template is compatible with Splunk Add-on for Unix and Linux v9.2.0 and v10.0.0.

Template description¶

This is a sample pipeline that reduces the size of bandwidth logs coming from the Splunk Add-on for Unix and Linux while preserving compatibility with the Splunk Common Information Model (CIM). This pipeline takes data that has the bandwidth source type and then does the following:

Replaces the repeated space characters used as delimiters in the logs with a different string. By default, a tab character (\t) is used.
Removes unnecessary data from the logs.
Optionally updates the values of the source and sourcetype event fields.

Supported sourcetype¶

This pipeline only works on complete bandwidth events that include the header row and all subsequent rows.

Template outline¶

The template consists of several custom functions followed by a pipeline that uses these functions.

Functions¶

The following table shows all functions, including possible configuration options.

Function name	Description	Configuration options
`replace_multiple_whitespaces`	This function changes the delimiter in the logs from repeated space characters to a single tab character (`\t`). You can choose a different replacement delimiter by configuring the $delimiter parameter in the function definition.	`$delimiter`: The replacement delimiter to use instead of repeated space characters (default: `\t`).
`remove_trailing_zeros`	This function removes trailing zeros from decimal values in the logs. For example, `10.00` is transformed into `10`.	No configuration options.
`update_source_and_source_type`	This function appends a suffix to the values in the `source` and `sourcetype` fields, if those fields exist in the event.	`$suffix`: The suffix to append to the `source` and `sourcetype` values.

Pipeline¶

The pipeline has the following outline:

Replaces delimiters using the replace_multiple_whitespaces function.
Removes trailing zeros using the remove_trailing_zeros function.
Optionally updates the source and sourcetype fields using the update_source_and_source_type function.
Sends the transformed logs to the destination.

Configuration instructions¶

Use the remove_trailing_zeros function to reduce log size without losing significant information. This approach ensures that minimally filtered data continues to flow through the pipeline with minimal impact on license usage.

Use the update_source_and_source_type function to append a suffix to the source and sourcetype fields. This helps distinguish modified logs from unmodified ones. The source and sourcetype fields must already exist in the event before you use this function.

Notes¶

Ensure that the bandwidth source type is properly configured in your Splunk deployment.
Customize the pipeline as needed to meet your specific requirements.
If using the update_source_and_source_type function, ensure that the suffixed source type is configured in your Splunk platform deployment with appropriate time extraction and line-breaking settings.