Windows event logs: Convert logs from XML to JSON

Disclaimer: BY USING SPL2 TEMPLATES FOR DATA PROCESSING (THE “TEMPLATES”), YOU UNDERSTAND AND AGREE THAT TEMPLATES ARE PROVIDED “AS IS”. SPLUNK DISCLAIMS ANY AND ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND WARRANTIES ARISING OUT OF COURSE OF DEALING OR USAGE OF TRADE OR BY STATUTE OR IN LAW. SPLUNK SPECIFICALLY DOES NOT WARRANT THAT TEMPLATES WILL MEET YOUR REQUIREMENTS, THE OPERATION OR OUTPUT OF TEMPLATES WILL BE ERROR-FREE, ACCURATE, RELIABLE, COMPLETE OR UNINTERRUPTED.

Use case

Convert Windows event logs from XML to JSON, reduce the size of the logs by removing unnecessary data, and extract event fields to ensure compatibility with the Splunk Add-on for Microsoft Windows and the Splunk Common Information Model (CIM).

Template details

Compatibility

This template is compatible with Splunk Add-on for Microsoft Windows v8.9.0 and v9.0.1.

Template description

This is a sample pipeline that converts Windows event logs from XML to JSON and prepares the logs for downstream processing by the Splunk Add-on for Microsoft Windows and the Splunk Common Information Model (CIM). This pipeline takes data that has the XmlWinEventLog source type and then does the following:

1. Transforms the logs from XML format to JSON format.
2. Extracts the top-level event fields required by the Splunk Add-on for Microsoft Windows and the CIM.
3. Removes the extracted fields from the JSON-formatted log data to avoid duplicating data.

Supported sourcetype

This template partitions by sourcetype matching the following regex: /XmlWinEventLog/i. The pipeline will process any data having this sourcetype.

Requirements

To ensure the Splunk Add-on for Microsoft Windows works correctly after the logs are processed, install or upload the custom app “Z_Splunk_TA_windows_override” to your destination Splunk instance.

Create the app

Perform the following steps to create the app:

1. Create a directory named “Z_Splunk_TA_windows_override”.
2. In that directory, create the following 3 files with the provided content. - Z_Splunk_TA_windows_override/default/app.conf

   [install]
   is_configured = 0
   
   [ui]
   is_visible = 0
   label = Z_Splunk_TA_windows_override
   
   [launcher]
   author = Splunk
   description = This application changes KV_MODE to auto for `source::XmlWinEventLog...` to handle SPL2 pipeline changes.
   version = 0.0.1
   

   - Z_Splunk_TA_windows_override/default/props.conf

   [source::XmlWinEventLog...]
   KV_MODE = auto
   priority = 1
   

   - Z_Splunk_TA_windows_override/metadata/default.meta

   ## shared Application-level permissions
   []
   access = read : [ * ], write : [ admin, sc_admin ]
   export = system
   

3. Package the app using one of the following commands: - For Unix systems: COPYFILE_DISABLE=1 tar --format ustar -cvzf Z_Splunk_TA_windows_override.tar.gz Z_Splunk_TA_windows_override - For Windows: tar --format ustar -cvzf Z_Splunk_TA_windows_override.tar.gz Z_Splunk_TA_windows_override

You can install or upload the “Z_Splunk_TA_windows_override” app to your destination Splunk platform instance.

Template outline

Template consists of several custom functions followed by a pipeline that uses these functions.

Functions

The following table shows all avilable functions.

Function name	Description
preprocess_xml_rows	This function removes the newline characters between the XML elements, removes non-essential text such as the ending tags of elements, and then filters out empty rows.
process_system	This function converts the elements that are nested inside the \ element into key-value pairs, and then additionally extracts the parameters from these nested elements into key-value pairs.
process_event_data	This function converts the elements that are nested inside the \ element into key-value pairs.
process_user_data	This function converts the elements that are nested inside the \ element into key-value pairs.
extract_inner_fields	This function groups the previously defined custom functions together, in order to provide a single function that extracts key-value pairs from innermost XML tags.
reduce_groups_to_json	This function converts the arrays of key-value pairs that were extracted from the \, \, and \ XML elements into a single JSON object.
flatten_to_event	This function converts all the key-value pairs in the JSON object into top-level event fields.
keep_fields_for_TA_transforms	This function keeps the top-level event fields that the Splunk Add-on for Microsoft Windows requires for index-time extractions, and drops all other fields.
serialize_to_json_raw	This function cleans up the original log by removing the data that has been extracted into top-level event fields, and then serializes the updated log into JSON format and stores it in the _raw field.
ta_compatibility_fixes	This function extracts additional event fields in order to make the logs compatible with the extractions done by the Splunk Add-on for Microsoft Windows and the CIM.
transform_xml_event	This function groups the previously defined custom functions together, to provide a single function that transforms the logs from XML to JSON, reduces the log sizes, and extracts the necessary fields.

Pipeline

The pipeline has the following outline:

1. Extracts inner fields using the extract_inner_fields function.
2. Extracts key-value pairs at top-level using the flatten_to_event function.
3. Cleans up the extracted event fields and serializes the updated log into JSON format using the serialize_to_json_raw function.
4. Extracts additional event fields to make the logs compatible with the extractions done by the TA and CIM using the ta_compatibility_fixes function.
5. Sends the transformed logs to the destination.