SC4S parsers

Prerequisites

Before reading this section, make sure you are familiar with Sources.

This and subsequent sections describe how to create new parsers. SC4S parsers perform operations that would normally be performed during index time, including line-breaking, source and sourcetype setting. You can write your own parser if the parsers available in the SC4S package do not meet your needs or you want to add support for a new sourcetype.

Before you start

• Make sure you have read our contribution standards.
• Obtain a raw log message that you want to parse. If you do not know how to do it, refer to Obtain raw message events.
• Prepare your testing environment. With Python>=3.11.0:

pip3 install poetry
poetry install

Parsers

Naming conventions and project structure

Parsers are .conf files with the naming convention: app-type-vendor_product.conf. Parsers that are part of the repository can be found at package/etc/conf.d/conflib or package/lite/etc/addons for Lite package.

Remember that adding your parser to the main or lite package in the repo requires building a new image for it to become available to your SC4S instance. If you want to add locally new parser, you can add it to /opt/sc4s/local directory on your existing SC4S installation.

Parser structure

The SC4S parser consists of application and block parser blocks. The application part uses filter clause to specify what logs will be parsed by the block parser block. Example of such parser is shown below:

block parser app-syslog-vmware_cb-protect() {
    channel {
        rewrite {
            r_set_splunk_dest_default(
                index("epintel")
                sourcetype('vmware:cb:protect')
                vendor("vmware")
                product("cb-protect")
                template("t_msg_only")
            );
        };
    };
};
application app-syslog-vmware_cb-protect[sc4s-syslog] {
    filter {
        message('Carbon Black App Control event:  '  type(string)  flags(prefix));
    };  
    parser { app-syslog-vmware_cb-protect(); };
};

Note

If you find a similar parser in SC4S, you can use it as a reference. In the parser, make sure you assign the proper sourcetype, index, vendor, product, and template. The template shows how your message should be parsed before sending them to Splunk.

The application filter will match all messages that start with the string Carbon Black App Control event:, and those events will be parsed by block parser app-syslog-vmware_cb-protect(). This parser then will route the message to index: epintel, set the sourcetype, source, vendor and product fields, and use the specified template.

To learn more about creating filters and parse blocks see pages: Filter Messages and Parse Messages.

Adding parser to SC4S Lite package

For SC4S lite, parsers are grouped into addons. Create a folder (if it does not already exist) in package/lite/etc/addons with the name of vendor. In this folder, also create an addon_metadata.yaml file with vendor name:

---
name: "<vendor_name>"

Lastly, add this addon to package/lite/etc/config.yaml.