Splunk for OT Security App (DA-ESS-OTSecurity) Configuration¶

Once the Splunk OT Security Solution app has been installed in your Splunk environment alongside your Splunk Enterprise Security app, you will need to take the following steps to configure the application for production use:

Core Integration Steps¶

Splunk for OT Security comes with navigation menus that can be edited to suit your Enterprise Security deployment. These navigation menus include links to dashboards and that are included in the Splunk for OT Security solution.

To Update the Navigation Menu follow these steps:

Open the Enterprise Security app in your Splunk instance
Go to the Enterprise Security app in Splunk
In the app navigation bar, navigate to the following location:
- Configure→General→Navigation
On the Edit Navigation screen, add existing menus by selecting:
- Add a New Collection→Add Existing→App:DA-ESS-OTSecurity→Select a Collection
The menu containing all the Operation Technology dashboards and reports will now appear. These can be dragged to the desired location in the menu hierarchy or can be modified to fit your organization's needs. For example, the Compliance menu containing NERC CIP dashboards may be removed if your organization is not under NERC CIP regulations.

Step 2: Configure the Asset Framework¶

The Splunk for OT Security Solution extends the ES Asset Framework to provide additional context and information about OT assets. Ideally all fields (including those from the core ES asset framework) are populated, but only the mandatory ones are required. Fields that are mandatory from the core framework include at least one of the following: dns, ip, mac, or nt_host.

To update the asset framework follow these steps:

Go the Enterprise Security app in Splunk
In the app navigation bar go to the following location:
- Configure→Data Enrichment→Asset and Identity Management
Go the Asset Settings table
- Update the Asset Framework by Adding New Fields (field names are case sensitive) as shown here:

Field Name	Tag	Multivalue	Mandatory	Key	Example
asset_id	Yes	No	No	No	189674, FUC456
asset_model	Yes	No	No	No	S7, 1762-OF4
asset_status	Yes	No	No	No	Hot, Ready, Standby
asset_system	Yes	No	Yes	No	Western Operations, Sandusky Plants
asset_type	Yes	No	Yes	No	Jump Server, PLC, Historian
asset_vendor	Yes	No	No	No	Siemens, Allen Bradley
asset_version	Yes	No	No	No	4.23, 7.1.4
classification	Yes	Yes	Yes (for NERC CIP)	No	cip:PCS, cip:ESP
description	No	No	No	No	Vibration for Sandusky Line 3
exposure	Yes	No	No	No	private, public, internal
location	No	Yes	No	No	Sandusky Prod Line 3/Bank 3/Press 2
site_id	Yes	No	Yes	No	Sanford Plant, Copperfield Power Plant
vlan	Yes	Yes	No	No	Sandford SCADA, 172.2.3.x
zone	Yes	Yes	Yes (for NERC CIP)	No	purdue:level3

Enable asset and identity correlation on the Correlation Setup tab and set up to either Enable for all sourcetypes or Enable selectively by sourcetype and supply the required sourcetypes. In most cases, Enable selectively by sourcetype is preferred as it results in less load on the Splunk infrastructure since it only searches for specific data sources and not across all data.

Step 3: Upload Asset Information¶

Now that you have updated the Asset Framework fields, asset and identity lookup files can be uploaded into ES. Note: previous versions of the add-on require specific lookup file names to be used, but this is no longer a requirement.

For NERC CIP use cases ONLY the following fields should have values as indicated here:

Field	Format	Example
classification	cip:<low,medium, or high>\|cip<BCA,PCA,TS A,EACM,EAP>	cip:high\|cip:EAP \|cip:EACM
category	nerc	nerc
zone	eap:<zone name>	eap:PPLT

Important: In order to leverage lookup files from apps outside of Enterprise Security, Lookup Definitions must be created within the Splunk Enterprise Security Suite app context. For more information on managing lookups and knowledge objects within Splunk Enterprise, please refer to the documentation linked at the beginning of this document.

To create the lookup files and link them to the asset & identity framework follow these steps:

Go to Settings→Lookups
Click on Lookup Definitions
Click on New Lookup Definition. This lookup defintion should be linked to the lookup file containing information regarding your assets or identities. In addition, this lookup defintion needs to be shared globally to be acessible in Enterprise Security. This step will need to be repeated if you have multiple asset lookup files and independently for assets and identities.
Open the Enterprise Security App
In the app navigation bar go to the following location: Configure→Data Enrichment→Asset and Identity Management
Go to the Asset Lookup Configuration Tab
Click on + New button and configure your new asset lookup to match the name of your Lookup Definition for Assets
Repeat steps 6 and 7 until you have created new asset configurations for each of your asset lookup definitions
Go to the Identity Lookup Configuration Tab
Click on the + New button and configure you new lookup to the match of your Lookup Definition for Identities in step 3 above
Repeat steps 9 and 10 until you have created new identity configurations for each of your identity lookup definitions.

Macros¶

Macros are leveraged in the Splunk OT Security solution for re-use of searches and so pre-configured indexes, sources, and sourcetypes can be automatically adjusted to represent a specific customer environment.

Note: Macros are designed for efficiency and should only include data sources relevant to the query being performed. Using default or otherwise overly-broad macro definitions may result in slow and process-intensive searches.

Many of the macros below can be updated via the General Configuration menu within Enterprise Security. This can be access within Enterprise Security by going to Configuration→General→General Settings

To update macros manually for the Splunk for OT Security app perform the following steps:

Go to Settings → Advanced Search → Search macros*
Update the following macros to reflect the indexes, sources, and sourcetypes present in your environment. If a data source is not present in your environment it can be modified to a non-existing index and sourcetype to reduce query time.

Macro Name	Purpose	Requires Manual Update
`exclude_internal_ips`	Should contain a subnets which are considered internal to the company	Yes
`get_2fa_indexes`	Should point to data sources relevant to multi-factor authentication (e.g. OKTA, RSA, etc. logs.)	No
`get_asset_type_icon_and_color`	This is used to consistently displaying asset types with specific icons and color	Yes
`get_app_datamodel`	Should point to the data source that contains information on updates and applications being installed (e.g. windows update events)	No
`get_asset_name`	Will determine what field to display by default on dashboards for an asset's name	Yes
`exclude_internal_ips`	Allows a user to designate the default name that should be used in tables and visualizations By default it is set to DNS, host, ip, mac in terms of displaying asset names	Yes
`get_asset_type`	Should contain a subnets which are considered internal to the company	No
`get_asset_type_datamodel`	Should point to the data source that contains information on updates and applications being installed (e.g. windows update events)	No
`get_backup_indexes`	Should point to the data source that contains client backup logs.	No
`get_cve_index`	Should point to the index where CVE and/or CPE definitions are indexed	No
`get_installedapps_datasources`	Should point to the data source which includes all the installed applications about hosts	No
`get_os_datasources`	Should point to the data source which contains OS information about hosts	No
`get_ot_vendors`	Should contain a list of vendors that are present in the environment that are considered OT vendors. This macro comes with a list of common OT vendors such as Siemens, ABB, etc.	No
`get_ot_security_events`	Should contain a filter to identify raw events from OT Security technologies	No
`get_ot_security_notables`	Should contain a filter to identify notable events from ES from OT Security technologies	No
`get_ot_device_asset_types`	Should contain a list of asset types which are considered OT devices and not devices in the OT environment (e.g. PLC's). This macro is pre-populated but should be adjusted to the customer's environment	Yes
`get_ot_networking_devices`	Should contain a list of asset_types or other attributes that identify a networking device as part of the OT environment	Yes
`get_ot_security_alerts`	Should contain the index and/or sources types associated with OT Security solution. This macro is pre-populated with some common source types but should be adjusted based on the customer's OT security solution	No
`get_perimeter_devices`	Should contain a list of asset_types or other attributes that identify a device as part of the OT Security perimeter	No
`get_perimeter_and_networking_devices`	Should contain a list of asset_types or other attributes that identify a device as part of the OT environment	No
`get_physicalaccess_records`	Should contain the index and/or sourcetype where visitor access logs are stored	No
`get_usb_datasources`	Should point to the data source that logs external media devices being connected to a host (e.g., endpoint monitoring, windows registry, etc.)	No
`get_visitoraccess_records`	Should contain the index and/or sourcetype where visitor access logs are stored	No
`label_internal_ips`	Used to identify which is considered internal and external to a organizartion	Yes
`ot_marker_icons`	Gets icons to display for nodes within visualizations for a consistent display across all visualizations	No
`ot_identifier`	Should contain the filter to positively identify an asset that belongs to the OT environment. By default, it is set to look at the category field for the label "ot"	Yes
`prohibited_ot_network_traffic_allowed_filter`	Used to identify data that should be prohibited in the ot environment and leverages several lookups to designate them as prohibited or allowed	Yes

Important Lookup Files¶

KV Store Lookups¶

The Splunk OT Security solution contains two critical kvstore lookups that are leveraged to build baselines for asset configurations. The two tables are linked and timestamped. The critical kv stores and their fields is defined below:

cip_baseline groups

Field	Description
`_key`	Auto-generated by Splunk, this key must be linked to group_id in the system_baselines lookup (similar to a foreign key)
`created_date`	Time in epoch when this particular group was created. Note: new or modified groups should create a new entry so group configurations can be maintained over time
`group_members`	A list of host names that belong to this group, pipe delimited
`group_name`	Name for the group for readability

Note: The cip_ component of this kvstore lookup name can be ignored and is preserved to maintain backwards compatability

system_baselines

Field	Description
`_key`	Auto-generated by Splunk, unique identifier for this baseline
`config`	The actual configuration for this kind of baseline. This is normally json, but if this is not available it is possible to modify search and dashboards to use a different format
`config_asset_type`	Type of asset that this configuration should be applied to - the NERC CIP app currently only uses two values: computer and network (device such as plc). It is possible to store other configuration asset types as needed but the current NERC CIP dashboards only leverage these two
`config_type`	The type of configuration, for example. NERC CIP reports only use the following types: os, applications, patch, port_config. It is possible to store other configuration types (for example, services) as needed
`created_date`	Time in epoch when this particular configuration was created. Note: new configurations should create a new entry so configurations can be maintained over time
`group_id`	This maps this configuration to a specific group in the cip_baseline_groups, telling Splunk that the two are connected. This is an essential field for Splunk to know which configuration to apply to a machine and what parameters to use for checking for deviations

CSV Lookups¶

Common Lookup Files¶

Various lookup tables are essential for populating dashboards with data or for presenting visualizations. The following sections break down each lookup and its intended purpose.

Lookup name: asset_type_mappings

Lookup file that is used to standard the names of asset types as well as provide consistent icons and colors in visualizations.

Lookup name: critical_ot_services.csv

This lookup is used to identify critical services which require notification or may result in loss of operations. One of the MITRE ICS rules requires identifying when critical services have been stopped. The name of the service should match the service name by the operating system. Additional host names or wildcards can be used in the host_names column.

Lookup name: industrial_ports.csv

This lookup contains a list of ports to designate port activities with specific port ranges and industrial applications (e.g. identifying ports associated with a particular SCADA system).

Lookup name: interesting_ot_ports.csv

This lookup is used to label port activity that should and should not be permitted across security boundaries. For example, ports 80 and 443 are often prohibited between OT environments and public networks is typically prohibited but may be permitted between specific IP's on a company's corporate network. CIDR ranges can be used to designate network segments for src and dest networks.

Lookup name: ot_firewall_object_group*

Lookup file designed to hold a listing for object-groups which may be used by firewalls and their definitions.

Lookup name: ot_firewall_rules

Lookup file designed to hold a listing of all firewall rules. Note: could be a superset of the cip_firewall_rules lookup.

Lookup name: prohibited_traffic.csv

Lookup file that is used to designate prohibited and allowed traffic, along with the note describing the app name and why the activity is being allowed or prohibited. This lookup is used in the add-on, but is provided by the SA-NetworkProtection app.

Lookup name: ot_firewall_object_groups.csv

This lookup is used to expand information about object groups contained in firewalls so users do not need to look up object groups. Object groups will often be contained in the ot_firewall_groups.csv.

Lookup name: ot_firewall_rules.csv

This lookup is used to hold firewall rules and annotations for OT firewalls.

NERC CIP Lookup Files¶

Lookup name: cip_ip_ranges.csv

This lookup is used to define subnets that are considered part of NERC CIP OT environments. Subnets can be single IP's or use CIDR notation.

Lookup name: cip_network_configs.csv

This lookup is used to contain information on network devices as well as port and state information. This lookup will normally be populated from network configs, either regularly or statically.

Lookup name: cip_patch_approvals.csv

This lookup is used to contain information about patches and whether they are approved. This data will often be populated from the patching management system (e.g. WSUS). It can also be used to generate baselines.

Lookup name: cip_pra_completion_records

Lookup file with a list of users and when their last personal risk assessment was completed. This lookup is used to verify individuals had a risk assessment completed at least every 15 months.

Lookup name: cip_site_classification

This lookup is used to classify physical security sites and locations and their respective CIP BES classification. While often sites may be classified as a single BES level this lookup provides flexibility to use alternative mechanisms for classification. Note: classifications should follow the naming convention of other lookups to include <regulation>:<classification>.

Lookup name: cip_training_materials

This lookup contains a list of all the training courses and materials that are available, including the title, description, the last time course updates were distributed, and whether the training is required or optional. It also included which groups are available to take the training.

Lookup name: cip_training_records

This lookup functions as a list of courses that have been taken by individuals including when the training was completed and when it needs to be repeated. The course title should be contains in the cip_training_materials lookup.

Lookup name: cip_distribution_lists

This lookup contains a list of distribution groups and the members of each group for cip training. Distribution_list_names are used in the cip_training_materials to identify individuals who would need specific training for NERC CIP compliance. Members of each list are pipe-delimeted.