Common Problems and Troubleshooting¶

While this guide is designed to be as comprehensive as possible, you may run into issues during installation and configuration. The larger body of Splunk Enterprise and Splunk Enterprise Security documentation at the top of this document will help you troubleshoot some of the more common issues encountered during this process. In addition, there are several gotchas that have been encountered during early adoption of the OT Security solution, you can resolve these quickly by double checking the following:

Cannot add lookups to asset or identity framework.​¶

If you cannot add lookups to the Asset and Identity Framework, perform the following checks:

a. Verify the permissions of the DA-ESS-OTSecurity folders and files. These should match other apps installed. Often when a file is manually extracted into the SPLUNK_HOME/etc/apps directory their permissions will be those of the person who manually installed the app. Typically installing via the web gui can avoid these problems. Also check the permissions of the transforms.conf file in the SPLUNK_HOME/etc/apps/EnterpriseSecurity/local directory since lookup definitions are written to this file.

b. Verify that the Lookup Definition has been created in the Enterprise Security App and permissions are set to share objects globally via the Lookup Definitions menu

Assets from lookup table are not showing up¶

Typically, when assets are not showing up in the OT Asset Investigator this is typically an issue with the data in one of the key fields, dns, ip, mac, and/or nt_host. At LEAST one of these fields must have data, should not be set to a default value such as the null string, the IP address must be a correct IPv4 or IPv6 address, and they cannot contain multi-values (the sample search provided in the . Review the data you have provided and validate that none of these contain the "null" string, multiple values, and the ip address is formatted correctly (note CIDR notation is not allowed, and the IPv4 address must contain all four octets). The search provided in the section Integration with OT Security Products step 1, should take care of these problems, and can be adapted to different data sources.

The Installation Guide provides key information on tagging and identification of OT Assets.

Risk based notables not showing up in OT Security Posture¶

Since the OT Add-on for Splunk relies on contextual data around assets to identify assets that are part the OT environment, it is important that macro's regarding risk objects also be utilized to provide the necessary context. When Risk Based Alerts are showing up in the incident review with OT Assets involved, but not the OT Security Posture dashboard, this contextual information may be missing (for example when using the default RBA alerts). To fix this, the query for the panel with Risk Based Alerts can be updated as follows:

Original Search

index=notable risk_object=*
| eval time=_time
| `get_asset_type(risk_object_asset_type)`
| eval key_field=CASE(
match(search_name, ".*(F|f)acility.*"), risk_object_site_id,
match(search_name, ".*(A|s)sset.*"), risk_object,
match(search_name, ".*(U|s)ser.*"), risk_object,
1=1, risk_object)
|  eval object_type=CASE(
match(search_name, ".*(F|f)acility.*"), "Facility",
match(search_name, ".*(O|o)bject.*"), "Facility",
match(search_name, ".*(A|s)sset.*"), "Asset",
match(search_name, ".*(U|s)ser.*"), "User",
1=1, risk_object)
| search `ot_identifier(risk_object_category)` \\\this line causes problems
... \\\rest of search

New Search

index=notable risk_object=*
| eval time=_time
| `get_asset_type(risk_object_asset_type)`
| eval key_field=CASE(
match(search_name, ".*(F|f)acility.*"), risk_object_site_id,
match(search_name, ".*(A|s)sset.*"), risk_object,
match(search_name, ".*(U|s)ser.*"), risk_object,
1=1, risk_object)
|  eval object_type=CASE(
match(search_name, ".*(F|f)acility.*"), "Facility",
match(search_name, ".*(O|o)bject.*"), "Facility",
match(search_name, ".*(A|s)sset.*"), "Asset",
match(search_name, ".*(U|s)ser.*"), "User",
1=1, risk_object)
| `get_asset_by("str", "risk_object")` \\\ this line is added
| search `ot_identifier(risk_object_category)` 
... \\\rest of search

The Installation Guide provides key information on tagging and identification of assets that fall under OT.

Assets and/or Identities are being combined across multiple sites¶

Some vendors will re-use IP address, DNS names, host names, and/or accounts at different locations. While the site_id field can help distinguish on dashboards, the asset & identity framework requires customization when this occurs. This specifically requires an additional field to be added to asset and/or identity lookup files used by Enterprise Security called cim_entity_zone. In many cases this can be set to be the same as the site_id but can be customized as needed. For documentation on enabling this feature see the Enabling entity zones for assets and identities in Splunk Enterprise Security documentation.

NERC CIP dashboards and reports are not populating automatically.​¶

This problem most often occurs when one of two errors occur:

a. Verify that the asset lookup has been created and the following fields exist and are populated:

b. Data is not present in the data models or disabled. The following data models should contain data:

• Authentication
• Intrusion Detection
• Inventory
• Malware
• Network Sessions
• Network Traffic
• Updates

More information on specific dashboards and requirements can be found in the The Administration Guide