OT Security Add-on for Splunk: Technical Guide and Documentation

Splunk for OT Security enables organizations that operate assets, networks, and facilities across both traditional IT and industrial (OT) environments to better apply the globally proven SIEM, Splunk Enterprise Security, to improve threat detection, incident investigation, and response. The Splunk for OT Security add-on expands the capabilities of Splunk’s platform to monitor for threats and attacks, compliance, incident investigation, forensics, and incident response across the broad spectrum of assets and topologies that define modern manufacturing, energy, and public sector organizations.

The solution, comprised of an app and related documentation, provides the following features:

1. Expanded Asset Framework and Asset Center: Ability to store and analyze additional asset attributes including facility/site id, asset criticality, asset types, classification, vlan, zone, and other data alongside traditional IT asset elements. Assets can be segmented by site or into multiple entity zones when attributes like IP Addresses and Host Names may be reused among different sites.
2. Integration with leading OT Security partner technologies: Ingest asset inventory, vulnerabilities, and alerts from leading OT-ready systems.
3. Making using OT Data easier: Prebuilt dashboards, reports, and other content related to perimeter monitoring, infrastructure monitoring, and centralized monitoring of multiple OT Security solutions. This content is in direct response to customers wanting quicker time to value from their OT data.
4. Prioritized vulnerability matching: Evaluate, filter, and score matching vulnerabilities using iteratively executing correlation queries and dynamically calculated Asset Risk scores.
5. Integrated OT Asset Behavior Profiling: Monitor asset behavior profiles to detect activity changes on critical assets that may represent increased threat risk.
6. Constructing and evaluating asset baselines: Create baseline groups and baselines to verify assets follow a consistent hardened setup. Baselines can be created from data and extended to customer baseline types.
7. OT ready Correlation Searches: Extend the deep bench of existing Enterprise Security correlation searches that monitor identity, endpoint, network and access in Splunk with OT-specific searches including mapping to common security frameworks including the MITRE ATT&CK for ICS.
8. Support for key elements of NERC CIP: Dashboards and associated reports reviewed by trusted practitioners and NERC CIP auditors to help clients focus on NERC CIP requirements where Splunk can be assistive in compliance monitoring and audit support.

Installation Guide

Splunk for OT Security App (DA-ESS-OTSecurity) Installation

The Splunk for OT Security Solution is packaged as a Splunk App and is available on Splunkbase to Splunk Enterprise Security customers. If you are not an Enterprise Security customer and would like to trial the OT Security Solution, please contact your Splunk sales representative or send an email to otsecurity@splunk.com

Download and install the most recent release of the Splunk OT Security Add-on from Splunkbase HERE

Single Instance Splunk Deployments

Install the app on the instance of Splunk Enterprise with Splunk Enterpise Security already installed and configured. Correlation rules are disabled by default and will need to be enabled based on your particular use cases.

Distributed Splunk Deployments

Install the app on the search head only. The app is safe to install in large size clusters and will not impact indexers as search and correlation rules are disabled by default. As correlation rules are enabled, this may impact indexer performance, especially if multiple correlation rules are enabled all at once. It is recommended that rules be enabled as needed and then incrementally to minimize any negative effects on indexer performance.

The app also contains templates for lookup tables. The lookup files related to assets and identities are essential for dashboards and reports to populate correctly.

Search Head Clusters

Splunk for OT Security can be installed in an SHC by following the standard installation instructions for the app.

ES Specific Considerations

The Splunk OT Security Solution is a companion app to Splunk Enterprise Security and must be installed alongside Enterprise Security in both ES Search Head and ES Search head clusters.

Note on Potential Performance and Other Impacts:

If you save and enable searches included with the app in your environment, you could see changes in the performance of your Splunk deployment.

As is true for all searches in Splunk, the amount of data that you search affects the search performance you see in your deployment. For example, if you search Windows logs for two HMIs or Process Historian Servers, even the most intensive searches in this app add no discernible load to your indexers. If you instead search domain controller logs with hundreds of thousands of users included, you will see an additional load.

The searches included with the app are scheduled to run regularly and leverage acceleration and efficient search techniques wherever possible. In addition, the searches have been vetted by performance experts at Splunk to ensure they are as performant as possible. If you are concerned about resource constraints, schedule any searches you save to run during off-peak times.

You can also configure these searches to run against cached or summary index data. If you have a large-scale deployment, use the lookup cache for "first time seen" searches and select the "High Scale / High Cardinality" option for time series analysis searches.

Splunk Required Apps and Add-ons

Minimum requirements

The OT Add-on for Splunk works on by both On Premise and Splunk Cloud instances.

Mininum requirements to use the OT Add-on for Splunk are:

• Splunk Enterprise 8.0.x or greater
• Splunk Enterprise Security 7.0.x or greater
• Splunk Common Information Model 5.0 or greater

In addition, Splunk Enterprise Security should already be installed and configured before installing the OT Security Add-on. This guide assumes that Splunk Enterprise Security has been installed and pre-configured.

Recommendations

Recommended requirements to use the OT Add-on for Splunk are:

• Splunk Enterprise 9.1.x or greater
• Splunk Enterprise Security 7.1.x or greater
• Splunk Common Information Model 5.1.x or greater

App/Add-on Requirements

Several apps are leveraged by the Splunk OT Security App to provide additional insights via visualizations. The following visualizations should be installed on the host where the OT Security Add-on is being used:

• Network Diagram Viz

• Maps+ for Splunk

• Parallel Coordinates

• Flow Map Viz

OT Security Technology Apps and Add-ons

OT Security tools provide information on what devices and protocols are being used and, in some, can detect changes to these devices or provide threat intelligence. In addition, these tools often provide visibility into specific OT protocols and devices such as PLC's and RTU's. They often utilize appliances that are placed at critical segments of the network and monitor traffic across these segments. In most cases, monitoring is done passively, although several of them now offer active monitoring of assets by speaking native OT protocols to the devices.

The intelligence and information provided by these solutions can be critical in identifying OT assets and provide valuable context for assets. Splunk allows a customer to monitor the entirety of the OT environment, including critical IT infrastructure and networks, and extend visibility to IT and OT environments and OT devices.

When possible, using these apps or add-on provide mapping to the OT Security Add-on out of the box, although it always best to consult the respective product's documentation for details on data integration and data sources provided to Splunk.

Splunk for OT Security App (DA-ESS-OTSecurity) Configuration

Once the Splunk OT Security Solution app has been installed in your Splunk environment alongside your Splunk Enterprise Security app, you will need to take the following steps to configure the application for production use:

Core Integration Steps

Step 1: Update Navigation Menus​

Splunk for OT Security comes with navigation menus that can be edited to suit your Enterprise Security deployment. These navigation menus include links to dashboards and that are included in the Splunk for OT Security solution.

To Update the Navigation Menu follow these steps:

1. Open the Enterprise Security app in your Splunk instance
2. Go to the Enterprise Security app in Splunk
3. In the app navigation bar, navigate to the following location:
   • Configure→General→Navigation
   • 
4. On the Edit Navigation​ screen, add existing menus by selecting:
   • Add a New Collection→Add Existing→App:DA-ESS-OTSecurity→Select a Collection
   • 
5. The menu containing all the Operation Technology dashboards and reports will now appear. These can be dragged to the desired location in the menu hierarchy or can be modified to fit your organization's needs. For example, the Compliance menu containing NERC CIP dashboards may be removed if your organization is not under NERC CIP regulations.

Step 2: Configure the Asset Framework​

The Splunk for OT Security Solution extends the ES Asset Framework to provide additional context and information about OT assets. Ideally all fields (including those from the core ES asset framework) are populated, but only the mandatory ones are required. Fields that are mandatory from the core framework include at least one of the following: dns, ip, mac, or nt_host.

To update the asset framework follow these steps:

1. Go the Enterprise Security app in Splunk
2. In the app navigation bar go to the following location:
   • Configure→Data Enrichment→Asset and Identity Management
   • 
3. Go the Asset Settings​ ​ table
   • Update the Asset Framework by Adding New Fields (field names are case sensitive) as shown here:

Field Name	Tag	Multivalue	Mandatory	Key	Example
asset_id	Yes	No	No	No	189674, FUC456
asset_model	Yes	No	No	No	S7, 1762-OF4
asset_status	Yes	No	No	No	Hot, Ready, Standby
asset_system	Yes	No	Yes	No	Western Operations, Sandusky Plants
asset_type	Yes	No	Yes	No	Jump Server, PLC, Historian
asset_vendor	Yes	No	No	No	Siemens, Allen Bradley
asset_version	Yes	No	No	No	4.23, 7.1.4
classification	Yes	Yes	Yes (for NERC CIP)	No	cip:PCS, cip:ESP
description	No	No	No	No	Vibration for Sandusky Line 3
exposure	Yes	No	No	No	private, public, internal
location	No	Yes	No	No	Sandusky Prod Line 3/Bank 3/Press 2
site_id	Yes	No	Yes	No	Sanford Plant, Copperfield Power Plant
vlan	Yes	Yes	No	No	Sandford SCADA, 172.2.3.x
zone	Yes	Yes	Yes (for NERC CIP)	No	purdue:level3

• Enable ​asset and identity correlation​ on the Correlation Setup tab and set up to either ​Enable for all sourcetypes​ or ​Enable selectively by sourcetype and supply the required sourcetypes. In most cases, ​Enable selectively by sourcetype​ is preferred as it results in less load on the Splunk infrastructure since it only searches for specific data sources and not across all data.

  • 

Step 3: Upload Asset Information​

Now that you have updated the Asset Framework fields, asset and identity lookup files can be uploaded into ES. Note: previous versions of the add-on require specific lookup file names to be used, but this is no longer a requirement.

For NERC CIP use cases ONLY the following fields should have values as indicated here:

Field	Format	Example
classification	cip:<low,medium, or high>|cip<BCA,PCA,TS A,EACM,EAP>	cip:high|cip:EAP |cip:EACM
category	nerc	nerc
zone	eap:<zone name>	eap:PPLT

Important:​ In order to leverage lookup files from apps outside of Enterprise Security, Lookup Definitions​ must be created within the Splunk Enterprise Security Suite app context. For more information on managing lookups and knowledge objects within Splunk Enterprise, please refer to the documentation linked at the beginning of this document.

To create the lookup files and link them to the asset & identity framework follow these steps:

1. Go to Settings→Lookups​

2. Click on Lookup Definitions​

3. Click on New Lookup Definition. This lookup defintion should be linked to the lookup file containing information regarding your assets or identities. In addition, this lookup defintion needs to be shared globally to be acessible in Enterprise Security. This step will need to be repeated if you have multiple asset lookup files and independently for assets and identities.

4. Open the Enterprise Security App

5. In the app navigation bar go to the following location: Configure→Data Enrichment→Asset and Identity Management

   • 

6. Go to the Asset Lookup Configuration ​Tab

7. Click on +​ New​ button and configure your new asset lookup to match the name of your Lookup Definition for Assets

8. Repeat steps 6 and 7 until you have created new asset configurations for each of your asset lookup definitions

9. Go to the Identity Lookup Configuration​ Tab

10. Click on the + New button and configure you new lookup to the match of your Lookup Definition for Identities in step 3 above

11. Repeat steps 9 and 10 until you have created new identity configurations for each of your identity lookup definitions.

Macros

Macros are leveraged in the Splunk OT Security solution for re-use of searches and so pre-configured indexes, sources, and sourcetypes can be automatically adjusted to represent a specific customer environment.

Note:​ Macros are designed for efficiency and should only include data sources relevant to the query being performed. Using default or otherwise overly-broad macro definitions may result in slow and process-intensive searches.

Many of the macros below can be updated via the General Configuration menu within Enterprise Security. This can be access within Enterprise Security by going to Configuration→General→General Settings

To update macros manually for the Splunk for OT Security app perform the following steps:

1. Go to Settings → Advanced Search → Search macros*

2. Update the following macros to reflect the indexes, sources, and sourcetypes present in your environment. If a data source is not present in your environment it can be modified to a non-existing index and sourcetype to reduce query time.

Macro Name	Purpose	Requires Manual Update
exclude_internal_ips​	Should contain a subnets which are considered​ internal to the company	Yes
get_2fa_indexes	Should point to data sources relevant to multi-factor​ authentication (e.g. OKTA, RSA, etc. logs.)	No
get_asset_type_icon_and_color	This is used to consistently displaying asset types with specific icons and color	Yes
get_app_datamodel​	Should point to the data source that contains information on updates and applications being installed (e.g. windows update events)	No
get_asset_name ​	Will determine what field to display by default on dashboards for an asset's name	Yes
exclude_internal_ips​	Allows a user to designate the default name that should be used in tables and visualizations By default it is set to DNS, host, ip, mac in terms of displaying asset names	Yes
get_asset_type ​	Should contain a subnets which are considered​ internal to the company	No
get_asset_type_datamodel	Should point to the data source that contains information on updates and applications being installed (e.g. windows update events)	No
get_backup_indexes ​	Should point to the data source that contains client​ backup logs.	No
get_cve_index ​	Should point to the index where CVE and/or CPE definitions are indexed	No
get_installedapps_datasources	Should point to the data source which​ includes all the installed applications about hosts	No
get_os_datasources	Should point to the data source which contains OS​ information about hosts	No
get_ot_vendors	Should contain a list of vendors that are present in the environment that are considered OT vendors. This macro comes with a list of common OT vendors such as Siemens, ABB, etc.	No
get_ot_security_events	Should contain a filter to identify raw events from OT Security technologies	No
get_ot_security_notables	Should contain a filter to identify notable events from ES from OT Security technologies	No
get_ot_device_asset_types	Should contain a list of asset types which are​ considered OT devices and not devices in the OT environment (e.g. PLC's). This macro is pre-populated but should be adjusted to the customer's environment	Yes
get_ot_networking_devices	Should contain a list of asset_types or other attributes that identify a networking device as part of the OT environment	Yes
get_ot_security_alerts​	Should contain the index and/or sources types​ associated with OT Security solution. This macro is pre-populated with some common source types but should be adjusted based on the customer's OT security solution	No
get_perimeter_devices	Should contain a list of asset_types or other attributes that identify a device as part of the OT Security perimeter	No
get_perimeter_and_networking_devices	Should contain a list of asset_types or other attributes that identify a device as part of the OT environment	No
get_physicalaccess_records	Should contain the index and/or sourcetype​ where visitor access logs are stored	No
get_usb_datasources	Should point to the data source that logs external​ media devices being connected to a host (e.g., endpoint monitoring, windows registry, etc.)	No
get_visitoraccess_records	Should contain the index and/or sourcetype​ where visitor access logs are stored	No
label_internal_ips	Used to identify which is considered internal and external to a organizartion	Yes
ot_marker_icons	Gets icons to display for nodes within visualizations for a consistent display across all visualizations	No
ot_identifier	Should contain the filter to positively identify an asset that belongs to the OT environment. By default, it is set to look at the category field for the label "ot"	Yes
prohibited_ot_network_traffic_allowed_filter	Used to identify data that should be prohibited in the ot environment and leverages several lookups to designate them as prohibited or allowed	Yes

Important Lookup Files

KV Store Lookups

The Splunk OT Security solution contains two critical kvstore lookups that are leveraged to build baselines for asset configurations. The two tables are linked and timestamped. The critical kv stores and their fields is defined below:

cip_baseline groups

Field	Description
_key	Auto-generated by Splunk, ​this key must be linked to group_id​ in the system_baselines lookup (similar to a foreign key)
created_date	Time in epoch when this particular group was created. Note: new or modified groups should create a new entry so group configurations can be maintained over time
group_members	A list of host names that belong to this group, pipe delimited
group_name	Name for the group for readability

Note: The cip_ component of this kvstore lookup name can be ignored and is preserved to maintain backwards compatability

system_baselines

Field	Description
_key	Auto-generated by Splunk, unique identifier for this baseline
config	The actual configuration for this kind of baseline. This is normally json, but if this is not available it is possible to modify search and dashboards to use a different format
config_asset_type	Type of asset that this configuration should be applied to - the NERC CIP app currently only uses two values: computer and network (device such as plc). It is possible to store other configuration asset types as needed but the current NERC CIP dashboards only leverage these two
config_type	The type of configuration, for example. NERC CIP reports only use the following types: os, applications, patch, port_config. It is possible to store other configuration types (for example, services) as needed
created_date	Time in epoch when this particular configuration was created. Note: new configurations should create a new entry so configurations can be maintained over time
group_id	This maps this configuration to a specific group in the cip_baseline_groups, telling Splunk that the two are connected. This is an essential field for Splunk to know which​ configuration to apply to a machine and what parameters to use for checking for deviations

CSV Lookups

Common Lookup Files

Various lookup tables are essential for populating dashboards with data or for presenting visualizations. The following sections break down each lookup and its intended purpose.

Lookup name: asset_type_mappings

Lookup file that is used to standard the names of asset types as well as provide consistent icons and colors in visualizations.

Lookup name:​ critical_ot_services.csv

This lookup is used to identify critical services which require notification or may result in loss of operations. One of the MITRE ICS rules requires identifying when critical services have been stopped. The name of the service should match the service name by the operating system. Additional host names or wildcards can be used in the host_names column.

Lookup name:​ industrial_ports.csv

This lookup contains a list of ports to designate port activities with specific port ranges and industrial applications (e.g. identifying ports associated with a particular SCADA system).

Lookup name:​ interesting_ot_ports.csv

This lookup is used to label port activity that should and should not be permitted across security boundaries. For example, ports 80 and 443 are often prohibited between OT environments and public networks is typically prohibited but may be permitted between specific IP's on a company's corporate network. CIDR ranges can be used to designate network segments for src and dest networks.

Lookup name: ot_firewall_object_group*

Lookup file designed to hold a listing for object-groups which may be used by firewalls and their definitions.

Lookup name: ot_firewall_rules

Lookup file designed to hold a listing of all firewall rules. Note: could be a superset of the cip_firewall_rules lookup.

Lookup name: prohibited_traffic.csv

Lookup file that is used to designate prohibited and allowed traffic, along with the note describing the app name and why the activity is being allowed or prohibited. This lookup is used in the add-on, but is provided by the SA-NetworkProtection app.

Lookup name:​ ot_firewall_object_groups.csv

This lookup is used to expand information about object groups contained in firewalls so users do not need to look up object groups. Object groups will often be contained in the ot_firewall_groups.csv.

Lookup name:​ ot_firewall_rules.csv

This lookup is used to hold firewall rules and annotations for OT firewalls.

NERC CIP Lookup Files

Lookup name:​ cip_ip_ranges.csv

This lookup is used to define subnets that are considered part of NERC CIP OT environments. Subnets can be single IP's or use CIDR notation.

Lookup name:​ cip_network_configs.csv

This lookup is used to contain information on network devices as well as port and state information. This lookup will normally be populated from network configs, either regularly or statically.

Lookup name:​ cip_patch_approvals.csv

This lookup is used to contain information about patches and whether they are approved. This data will often be populated from the patching management system (e.g. WSUS). It can also be used to generate baselines.

Lookup name: cip_pra_completion_records​

Lookup file with a list of users and when their last personal risk assessment was completed. This lookup is used to verify individuals had a risk assessment completed at least every 15 months.

Lookup name: cip_site_classification​

This lookup is used to classify physical security sites and locations and their respective CIP BES classification. While often sites may be classified as a single BES level this lookup provides flexibility to use alternative mechanisms for classification. Note: classifications should follow the naming convention of other lookups to include <regulation>:<classification>.

Lookup name: cip_training_materials​

This lookup contains a list of all the training courses and materials that are available, including the title, description, the last time course updates were distributed, and whether the training is required or optional. It also included which groups are available to take the training.

Lookup name: cip_training_records​

This lookup functions as a list of courses that have been taken by individuals including when the training was completed and when it needs to be repeated. The course title should be contains in the cip_training_materials lookup.

Lookup name: cip_distribution_lists​

This lookup contains a list of distribution groups and the members of each group for cip training. Distribution_list_names are used in the cip_training_materials to identify individuals who would need specific training for NERC CIP compliance. Members of each list are pipe-delimeted.

Integration with Security Frameworks and Security Alerts

MITRE ATT&CK for ICS

The MITRE ICS ATT&CK model was released in January 2020 and provides a framework of common technique, tactics, and procedures (TTP) for attacks on Industrial Control Systems (ICS) and OT environments. While some of these TTP's can be identified using Splunk alone, several of them require third-party support of industrial and network protocols. As a result, the OT Security Add-on for Splunk has two main kinds of alerts, those in which a TTP is identified by a third-party OT Security product or those which can be detected with Splunk alone.

Sec - <TTP> (e.g. OT Sec - Data Historian Compromise​​). Items in yellow are covered by the OT ​Sec - MITRE ICS Alert ​correlation search. This correlation search requires a TTP identifier (specifically a field named technique_id) present in the events sent by the third-party product. This field is already supported by several third-party partner integrations.

By default Splunk will attempt to reach out for the latest version of the MITRE ATT&CK ICS framework; however, due to limitations with many OT environments a static lookup is provided an leveraged when tagging alerts. Any alerts from the MITRE ATT&CK for ICS will begin with a prefix of T0.

Event types

Event types provide a mechanism to classify logs and events and tag them with categories that can be searched and aggregated across multiple indexes, sources, and source types. The OT Security Add-on for Splunk includes event typing specific to MITRE ICS alerts and data from third-party OT Security solutions.

The following event types are used within the solution:

mitre_ics_alert:** this event type is used for all MITRE ICS-related alerts and requires the macro​ get_ot_security_alerts to define which data sources should be included. This event type is used in the correlation search OT Threat - MITRE - ICS Alert ​ to generate notable alerts. In addition, the tag mitre_ics can be used to identify these same events.

Third-party OT Security Product Alerts

Third-party OT Security products often generate alerts and events relevant to Splunk Enterprise Security. These alerts are particularly valuable when they leverage the Alerts data model included in Splunk's Common Information (CIM). To include these events as notables within Splunk, enable the correlation search OT Threat - Non-MITRE ICS OT Security Alert and OT Threat - MITRE ICS Alert to show both non-tagged security alerts and also those tagged with a specific MITRE indicator​. We recommend that this correlation search is run and results are validated and searches tuned prior to enabling the rule globally. This will prevent unnecessary alerts showing up in the Enterprise Security Incident Review dashboard.

Upgrade Guide for the OT Security Add-on for Splunk

Upgrading from Version 2.1 to 2.2

Version 2.1 to 2.2 mostly contains additional dashboards and content that can be leveraged and upgrading via the web UI is recommended.

Additional changes/guidance after upgrade include:

1. After upgrading, the old navigation menu will need to be removed and configuration step 1 below will need to be performed for Navigation Menu Updates to be reflected.
2. Any customizations to existing dashboards will not be overwritten during the upgrade phase. Depending on your requirements you may wish to review the default versions of these dashboards as small changes have been made. It is recommended that you back up any local version of the dashboard before reverting to the default dashboard.

Videos / How-to Guides

Ended: Videos / How-to Guides

Ended: Installation Guide

Administration Guide

Included Detections and Key Security Indicators (KSI)

The following sections contains a list of included Correlation Searches and Detections included with the OT Add-on for Splunk. These searches can be found under Content Management within Enterprise Security and are linked appropriately in the Use Case Library for OT Security.

Detections - Notables

• Access Granted for Uncertified Official (NERC CIP related)
• Exploitation of Remote Services
• External Report Services
• Graphical User Interface Usage
• Remote File Copy
• Suspicious Local Log On after Denied Physical Access
• Theft of Operational Information
• Command-Line Interface
• Data Destruction
• Data Historian Compromise
• Engineering Workstation Compromise
• Replication Through Removable Media
• Scripting
• Service Stop
• Upapproved Removable Media Use on OT Asset
• User Execution
• Default Credentials Detected
• Detected Use of Default Accounts from External System
• Detected Use of Priviledge Accounts from External System
• Valid Account Usage
• Commonly Used Port
• Connection Proxy Detected
• Denial of Service
• Exploit Public Facing Application
• Internet Accessible Device in OT
• Network Connection Enumeration
• Network Service Scanning
• Network Sniffing
• Remote System Discovery
• Standard Application Layer Protocol Usage
• Data from Information Repositories
• Drive by Compromise
• Masquerading
• Project File Infection
• Spearphishing Attached Related Activity

Detections - Risk Based Alerts (RBA)

• Risk Threshold Exceeded for OT Asset Over 24 Hour Period
• Risk Threshold Exceeded for OT Facility Over 24 Hour Period
• Risk Threshold Exceeded for OT User Over 24 Hour Period

Detections - Importing of Notables from Other Platforms

• MITRE ICS Alert - leveraged when integrations provide technique_id fields for MITRE ATT&CK for ICS
• Non-MITRE ICS OT Security Alert - leveraged when integrations do no provide MITRE ATT&CK for ICS related information

Key Security Indicators (KSI)

• Vulnerabilities - Detected CVE's
• Active IT/OT Devices
• Active Notables for OT
• Active OT Devices
• Asset Count with Notables
• Critical Assets with Known Vulnerability
• OT Firewall Changes
• Aggregated Risk in OT
• Average Risk Score in OT
• Total Assets in OT
• Total OT Devices
• Total Unknown Assets
• Total CVE Defintions Imported
• Total Assets Missing Endpoint Protection
• Total Assets Not Receiving Malware Signature Updates
• Total Assets with Endpoint Protection
• Total Assets without Malware Signature Updats within the last 7 Days
• AD Group Changes
• Number of Inbound External Connections
• Number of Outbound External Connections
• Count of Prohibited Traffic Allowed
• Count of Prohibited Traffic Blocked
• Number of RDP Sessions
• Number of Remote Access Sessions
• Number of Screen Share Sessions
• Total Number of IDS Alerts
• Total Number of VPN Sessions
• Total Number of Assets Using an Industrial Protocols
• Total Number of Industrial Protocols Detected
• Total Number of Perimeter Devices Sending Data to Splunk
• Number of Account Changes
• Number of Account Lockouts

Reports

The following section contains a list of included reports with the OT Security Add-on for Splunk. These reports can be scheduled, customized, or adjusted as needed.

• Authentication Attempts Outside Normal Working Hours - All attempts
• External Login Activity from External Systems (e.g. VPN, RDP, etc)
• Login Activity to OT Assets Outside of Normal Working Hours - Successes Only
• Perimeter Device Changes Over Last 7 Days
• Prohibited Traffic Allowed Over Last 24 hours
• Prohibited Traffic Blocked Over Last 24 hours

Overview

Splunk for OT Security is designed to work to help Splunk Enterprise Security customers understand more about their OT environments and create end-to-end security visibility across both OT and IT systems. In addition, customers under NERC CIP regulations will be able to leverage Splunk's platform to ensure compliance and auditing requirements. Splunk for OT Security focuses around extending the current capabilities of Splunk Enterprise Security in two primary areas explained below.

Providing Asset Context

While Splunk Enterprise Security provides existing dashboards and capabilities around understanding Assets, OT systems often require additional context and investigation before action is possible. Splunk for OT Security extends Splunk Enterprise Security's Asset & Identity Framework to include additional important fields such as site, role of machines, and location within ISA 99 models.

Dashboards that cover these areas can be found via the Operational Technology navigation menu item as shown here:

Extended Dashboards and Investigative Capabilities

While in most cases Splunk for OT Security will be integrated in a combined Security Operations Center for both IT and OT, specific views into the OT environment can help an analyst understand current security posture. This includes specific views around security posture, OT assets, as well as vulnerabilities in OT Environments. In addition, OT assets are now integrated directly with Splunk Enterprise Security existing dashboards, reports, and incident management capabilities.

For example, OT assets can be tagged in the Asset and Identity Framework directly to produce a list of incidents related to OT Assets and Identities. This is shown here by simply putting in the word *ot* into the search criteria to produce a list of OT incidents which can be reviewed.

Incident Review Example

Dashboard Content

As a result of extending the Asset and Identity Framework, any field such as site, environment, or classification can now be used as a filter by specifying a tag in the filter criteria.

OT Security Posture

The OT Security Posture dashboard is designed to provide a high level overview of an organization\'s security posture for their OT environment. New Key Performance Indicators (KPI's) have been created that focus around the health and risk of OT security operations. Notable security events are also pre-filtered and include both existing correlation rules as well as new MITRE ICS ATT&CK correlation rules. Drilling down on a notable allows the security analysis to start an investigation within ES easily.

OT Security Posture Example (OT Security Posture):

In addition, OT Asset Activity is summarized below so that the security analyst can understand how asset's network behavior might be changing or how their risk is changing over time. Additional filters at the top of the dashboard allow panels below to view details for specific sites, systems, or business units.

OT Asset Activity Example (OT Security Posture):

Network dashboard panels provide both high level and detailed views including the ability to show basic information or more detailed statistical information about the asset. Furthermore, drilling down on an asset allows the security analyst to quickly understand more about the asset in the OT Asset Investigator dashboard.

Assets by Network Activity Detailed Example (OT Security Posture):

OT Security Technology Integrations

The OT Security Dashboard is intended to bring together diverse OT Security tools into a single dashboard and provide a way to search across these different technologies. In the dashboard a user can search and filter across different sourcetypes associated with these products. This dashboard leverages the macros get_ot_security_events and get_ot_security_notables to specify which products should be presented in this dashboard.

Dependencies

• Asset & identity framework
• Indexes
  • notable
• Macros
  • get_asset_name
  • get_asset_type
  • get_ot_security_events
  • get_ot_security_notables
• Lookups
  • asset_type_standard_lookup

OT Asset

OT Asset Investigator

The OT Asset Investigator dashboard is designed to provide OT security practitioners with additional context to and understanding of OT asset behavior over time. In addition, the dashboard provides an investigative workflow by presenting meaningful metrics and information that a security practitioner might be able to use for ad-hoc analysis, either for a specific asset or data source.

The dashboard upper section presents information about the asset itself, such as location, operational role, and critical information to determine the priority of the asset relating to a security incident.

Dependencies

• Asset & identity framework
• Indexes
  • notable
  • risk
• Macros
  • get_asset_name
  • get_asset_type
• Lookups
  • asset_type_standard_lookup

OT Asset Details Example (OT Asset Investigator):

The top selection provides information that is known about an asset, such as its characteristics, classifications, and physical location (when latitude and longitude are provided). In addition, the dashboard contains several high-level indicators including known vulnerabilities related to this asset, any notable events, as well as any traffic that has been detected as prohibited.

The OT Asset Behavior Indicators provide base metrics on the networking behavior of the selected asset and visualizes the types of communication with others hosts in the environment. This can prove important when trying to identify for example communication that should not be permitted between subnets. Subnets listed on the left can be clicked on to filter the network communication graph on the right-hand side.

OT Asset Behavioral Indicators Example (OT Asset Investigator):

The last sections of the dashboards provide information around data sources which contain the selected asset and can help guide analysis to different data sources within Splunk to investigate. By selecting a data source in the bottom panel, analysts can quickly drill down into a Splunk search showing that data source and the asset selected for ad hoc analysis.

Sourcetype Timeline Drilldown Example (OT Asset Investigator):

OT Asset Center

The OT Asset Center is designed to provide visibility into OT systems and provide meaningful metrics related to vendors, models, and asset types. Unlike the OT Asset Investigator, this dashboard focuses on the entire environment and not specific assets and allows users to search for specific software or hardware in the OT Environment.

Additional filters at the top of the dashboard allow panels below to view details for specific sites, systems, or business units.

OT Asset Indicators Example (OT Asset Center):

The OT Asset Indicators panels provide metrics on assets that are present in the environment, both in terms of asset types, models, vendors, as well as operating systems and software vendors. Each of these charts can be clicked to populate and filter other parts of the dashboard.

Dependencies

• Asset & identity framework
• Indexes
  • Notable
  • Risk
  • Non internal indexes (for sourcetype drilldown)
• Data Models
  • Vulnerabilities
  • Network_Traffic
• Macros
  • ot_identifier
  • get_perimeter_devices
  • get_asset_type_icon_and_color
  • make_asset_markers
  • get_asset_name
  • get_asset_type

OT Asset Information Example (OT Asset Center):

This part of the dashboard is intended to provide a security analyst a method to analyze software and hardware present in the OT environment. Various filters provide the capabilities to search for specific hardware or software components (e.g. checking for the installation of java for a particular asset type). Note, viewing hardware and software components can be toggled under the Asset View Heading. Both hardware and software assets provide additional filters specific to those data types. Also when clicking upon a vulnerability an additional panel will appear giving the specifics of that CVE indicator if present.

OT Vulnerability Center

Dependencies

• Asset & identity framework
• Indexes
  • Customer specified index containing field cpe_uri
  • Index containing installed applications on hosts
  • Index containing OS information
• Data Models
  • Vulnerabilities
• Macros
  • ot_identifier
  • get_asset_name
  • get_asset_type
  • get_installedapps_datasources
  • get_windows_os_info

OT Computer – Baseline Monitoring

This dashboard takes data from configured baselines and identifies deviations from the expected baseline. By default it is set to show OS, Services, and Application baselines (those provided by the add-on out of the box); however, if you desire to add customer baselines for computers additional panels will need to be added by the user to show those on this dashboard.

For more information setting up baselines go to section

Dependencies - Asset & identity framework - Indexes - Those related to OS information, service state, and applications installed - Data Models - Inventory - Lookups - system_baselines - cip_baseline_groups - Macros - ot_identifier - get_asset_name - get_asset_by - get_service_states - get_installedapps_datasources

OT Networking – Baseline Monitoring

This dashboard takes data from configured baselines and identifies deviations from the expected baseline for networking devices. By default, it is set to show port config baselines (those provided by the add-on out of the box); however, if you desire to add customer baselines for computers additional panels will need to be added by the user to show those on this dashboard.

For more information setting up baselines go to section

Dependencies - Asset & identity framework - Indexes - Those related to port state configuration - Lookups - system_baselines - cip_baseline_groups - cip_network_configs - Macros - ot_identifier - get_asset_name

OT Perimeter

Perimeter Monitor

This dashboard is designed to provide an overview of the perimeter of an OT system as defined in the get_perimeter_devices macro. It focuses on traffic traversing in and out of the security perimeter and whether it is allowed, blocked, and/or perimeter based on a combination of the interesting_ports_lookup and prohibited_traffic_lookup_by_category lookup definitions. The interesting_ports_lookup allows you to proactively identify traffic and the associated application. For example, a user could include port tcp 502 as an interesting port and label it as the app modbus. Using the prohibited traffic lookup file they could then specify that traffic on tcp port 502 is prohibited between an OT zone and Corporate zone. Another common example would be internet traffic outbound from the OT zone to another zone. Each of these lookup tables support wildcards for fields as well. While in many cases this traffic should be blocked automatically via the perimeter this dashboard helps validate that these assumptions are correct. In addition, the dashboard provides changes in traffic such as new ports or significant changes in traffic volumes.

Dependencies

• Asset & identity framework
• Indexes
  • Notable
  • Risk
  • Non internal indexes (for sourcetype drilldown)
• Data Models
  • Network_Traffic
• Macros
  • ot_identifier
  • get_perimeter_devices
  • get_asset_name
  • get_asset_type
  • prohibited_ot_network_traffic_allowed_filter
  • label_internal_ips
• Lookups
  • interesting_ports_lookup
  • prohibited_traffic_lookup_by_category

Remote Access

This dashboard is focused on remote access into an OT environment, specifically VPN, RDP, Screen Sharing Sessions, or other remote access. It provides KSI\'s related to this activity as well as specifics on the remote access activity. It also provides a visual diagram showing the access path to assets by users visually to identify prohibited access (for instance if all remote access should come via a remote access or jump server).

Dependencies

• Asset & identity framework
• Data Models
  • Network_Sessions
  • Authentication
  • Endpoint.Processes
• Macros
  • ot_identifier
  • get_perimeter_devices
  • get_asset_name
  • get_asset_type

Device Audit

This dashboard focused on changes that have been made to devices identified as perimeter devices as defined in the get_perimeter_devices macro. It provides some KSI's on those device changes as well as an overview of the perimeter devices, ports and protocols being used by those devices, access to perimeter devices, and audit logs of both firewall rule changes as well as the rules themselves. It can be useful for identifying what changes are made on devices as well as who is logging into perimeter devices. Firewall rules and object definitions are contained in the ot_firewall_rules and ot_firewall_object_groups lookup files.

Dependencies

• Asset & identity framework
• Indexes
  • Notable
  • Risk
  • Non internal indexes (for sourcetype drilldown)
• Data Models
  • Authentication
  • Change
  • Network_Traffic
• Macros
  • ot_identifier
  • get_perimeter_devices
  • get_asset_name
  • get_asset_type
  • prohibited_ot_network_traffic_allowed_filter
  • label_internal_ips
• Lookups
  • industrial_ports
  • interesting_ports_lookup
  • ot_firewall_rules
  • ot_firewall_object_groups

Traffic Investigator

This dashboard focuses on visually showing traffic paths between devices. It can be used to quickly identify how traffic is moving through perimeter devices to individual devices (inbound or outbound) as well as via networking equipment such as routers and switches. Perimeter devices are identified by the get_perimeter_devices macro and additional equipment can be specified via the macro get_perimeter_and_networking_devices macro.

Dependencies

• Asset & identity framework
• Data Models
  • Network_Traffic
• Macros
  • ot_identifier
  • get_perimeter_devices
  • get_asset_name
  • get_asset_type
  • get_perimeter_and_networking_devices
• Lookups
  • industrial_ports
  • interesting_ports_lookup

OT Infrastructure

OT Account and Domain Monitoring

This dashboard focuses on monitoring account usage and changes as well as monitoring OT related domains. It provides information on invalid accounts, lockouts, password expiry, as well as information regarding group and other user account modifications.

Dependencies

• Asset & identity framework
• Indexes
  • Windows security logs
• Data Models
  • Changes
• Macros
  • ot_identifier
• Lookups
  • access_tracker

Endpoint Protection

This dashboard is focused on monitoring of endpoint protection products and ensuring that endpoints are adqueately protected. While many OT vendors do allow endpoint protection like antivirus, there are can be gaps in understanding how protected an entity might be. This dashboard is meant to provide information such as the kinds of endpoint protection activity occur, detections, as well as endpoints with no protection, outdated signatures, and last updates.

Dependencies

• Asset & identity framework
• Indexes
• Data Models
  • Malware
• Macros
  • get_asset_name
  • ot_identifier
• Lookups
  • malware_operations_tracker

OT External Media and Shares

This dashboard is designed to show the use of external media devices on OT Assets as well as File Shares. Both are known ways to bypass security controls or exploit hosts. It includes both successful and failed share access attempts and usage of external media devices over time.

Dependencies - Asset & identity framework - Macros - get_share_info_windows_events - get_removable_media_indexes

OT Host Access Monitoring

This dashboard is designed to show access by users to hosts in the OT environment. This includes both remote, local, and physical access to assets (across workstations, servers, databases, networking equipment, etc). This includes both successful as well as failed logins

Dependencies

• Asset & identity framework
• Indexes
• Data Models
  • Authentication
• Macros
  • get_asset_name
  • get_asset_type
  • ot_identifier
• Lookups

Configuration Baselining

Baseline Creation and Editing

There are two components in the OT Security Add-on that allow baselines to be created, the group editor and the baseline editor. To use the baseline editor, groups must first be set up. Several configurations for baselines come included with the OT Security Add-on:

• Computer – Operating Systems
• Computer – Applications
• Computer – Services
• Network – Port Configuration

Although these 4 are included it is possible to create different baselines by updating the baseline_type_definitions.csv file. In addition, the existing baselines configurations might need to be adjusted based on your specific environment. Also, adding additional baselines requires additional tables to be added to OT Computer – Baseline Monitoring and OT Network – Baseline Monitoring dashboards.

Baseline Type Definitions

This file contains a definition of baselines that are possible. Updating this table will automatically be refresh on various configuration panels within the Editor dashboards. The following show definitions for each column:

• config_asset_type: This column should contain a high level organizational category for the baseline grouping. By default, there are only two - networking and computer.
• config_type: This defines the actual baseline type that can be created.
• parse filters: This definition can be used to pre-filter results
• new_entries_search: This should define how to identify possible choices that are not already part of the config. For example, it should contain a list of potential software applications a user can select to be added to the config. It should always begin with a “| append [your search]”. Lastly it should contain an entry for a field named Action with a value of “Add”.
• fields_to_show: This is the name of the columns that should be shown when allowing users to select configuration items. It is not necessary to add the Add column from the new_enties_search.

OT Tools – Group Editor

This dashboard allows users to create groups of assets and then assign them baselines. This allows user to simplify the need to set up individual baselines for assets that leverage the same configurations. It leverages an interactive GUI for creating new groups, updating existing groups, delete existing groups. In addition, it provides an auditing mechanism for changes made to groups. The general flow for creating a new group is as follows:

1. Create a new group with the Create New Group button.
2. On the form that appears select the hosts that should be included in the group
3. A new group will appear with NO MEMBERS DEFINED and the group you created will show up in the Pending Group Changes table.
4. Once all changes are made, click the approve button on the row and provide an approval comment. The page will automatically load and show up in the existing baseline details table.

Dependencies - Asset & identity framework - Lookups - cip_baseline_groups (used for OT and CIP) - audit_group_baseline_definitions

OT Tools – Baseline Editor

This dashboard can be used to create new baselines for groups and assign items to each baseline. For example, a group name SCADA devices can be configured to allow a user to identify what applications should be installed on these hosts. This is then used to identify deviations on members of the group.

Dependencies - Asset & identity framework - Lookups - system_baselines - audit_baseline_definitions

Overview and Purpose of NERC CIP Content

The Splunk for OT Security CIP components are meant to help automate CIP investigations and reports that are mandated by the NERC CIP standard. This focus is reflected in how the Scorecards and Reports are organized and how they are populated. Whenever possible, they leverage Splunk's Common Information Model (CIM) and rely on Enterprise Security's Asset framework to make implementation easier and faster for organizations.

The Administrative Guide provides key information on tagging and identification of assets that fall under NERC CIP classification.

Scorecards and Reports

The NERC CIP components of Splunk for OT Security are broken into two main categories:

Scorecards and Reports. Navigation in the app differentiated between the two as shown here!

All Scorecards and Reports are grouped together under CIP Numbers (e.g. CIP 002, 004, etc) to help organizations quickly navigate between the various requirements; however, there are differences between the two that are important to understand when using them for reporting purposes.

Scorecards are a collection of CIP Requirements (R2, R4, etc) for each area. They are not prefiltered for reporting purposes to auditors, but allow organizations to understand their overall posture and state and maximize flexibility when needing to investigate potential issues.

Reports are more granular and designed as a mechansm for an organization to review and then hand over the results to an auditor. In addition, reports are pre-filtered​ based on the NERC CIP requirements and are designed to be used as​ input for an audit (although depending on an auditor's request may need additional filtering). Reports do allow organizations to do some filtering to answer specific queries from auditors, for example, requests for records regarding a specific asset or user. This guide does not cover reports in depth as many are derived from similar panels on the scorecards. It is recommended that any evidence from Splunk given to an auditor first be reviewed internally.

The following section breaks down the Scorecards and Reports by their major classifications:

NERC CIP 002

Critical Cyber Assets

NERC CIP requires BES Cyber Systems to be classified and broken down by specific classifications and security zones. The classifications require an asset to be assigned a CIP criticality as well as CIP asset type. CIP explicitly defines criticality as Low, Medium, or High and has numerous asset types (BCA, PCA, EAP are all examples). In addition, typically at a site level an asset will be assigned to a CIP asset zone, although Splunk does not explicitly make that mandatory. The Critical Cyber Asset Scorecard provides a customer to method to understand how assets are classified in their environment. As a result, assets should be tagged with these fields as explained in the Administrative Guide. If these assets are tagged appropriately they will show up CIP 002 Scorecards and Reports automatically.

Critical Cyber Assets Example (CIP 002 R1):

NERC CIP 004

Security Awareness Training

NERC CIP requires that users and operators in regulated environments complete specific training as part of the certification process. For users who require certification, they must be classified into groups which determine which training is required. In addition, updates to course materials should be communicated to individuals who have previously taken this training (normally via email). The email data model is used to identify whether notifications have been received.

Security Awareness Training Example (CIP 004 R1):

Cyber Security Training

Individuals accessing NERC CIP environments must be trained and certified before accessing assets within NERC CIP environments, remotely, onsite, or physically. NERC CIP requires that training requirements be tracked and monitored for expired certifications and then correlated with access records. This dashboard makes use of the authentication data model to track to determine remote or local access to systems by individuals required to be NERC CIP certified.

Cyber Security Training Example (CIP 004 R2):

Personnel Risk Assessment (PRA) Program

Individuals accessing NERC CIP environments must periodically have personnel risk assessments (PRA) performed not to exceed every 7 years. This certification may be performed by outside entities, but should be tracked and recorded by operators of NERC CIP assets. In addition, access to NERC CIP environments should be monitored for individuals out of compliance.

Cyber Security Training Example (CIP 004 R3):

NERC CIP 005

Electronic Security Perimeter and Remote Access

NERC CIP 005 is focused on protecting the perimeter of the NERC CIP environment by monitoring the Electronic Security Perimeter (ESP) and Remote Access into the environment. For assets which are part of the ESP (normally firewalls and data diodes), the devices must be explicitly tagged with the classification of cip:EAP in the asset framework.csv. For devices that are on the perimeter and are often responsible for monitoring ingress and egress traffic (e.g. IDS, IPS) they should be tagged with the classification of cip:EACM in the asset framework. Most ESP devices are also EACMS devices and should be tagged with both classifications (pipe delimited). Additional data sources from equipment such as networking equipment and multi-factor authentication systems help to determine if remote communications are properly secured. The macro get_2fa_indexes can be used to point to specific indexes, sources, and sourcetypes to capture multi-factor addition logs and improve search performance.

Electronic Security Perimeter Example (CIP 005 R1):

Interactive Remote Access Management Example (CIP 005 R2):

NERC CIP 007

Ports and Services

Ports and services being used and present on machines is mandated by NERC CIP and involves a wide plethora of data sources relevant to these requirements. Firewall, routers, switches can provide information on ports being used, as well as logs from machines themselves or data can be collected via Splunk Stream. In addition, events from endpoint protection logs that monitor USB usage, or directly from Windows Events (current OS) as well as windows registry (older OS's) play a role in determining when remote media is being used. Since the source of remote media usage can vary, the macro get_removable_media_indexes can be used to point to specific indexes, sources, and sourcetypes that contain this information.

Ports and Services Example (CIP 007 R1):

Security Events, Malware Alerts, and Monitoring

The monitoring of the NERC CIP environment is essential to understand threats to the environment. Monitoring of those alerts requires NERC environments to collect security logs and endpoint protection logs such as antivirus. In addition, NERC regulations require that logs be kept for at least 90 days and be periodically reviewed. The 5 dashboards in this section depend completely on Splunk's Common Information Models for malware, network sessions, and authentication. Since several of the requirements specify there should be a method to trigger and investigate incidents, Splunk's Enterprise Security notable and investigations features are essential to meeting several of these requirements.

Dashboards involving malware require endpoint protection to be installed, updated with signatures, and malware alerts be sent to Splunk. Those involving security logs are primarily focused on logon events, triggering security events for suspicious activity, and review of those logs.

Malicious Code Prevention Example (CIP 007 R3):

Security Event Investigations Example (CIP 004 R4.1):

Security Event Monitoring Example (CIP 007 R4.2):

Security Log Retention Example (CIP 007 R4.3):

Summary of Events Review Example (CIP 007 R4.4):

Patching

NERC CIP requires that NERC environments be updated and monitored for missing patches. In many cases, NERC environments already utilize patching solutions to help manage and approve patches (such as WSUS). In order to prevent duplication of effort, if it recommended that most of the approved patches come from existing solutions and be periodically updated. These scorecards rely on the patch baselines outlined in CIP 010 to determine when patches should be installed, as well as logs indicating patches back been installed either from the endpoint or the patching solution.

Security Patch Management Example (CIP 007 R2):

Identities

The behavior of identities and user accounts is a critical part of CIP 007, especially the use of default and privileged accounts. Since this section primarily deals with identities and heavily leverages content from the asset framework. As a result this lookup should be part of the asset framework as detailed in Section 3. Identities utilized the category field in the identities framework to designate when an account is default, generic, or privileged. This field can also be used to designate certain accounts as nerc, operations, or other classifications as needed. In addition, data regarding password changes should be included as essential for reporting purposes.

System Access Controls Example (CIP 007 R5):

NERC CIP 008

Cyber Security Incident Response Plan

NERC CIP operators are required to have defined cyber security incident response plans (IRP) that identify how to respond to cyber incidents or violations of NERC CIP regulations. Part of this regulation requires a method to show notable or cyber incidents. This dashboard provides an overview of all the notable alerts that have been generated for NERC CIP regulated assets and is dependent on existing correlation rules built into Enterprise Security. The status and incident owner of each notable is reported to ensure incidents have been assigned and/or resolved. IRP plans should be reviewed at least yearly and updated and this dashboard provides a method to report on changes to IRP plans.

System Access Controls Example (CIP 008 R1):

NERC CIP 009

Recovery Plan Specifications

NERC CIP Operators are required to ensure their BES Cyber Systems can be restored quickly in case of failure or cyber attack. This includes monitoring not only the BES Cyber System but also any CIP assets which require backup. This dashboard provides information about the Splunk environment, including index retention, clustering, and Splunk features tied to High Availability and Disaster Recovery. This dashboard also brings in data from backup logs to ensure CIP assets are being backed up. The macro get_backup_indexes is used to specify data sources that contain records around client backups.

Recovery Plan Specifications Example (CIP 009 R1):

NERC CIP 010

Baselining of Assets

Baselining of computers and network devices is required for CIP 010 compliance. Baselines can be the result of static configurations (e.g. a list of approved patches) but ideally are generated from data sources. Some good examples of data sources to consider when generating baselines are patching approval systems (such as WSUS), asset information (endpoint protection, Splunk forwarders, etc), installed software inventories, as well networking management systems. The baseline features implemented in Splunk are designed to keep track of baselines so that assets can be reviewed against specific baselines. Keeping these baselines is also required by regulation.

Assets can be grouped together so that assets within a group should match a particular configuration. Currently computer and network baselines are the only requirement kinds of baselines and each has specific elements which must be baselined. In the case of each Scorecards are designed to identify deviations from the baseline and provide information of how the item deviates from the baseline (for example, software that is installed but not approved) including hosts.

For more information of creating baselines please see Configuration Baselining Section. These dashboards are modified from the default baselining dashboards to review specifically only assets that are classified as part of NERC CIP.

Computer Baselines Example (CIP 010 R1 - Computers):

Network Baseline Example (CIP 010 R1 - Network Devices):

Transient Assets and Removable Media

Requirements exist to track transient assets such as laptops and maintenance equipment and removable media. Since transient assets are not continuously connected to networks and systems, their activity on the system should be closely tracked. The standard allows for certain removable media to be approved for general use. Assets and removable media should be classified as cip:TSA in the asset lookup for approved devices.

Transient Cyber Assets and Removable Media Example:

Videos / How-to Guides

Perimeter Montoring

Asset & Vulnerability Dashboards

NERC CIP Setup

Ended: Videos / How-to Guides

Ended: Administration Guide

Guide for data sources and getting data in

Important Data Sources

The OT environment is a combination of traditional and legacy IT technologies (e.g. firewalls, servers, workstations) combined with OT specific technologies (e.g. PLC's, RTU's). The following table outlines common data sources that should be integrated to provide full functionality with the existing OT Security Add-on, along with related data models.

The following data sources often produce value and are recommended to be collected

Data Source	Criticality to App	Data Models
Windows Security Events	Critical	Authentication, Change
LDAP (e.g. Active Directory)	Critical	Authentication, Change
Firewall Traffic & System Logs	Critical	Network Traffic, Network Session, Change, Authentication
OT Security Solutions	High	Authentication, Intrusion Detection, OT Asset, Vulnerability
Endpoint Protection	High	OT Asset, Malware
Network Traffic & System Logs	Medium	Network Traffic, Network Session, Change, Authentication
Patching Logs	Medium	Updates, OT Asset
Host Information (Application, Services, OS)	Low	Inventory, Change, OT Asset, OT Software

Other Important Information

There are a number of data sources which may provide contextual value around the OT environment. These data sources can help with macros and lookups which the OT Add-on leverages or to ensure the dashboards are reporting accurately. While not all of these data sources are required, having them can help enhance reporting.

Data Sources	Value
OT VLAN's and Subnets	In many organizations, the use of particular subnets is common practice for OT environments. This information can be used for some of the following purposes: Identification of sites, prohibited and allowed network traffic, identify locations of devices based on subnet.
Identification of Perimeter Assets and Characteristics	Within the OT Add-on several macros attempt to identify devices classified as perimeter or boundary devices. Identifying these assets either based on asset type, ip, or other characteristics is essential for the perimeter monitoring dashboards
List of prohibitted traffic types and flow direction	For the OT Prohibited Traffic Dashboards, the dashboards rely on building a simplified lookup that identifies all traffic that should be explicitly prohibited (for example, http or https) including direction (inbound vs outbound) and is crucial in identifying this suspicious traffic
List of internal IP ranges (IT and OT)	This list is used to identify not only the environment traffic sources and destinations, but also to identify traffic that might be outside a organization. I list of known IP's ranges can help in configuring macros when identifying OT, IT, or External traffic
Normal operating hours per site_id	Several reports attempt to identify activity normal working hours for sites. Having the normal operating hours including time offsets from GMT helps those reports to be accurate when reporting activity after normal working hours.
List of permitted External Media devices	To help identify external media devices that are allowed within the OT environment, having information on those devices perimeted included parameters like host restrictions can help in identifying allowed and prohibbited activity
National Vulnerability Database - CVEs	CVE defintions from the National Vulnerability Database (NVD) are used to correlate against detected vulnerabilities as well as identifying potential vulnerabilities. Various plugins exist to pull in this data into Splunk.

Getting Data In (OT Specific Considerations)

In OT environments and use of agents and similar mechanisms may not be approved by the system vendor. The following table outlines various mechanisms to pull in data from OT environments based on customer implementations. For more details

Method	Data Source	Notes	Access Type
Universal Forwarder	Host bases logs including Windows events, applications installed, service configuration, ICS logs, performance monitoring	More versatility and control is data types collected from hosts	Agent based
Existing agents	Depending on agent but could include malware, security events, and asset information	Examples include (but not limited to) Snare, Endpoint Protection, WhatsUp Gold, SCCM, and SCOM	Agent based
SFTP/FTP	Typically text based logs	SFTP (preferred) and FTP can be used to export logs periodically to systems using a Universal Forwarder to forward the logs	Agent based
Windows Event Forwarding	Windows Events	Can be used to collect Security, System, Application, or other specific windows events	Configuration
Syslog	Network and firewall logs, netflow, security alerts from other products	Best practice to leverage a syslog server rather than sending directly to Splunk	Configuration
Zeek	Networking information	Zeek can collect information on network activity such as network traffic and in some cases may support industrial protocols	Configuration
OT Security Solutions	asset information, alerts, vulnerabilities	Most OT Security Solutions have the ability to send asset, alerts, and vulnerability information to Splunk but may change as capabilities mature. In most cases they provide this information via syslog and REST API's	Remote
REST API's	Depending on application but could include malware, security events, vulnerability information, and asset information	Common mechanism leveraged by OT Security Products to collect alerts, asset info, and vulnerabilities	Remote
DBConnect	ICS Logs, Alarm Information, Configuration, Patching Info, Host Based Information	Leveraged by data historians, patching solutions, ICS systems, and other systems	Remote
WMI Collector	OS components, process & service information, applications, user accounts, security settings	Consideration should be given to scaling of WMI for large environments	Remote
HTTP Event Collector (HEC)	System health and state information	Newer products are providing methods to collect via HEC but depends on the application	Remote

For more general information about indexing data in Splunk Enterprise, please refer to the following documentation: Getting Data In.​

Integrating with specific OT Data Models

Splunk for OT Security includes several data models that can be leveraged to automatically generate asset lookups. In addition, OT partners of Splunk should populate any hardware and software data captured or created by their add-ons to these data models.

Two data models have been created to facilitate populating assets into Splunk for Enterprise Security. The most critical model​ for asset information in the Splunk OT Security Solution is​ the OT Asset​ ​ data model contained in the Splunk for OT Security app. This data model is designed to be used with hardware assets such as servers, PLC's, workstations, etc. and contains all fields in the OT Asset Framework. An additional data model also exists called OT Software Asset​ which is used to populate additional information regarding firmware, operating system, and software present on each OT asset. Together data from each can be combined to provide additional context around an asset as well as components installed on each asset.

The OT Add-on for Splunk does have specific requirements for parts of the ES Asset Framework field values and formats. These fields are used to tag and identify assets as belonging to OT systems or specific classifications. The following outline these requirements

Field	Restriction\Format	Sample
asset_system	Asset systems are often collections of site that may refer to a grouping of assets. While not require it is suggested for filtering purposes.	Western Operations
asset_type	Asset types classification the purpose or function of the asset	Historian
category	The use of static text "ot" (without quotes) is used broadly to denote which assets are part of the OT Environment.	ot | windows | nerc
classification	Classifications related to specific frameworks should follow the format - <framework>:<value>	cip:high | cip:BCA
site_id	Ideally this should be populated with a name of a facility of site where the asset may reside. It is used on multiple dashboards as a filter.	Johnson Refinery
zone	Purdue zone mappings should following the following format -- purdue:level<level #>	purdue:level3

Integration Guidance with the OT Security Add-on for Splunk

Integrating with specific OT Data Models

Splunk for OT Security includes several data models that can be leveraged to automatically generate asset lookups. In addition, OT partners of Splunk should populate any hardware and software data captured or created by their add-ons to these data models.

Two data models have been created to facilitate populating assets into Splunk for Enterprise Security. The most critical model​ for asset information in the Splunk OT Security Solution is​ the OT Asset​ ​ data model contained in the Splunk for OT Security app. This data model is designed to be used with hardware assets such as servers, PLC's, workstations, etc. and contains all fields in the OT Asset Framework. An additional data model also exists called OT Software Asset​ which is used to populate additional information regarding firmware, operating system, and software present on each OT asset. Together data from each can be combined to provide additional context around an asset as well as components installed on each asset.

The OT Add-on for Splunk has specific requirements for parts of the ES Asset Framework field values and formats. These fields are used to tag and identify assets as belonging to OT systems or specific classifications. More information on the data model fields can be found below. The following outline these requirements.

Field	Restriction\Format	Sample
ip, mac, nt_host, dns	At least one of these fields must be populated to identify an asset, multivalue values can be delimeted by using the pipe operator	172.1.1.1
asset_system	Asset systems are often collections of site that may refer to a grouping of assets. While not require it is suggested for filtering purposes.	Western Operations
category	The use of static text "ot" (without quotes) is used broadly to denote which assets are part of the OT Environment.	ot | windows | nerc
classification	Classifications related to specific frameworks should follow the format - <framework>:<value>	cip:high | cip:BCA
site_id	Ideally this should be populated with a name of a facility of site where the asset may reside. It is used on multiple dashboards as a filter.	Johnson Refinery
zone	Purdue zone mappings should following the following format -- purdue:level<level #>	purdue:level3

Overlapping Asset Information Across Sites

Some vendors will re-use IP address, DNS names, and/or host names at different locations. While the site_id field can help distinguish on dashboards, the asset framework requires customization when this occurs. This specifically requires an additional field to be added to asset lookup files used by Enterprise Security called cim_entity_zone. In many cases this can be set to be the same as the site_id but can be customized as needed. For documentation on enabling this feature see the Enabling entity zones for assets and identities in Splunk Enterprise Security documentation.

Integration with Asset Inventory

This section outlines the basic procedures that should be used to integrate data from OT Security products with ES's Asset Framework. While each product's implementation may vary, these steps should provide additional steps to validate OT Asset information and avoid issues with bad or default values. The OT Asset Data Model should serve as the guide for field names and in the search below it is assumed mapping of field names has also been completed.

1. Run the following query within the search context of the OT Security vendors add-on :

<base search>
| makemv ip delim=\"\|\"
| mvexpand ip \`\`\`split multiple ip\'s into multiple records\`\`\`
| eval ip=if(ip=\"null\", \"\", ip), nt_host=if(nt_host=\"null\", \"\", nt_host), dns=if(dns=\"null\", \"\", dns), mac=if(mac=\"null\", \"\", mac) \`\`\`remove any null string values\`\`\`
| eval ip=if(match(ip, \"\^\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\$\"), ip, \"\") \`\`\`check to verify ip is an actual ip - if you are using IPv6 you need to change this\`\`\`
| fillnull ip, dns, nt_host, mac value=\"\"
| makemv mac delim=\"\|\"
| mvexpand mac \`\`\`split multiple macs into multiple records\`\`\`
| eventstats count as mac_count by mac \`\`\`this eliminates any questionable mac\'s that may be repeated\`\`\`
| eval mac=if(mac_count \> 1, \"\", mac)
| dedup ip, dns, nt_host, mac
| table asset_id, asset_model, asset_status, asset_system, asset_type, asset_vendor, asset_version, asset_criticality, bunit, category, city, classification, country, dns, end_of_support, exposure,ip, is_expected, lat, lon, location, mac, nt_host, owner, parent_asset_id, pci_domain, priority, requires_av, serial, should_timesync, should_update, site_id, vlan, zone

This search takes care of several formatting issues that sometimes occur with fields like IP and ensures that they map a IPv4 format. In addition, some products may report mac's which are duplicated among assets, often belonging to a common switch, router, firewall, etc. Any duplicated mac are therefore considered suspect and removed from the asset information.

Note: The tag ot_asset is applied across all data sources. If you are focused only on a particular data set, please modify the query to specify only the selected host, sources, or sourcetypes appropriately.

1. Export the above results to a csv file using the export button

   • 

2. Open Settings → Lookups → Lookup table files +Add New

   • 

   • 

3. Provide the CSV file as well as an alias that will be referenced and click save

   • 

4. Change the Permissions of the lookup file in step 4 and verify is set to be shared globally

5. Open Settings → Lookups → Lookup Definitions and click +Add New

   • 

   • 

6. Create a lookup definition which references the file from step 4 and Save (note: If the file from step 4 does not show up in the lookup file dropdown list you may need to check the lookup permissions to validate it is shared globally)

   • 

7. Change the permissions on the lookup definition so it is shared globally

8. Go to the Enterprise Security App and click on the menu Configure → Data Enrichment → Asset and Identity Framework

   • 

   • 

9. Under the Asset Lookups tab click New → New Configuration

   • 

10. Specify the following required fields: Source, Name, Category, asset_type, site_id, and asset_system. You can leave the other field with their default values (note: if you cannot find your lookup definition you might need to validate permissions for the lookup definition is set to global)

11. Validate that the file is working correctly by going to the Search Preview tab and click on asset_lookup_by_str → Open in search to validate assets are showing up. If your assets do not show up, look at the troubleshooting section of this document.

    • 

OT Asset Data Model

The following is a description of the OT Asset Data Model used in the OT Security Add-on

Field	Sample	Description
asset_id	991351	Unique asset name or id
asset_model	cpu-1200	model number of other version indicator for the asset
asset_status	Operational	Operational status of the asset
asset_system	Western Operations	Local grouping of assets, often a combination of sites or facilities into a common operational system
asset_type	PLC	Type of asset such as a PLC, Historian, Engineering Workstation
asset_vendor	Siemens	Vendor and/or Product Name
asset_version	3.2	Version of device such as a firmware version
asset_criticality	Critical	Operational criticality of the asset
bunit	commercial	Business unit that the asset belongs to
category	ot|nerc|distribution	Asset category, in most cases should include the tag "ot" to identify an OT device
city	Albuquerque, NM	Geographic location, normally including city and state
classification	cip:high|cip:BCA	Regulatory or other classications
country	USA	Geographic location, normally the country where the device resides
dns	scada01.ops.local	DNS name of the asset if it exists
end_of_support	TRUE	Boolean flag indicating whether the device is no longer supported
exposure	private	Measure of network exposure
ip	172.1.1.1	IP address of the asset if it exists
is_expected	TRUE	Flag to indicate whether the device is expected to send data to Splunk
lat	-45.095	Latitiude of the asset location
location	line1.conveyer.plc	Logical description of where the asset resides within a site or facility
long	27.2345	Longitude of the asset location
mac	00:0E:8C:41:49	mac address of the asset
nt_host	scada01	Host name of the asset
owner	Field Operations	Asset owner
parent_asset_id	887456	ID if the device is connected to a parent asset
pci_domain	POS	PCI domain that the asset belongs to
priority	medium	Asset priority (combined operational and security)
description	Safety PLC for Inbound Conveyer	Description of the asset itself
requires_av	TRUE	Flag indicating whether antivirus should be installed on the asset
serial	10003456	Serial number of the asset
should_timesync	TRUE	Flag to indicate whether an asset should be monitored for time sync events
should_update	FALSE	Flag to indicate whether this asset should be monitoring for patch updates
site_id	Borading Refinery	Facility or site name
vlan	172.16.16|ot_ops	Name of the subnet or VLAN that an asset belongs to
zone	purdue:level3|corporate	Security zone that the asset resides in

OT Software Asset Data Model

The following is a description of the OT Software Asset Data Model used in the OT Security Add-on

Field	Sample	Description
asset_id	1234567	Unique ID assigned to software possibly from an asset management system
asset_name	Acrobat Reader	Name of the software, sometimes including the vendor and version information
category	Application	Category of the software (Firmware, OS, Application)
dns	scada01.ops.local	DNS name of the asset with the software installed
end_of_support	FALSE	Flag indicating whether the software is supported currently by the vendor
hash_method	SHA-2	Function used to calculate the hash value of the software
hash_value	d14a028c2a3a2bc94	Hash value of the software
install_date	1686687861	Epoch timestamp of when the software was installed
ip	172.1.1.1	IP Address of the asset with the software installed
licence_key	xxxxx-xxxxx-xxxx	Vendor provide asset_key for software activitation
mac	00:0E:8C:41:49:C6	Mac address of the asset with the software installed
nt_host	scada01	Host name of the asset with the software installed
parent_asset_id	1233445	ID of parent asset
type	FTP Client	Type of software (e.g. ICS, Historian, etc)
vendor	Adobe	Vendor who created the software
version	9.1	Version identifier for the software

Common Problems and Troubleshooting

While this guide is designed to be as comprehensive as possible, you may run into issues during installation and configuration. The larger body of Splunk Enterprise and Splunk Enterprise Security documentation at the top of this document will help you troubleshoot some of the more common issues encountered during this process. In addition, there are several gotchas that have been encountered during early adoption of the OT Security solution, you can resolve these quickly by double checking the following:

Cannot add lookups to asset or identity framework.​

If you cannot add lookups to the Asset and Identity Framework, perform the following checks:

a. Verify the permissions of the DA-ESS-OTSecurity folders and files. These should match other apps installed. Often when a file is manually extracted into the SPLUNK_HOME/etc/apps directory their permissions will be those of the person who manually installed the app. Typically installing via the web gui can avoid these problems. Also check the permissions of the transforms.conf file in the SPLUNK_HOME/etc/apps/EnterpriseSecurity/local directory since lookup definitions are written to this file.

b. Verify that the Lookup Definition has been created in the Enterprise Security App and permissions are set to share objects globally via the Lookup Definitions menu

Assets from lookup table are not showing up

Typically, when assets are not showing up in the OT Asset Investigator this is typically an issue with the data in one of the key fields, dns, ip, mac, and/or nt_host. At LEAST one of these fields must have data, should not be set to a default value such as the null string, the IP address must be a correct IPv4 or IPv6 address, and they cannot contain multi-values (the sample search provided in the . Review the data you have provided and validate that none of these contain the "null" string, multiple values, and the ip address is formatted correctly (note CIDR notation is not allowed, and the IPv4 address must contain all four octets). The search provided in the section Integration with OT Security Products step 1, should take care of these problems, and can be adapted to different data sources.

The Installation Guide provides key information on tagging and identification of OT Assets.

Risk based notables not showing up in OT Security Posture

Since the OT Add-on for Splunk relies on contextual data around assets to identify assets that are part the OT environment, it is important that macro's regarding risk objects also be utilized to provide the necessary context. When Risk Based Alerts are showing up in the incident review with OT Assets involved, but not the OT Security Posture dashboard, this contextual information may be missing (for example when using the default RBA alerts). To fix this, the query for the panel with Risk Based Alerts can be updated as follows:

Original Search

index=notable risk_object=*
| eval time=_time
| `get_asset_type(risk_object_asset_type)`
| eval key_field=CASE(
match(search_name, ".*(F|f)acility.*"), risk_object_site_id,
match(search_name, ".*(A|s)sset.*"), risk_object,
match(search_name, ".*(U|s)ser.*"), risk_object,
1=1, risk_object)
|  eval object_type=CASE(
match(search_name, ".*(F|f)acility.*"), "Facility",
match(search_name, ".*(O|o)bject.*"), "Facility",
match(search_name, ".*(A|s)sset.*"), "Asset",
match(search_name, ".*(U|s)ser.*"), "User",
1=1, risk_object)
| search `ot_identifier(risk_object_category)` \\\this line causes problems
... \\\rest of search

New Search

index=notable risk_object=*
| eval time=_time
| `get_asset_type(risk_object_asset_type)`
| eval key_field=CASE(
match(search_name, ".*(F|f)acility.*"), risk_object_site_id,
match(search_name, ".*(A|s)sset.*"), risk_object,
match(search_name, ".*(U|s)ser.*"), risk_object,
1=1, risk_object)
|  eval object_type=CASE(
match(search_name, ".*(F|f)acility.*"), "Facility",
match(search_name, ".*(O|o)bject.*"), "Facility",
match(search_name, ".*(A|s)sset.*"), "Asset",
match(search_name, ".*(U|s)ser.*"), "User",
1=1, risk_object)
| `get_asset_by("str", "risk_object")` \\\ this line is added
| search `ot_identifier(risk_object_category)` 
... \\\rest of search

The Installation Guide provides key information on tagging and identification of assets that fall under OT.

Assets and/or Identities are being combined across multiple sites

Some vendors will re-use IP address, DNS names, host names, and/or accounts at different locations. While the site_id field can help distinguish on dashboards, the asset & identity framework requires customization when this occurs. This specifically requires an additional field to be added to asset and/or identity lookup files used by Enterprise Security called cim_entity_zone. In many cases this can be set to be the same as the site_id but can be customized as needed. For documentation on enabling this feature see the Enabling entity zones for assets and identities in Splunk Enterprise Security documentation.

NERC CIP dashboards and reports are not populating automatically.​

This problem most often occurs when one of two errors occur:

a. Verify that the asset lookup has been created and the following fields exist and are populated:

Field	Format	Example
classification	cip:<low,medium, or high>|cip<BCA,PCA,TS A,EACM,EAP>	cip:high|cip:EAP|cip:EACM
category	nerc	nerc
site_id	<site name>	Pleasanton Plant
zone	eap:<zone name>	eap:PPLT

b. Data is not present in the data models or disabled. The following data models should contain data:

• Authentication
• Intrusion Detection
• Inventory
• Malware
• Network Sessions
• Network Traffic
• Updates

More information on specific dashboards and requirements can be found in the The Administration Guide