Splunk App for Behavioral Profiling¶

The Splunk App for Behavioral Profiling is a collection of workflows which enable you to operationalise detection and scoring of behavioral anomalies at scale in complex environments, correlated to profile and highlight the entities which require investigation.

Features¶

Deploy Behavioral Anomaly Rules: Define and schedule behavioral indicator searches and anomaly scoring rules with the help of a guided workflow, which turns the behaviors you want to track into operationalised anomaly detection rules, outputing behavioral scores aligned to entities.
Investigate Entities: Utilise the dashboards provided to view and drill-down on the entities which have the highest behavioral scores, using the panels provided to investigate the pattern of activity and mark the entity as reviewed or allow listed if required.
Monitor Performance: Ensure your rules continue to execute effectively by monitoring their performance and output via the provided views, allowing easy adjustment if necessary to continue to allow you to find the entities that matter.

Target Users¶

Fraud Teams tasked with locating increasingly sophisticated attackers that employ evolving methods across physical and digital channels to avoid simplistic detection rules.
IT Operations supporting modern infrastructure, services and solutions comprising many disparate parts, any of which could be creeping towards an issue that compromises the resilience of the platform and brings the entire service down.
Insider Threat analysts attempting to hunt the unusual behaviours pointing towards criminal activity whilst grappling with inflexible solutions offering too little transparency and too much complexity for teams to tune to their operations.
Platform Moderators chasing malicious users, posting abusive or illegal material, directly affecting revenue and customer sentiment without detection amid a marketplace of inapplicable solutions for bespoke platforms.
Anyone who needs to understand the specific entities in their domain which are behaving anomalously compared with a wider group or the entity's historic behavior.

Demo¶

Not a reader? App demonstrations available to watch here

Installation Guide ↵

Installation¶

The Splunk App for Behavioral Profiling is available on Splunkbase

Single Instance Splunk Deployments¶

Install the app on the Splunk Enterprise/Cloud Search Head.

Distributed Splunk Deployments¶

As in single instance Splunk deployments, install the app on the Splunk Enterprise/Cloud Search Head. Note there will be separate instructions on the Configuration page for distributed Splunk environments for the indexes required by the app.

Search Head Clusters¶

The Splunk App for Behavioral Profiling can be installed in an SHC by following the standard installation instructions for the app.

Note on Potential Performance and Other Impacts:¶

The app is designed to efficiently scale, and utilises components such as summary indexing, KV storage and schedule windows to do so. That being said, as more indicator searches and scoring rules are deployed, you may see changes in the performance of your Splunk deployment - please leverage the monitor views within the app to understand where scheduled searches are contributing high load if you see performance degredation.

Dependencies:¶

The Splunk App for Behavioral Profiling is supported by both on premise and Splunk Cloud instances. Note, the versions listed below is the earliest tested, there may/will be earlier versions which support the Splunk App for Behavioral Profiling:

Splunk Enterprise/Cloud 9.0.x or greater

Required Apps:¶

To leverage the machine learning based anomaly detection functionality within the app, the user must install the below apps. Note, the versions listed are the earliest tested, there may/will be earlier versions which support the Splunk App for Behavioral Profiling:

Splunk Machine Learning Toolkit (5.4.0)
Python for Scientific Computing (4.1.0)

Configuration¶

Having installed the app for the first time, navigate to the Configure Indexes and Collections dashboard underneath the Configure dropdown menu on the navigation bar. It should look like this:

Indexes¶

The dashboard will instruct on the indexes that are required for the app to function, unlike in previous versions where these indexes were pre-defined (indicator_index and scoring_index), from version 2.0 onwards the application utilises search macros which need to be defined as indexes present on the Splunk instance:

indicator_index - the index used to store output of behavioral indicator searches.
scoring_index - the index used to store output of anomaly scoring rules.

The dashboard will advise on whether the index macros are defined or missing and what the current retention settings are. If the app is being installed for the first time they will NOT be defined.

Creating new indexes on Single Instance Splunk Deployments¶

Index definitions may be created using the indexes page in Splunk Cloud, but should usually be deployed using configuration files in Splunk Enterprise, both methods are outlined here.

Creating new indexes on Distributed Splunk Deployments¶

Deploying indexes in a distributed environment has different implications which are discussed at length in the Splunk docs.

Note that for either method on the configuration dashboard there is a link to a sample indexes.conf template for two example indexes.

Defining the search macros¶

Once indexes have been created, use the links on the dashboard to navigate to the search macro definitions. The only input required is the name of the index under the Definition field:

Collections¶

The dashboard will also instruct on the collections required by the app to store metadata associated with searches built by the user, they are:

indicator_metadata
scoring_search_metadata

By default these collections should be defined within the app. If the dashboard produces False for collection_is_defined investigate the default/collections.conf file in the app directory.

Ended: Installation Guide

Administration Guide ↵

Background¶

The Splunk App for Behavioral Profiling operates around the concepts of entities and behavioural anomalies:

An Entity is considered any mappable group of “things” represented in data which you wish to profile using the app. Examples include: customers, employees, business units, applications, servers, branches, etc.
A Behavioral Indicator Search is a scheduled search which tracks a behavior metric exhibited by the group of entities. It populates the index defined by the search macro indicator_index.
A Behavioral Anomaly is a deviation from expected behavior; either a difference between an entity and its peers, an entity and its historic behavior, or some combination of the two.
An Anomaly Scoring Rule is a scheduled search which determines which of the Behavioral Indicator values are considered behavior anomalies and assigns scores to them. It populates the index defined by the search macro scoring_index.

Operationalising behavioral anomaly detections at scale, and resolving back to the entities they relate to, has in the past been a complex task with Splunk. It required understanding and implementation of:

Splunk’s Search Processing Language (SPL) to define a search
Search scheduling to operationalise that search
Machine learning and ML-SPL to define and interpret anomaly criteria

The Splunk App for Behavioral Profiling comes to the rescue by orchestrating all of the above behind a simple click-through workflow, enabling you to deploy behavioral indicator searches with one line of SPL, and introducing a simple scoring mechanism to focus attention away from the false positives and towards the entities that truly matter.

App Architecture¶

The app uses a three layer architecture to turn your raw data sources into behavior profiled entities:

Behavioral Indicator Searches are deployed to track entity behaviors and output the metrics to the indicator_index on a user-defined schedule.
The defined indicator_index is then used as the feed for the Anomaly Scoring Rules, which leverage criteria ranging from simple conditional logic (i.e. indicator value > 5000) to machine learning based anomaly detection, enabling identification of the anomalously behaving entities and attribute dynamic scores in the scoring index.
The defined scoring_index then populates the Entity Behavioral Scores dashboard to display a list of entities that are prioritized by aggregated behavioral score for investigation.

Building a Behavioral Indicator Search:¶

Note: Please ensure you've deployed both the index defining search macros required by this app before building an indicator for the first time. You can validate by going to Configure > Configure Indexes and Collections within the app

Build a new behavioral indicator search by enterring the Behavioral Indicators page underneath the Deploy dropdown menu on the navigation bar and clicking Create Behavioral Indicator. Behavioral indicator searches are effectively KPIs and are used to turn entity data points into statistical aggregations which are used to profile entity behaviour against their peers and historic behaviour.

The Create Behavioral Indicator workflow allows you to build and schedule indicator searches which are used to populate the defined indicator index in the following steps:

Define the data source for the indicator search
Select the entity field and any other relevant fields
Define the indicator search
Save and schedule the indicator search

1) Select Mode¶

Within the Splunk App for Behavioral Profiling there are two modes to build new indicators in, these are chosen by toggling the radio bar and can be switched between at any point up until saving the behavioral indicator by returning to the Select Mode step by using the next/back buttons in the bottom right hand corner of the screen:

Guided Mode: Defining the indicator is performed using a workflow interface to select the data points and statistical aggregations required by the indicator. This is the recommended approach for the majority of use cases and for SPL query language beginners.

Custom Mode: Defining the indicator is performed by entering your own custom SPL query at the "Define Data Source" step. This is only recommended where the guided workflow doesn't support your use case.

2) Define Data Source¶

Define and filter the data source for your indicator search. For guided mode this is simply pointing the workflow towards the base dataset which you're going to build an indicator with, for custom mode the query needs to generate the behavioral indicator search and the Entity Field (see below) must be defined.

3) Select Fields (Guided Mode)¶

Select the Entity Field which represents and identifies the entity you are building the behavourial indicator for, this should be field for which the values uniquely resolve to the individual entities you're monitoring.

Additionally any other fields which will be used to either calculate the indicator, or provide context in investigation of an indicator value need to be defined in the Other Fields(s) selection box here.

4) Define Indicator (Guided Mode)¶

This step of the process is done via the search bar for custom mode, in guided mode this is where the field(s) defined in the previous step are used to calculate a behavioral indicator using the input boxes to specify a statistical function and chosen field(s) to calculate the metric(s) used as the behavioral indicator against your entity field.

There's the additional option to split the timespan into bins, so that the indicator can be split by this span for each entity - providing more indicator values each time the search is run.

5) Save & Schedule¶

Specify a name, description and schedule for the behavioral indicator. These are used, when Save is clicked and Submit selected in the pop up modal, to generate and schedule a saved search which runs on the provided schedule and outputs to the defined indicator_index. If backfill is ticked, the search will additionally run immediately over the selected time window so it can populate views and be used to build a new anomaly scoring rule.

Additionally, logging is provided on the build status of the search, its associated KV metadata and backfill.

Building an Anomaly Scoring Rule:¶

Note: Please ensure you've defined both the index definitions required by this app before building an anomaly scoring rule for the first time (which you can validate by going to Configure > Configure Indexes and Collections within the app) in addition to at least one behavioral indicator search

Following creation of at least one behavioral indicator search, you can deploy a new anomaly scoring rule by navigating to the Anomaly Scoring Rules page underneath the Deploy dropdown menu on the navigation bar and selecting Create Anomaly Scoring Rule. Anomaly scoring rules are used to attribute scores to entities based on the output of underlying Behavior Indicator Searches which they take as an input. Together anomaly scoring rules create an aggregated profile of entity behavior and a ranking across your set of entities. The Create Anomaly Scoring Rule workflow guides you through the following steps:

Choose the anomaly defining mode the rule will use
Select the behavioral indicator profiled
Define the scoring criteria based on the chosen scoring method
Size the scores attributed to events which meet the scoring criteria
Save and schedule the scoring rule

1) Select Mode¶

The first step is to choose which mode your rule will operate in. The options fall into two categories:

Entity Group Rules where the anomaly criteria is the same across the entity group (this is the recommended approach for groups of entities where "normal" behaviour is likely to be similar across the entity group or where there is unlikely to be enough historic data to create per entity baselines). This category comes in three variants:
- Conditional: events are filtered for scoring based on a comparison between their scoring field value and a static input.
- Statistical: anomalies are determined by appearing outside a threshold based on standard deviations away from the mean indicator value.
- Machine Learning: a distribution profile of the scoring field is defined and a threshold tolerance is specified to detect anomalous values with machine learning.
Entity Specific Rules where each entity will have its own statistical baseline defining normal behavior for an indicator.

You can change between these options at any time in the anomaly scoring rule creation process by returning to this screen.

2) Define Indicator Source¶

Next, you must define the underlying settings which set the context of the scoring rule. This consists of selecting the behavioral indicator search, and scoring field within the output of that indicator search, which will be used as an input for the scoring criteria logic. Addionally, the time range set here is the window used for calculating example anomalies in the rest of the creation workflow.

3) Anomaly Criteria¶

Entity Group Scoring¶

Conditional¶

This method enables you to create a basic conditional logic function using a static comparison, entered in the free text box, which is compared using the dropdown menu input, to the defined scoring field. Any values which fulfil the logic criteria are marked as outliers and scored.

Statistical¶

This method calculates a mean value for your chosen scoring field and calculates upper and lower bounds for an outlier threshold by calculating standard deviations away from the mean. You can adjust the max upper and min lower bound values manually for specific data sets (where for example a value above or below would be impossible) via the advanced settings toggle and alter the number of standard deviations used as well as the number of bins represented on the distribution histogram chart.

Machine Learning¶

This method utilises the machine learning toolkit density function to profile the distribution of the chosen scoring field and detect outliers based on your input Threshold from the slider, with outliers shown on the histogram and in the table to its left side. By default the Distribution Type is Auto which runs all distribution types from the dropdown menu and produces the optimal result, but you can also specify a preferred type based on the distribution of your data.

Entity Specific Scoring¶

This method enables you to define a stastical baseline for your entities, using a set standard deviation to define the non-outlier threshold for your entities and the option to have the outlier threshold for each entity split further to profile their behaviour at different times of the day, week or month. Additionally, you can adjust the max upper and min lower bound values manually for specific data sets (where for example a value above or below would be impossible) via the advanced settings toggle.

4) Scoring Method¶

Having specified the logic which decides which behavioral indicator values trigger a scoring event, the next step is to use the dropdown menus to select a methodology and input scoring value for score sizing of these events. Currently the options are either Static which attributes the value in the Scoring Value text box to the event or Proportional which attributes the multiple of this value and the scoring field value, which has been determined to be an outlier, to the event.

5) Save & Schedule¶

Specify a name, description and schedule for the anomaly scoring rule. These are used, when Save is clicked and Submit selected in the pop up modal, to generate and schedule a saved search which runs on the provided schedule and outputs to the defined scoring_index. If backfill is ticked, the search will additionally run immediately over the selected time window so it can populate views and be used to build a new anomaly scoring rule.

If deploying an anomaly scoring rule leveraging the Entity Specific method, you will also have to enter the refresh frequency and method, chosing between KV Store and Lookup File, for the baseline object here. KV Store is reccomended here for all instances in which your Splunk user permissions support creation of collections.conf and transforms.conf stanzas.

Additionally, logging is provided on the build status of the search, its associated KV metadata and backfill.

Investigating Entities:¶

In order to accelerate time to value, there are several dashboards within the app which will populate with data as soon as scoring rules are set up and enabled.

Entity Behavioral Scores:¶

This dashboard takes the input of a particular set of entities, defined by the Entity Field dropdown which lists every entity associated with an anomaly scoring rule within the environment and provides a consolidated view of the scores for a given window provided by the Profile Window dropdown.

This consists of a Entity Behavior Scores section which lists the total score across all entities within the window (and a comparison with previous windows through the timechart beneath), the distribution of all the entity scores in a histogram for ease of score model health monitoring and perhaps most importantly a ranked list showing entities prioritised by behavioural score (which also includes change from average score for that entity across recent windows and the specific rules their indicator values have triggered). This list is also clickable and drills down to the Single Entity Profile dashboard.

From the list you can select to either Allow List Entity which removes the entity from all indicator and scoring searches permanently (editable via a created lookup) or Mark Reviewed to indicate to yourself and colleagues that an entity has been investigated today.

Additionally there is a Score Rule Health section which contains information on the number/percentage of rules associated with the entity which triggered within the given profile window, a pie chart showing the split of the total score attributed between the various scoring rules and a timeline showing the frequency particular rules triggered.

Single Entity Profile:¶

This dashboard takes the input of a particular set of entities, defined by the Entity Field dropdown which lists every entity associated with a scoring rule within the environment and a chosen entity from that set (entered in the text box input) to display the profile of the chosen entity across a given Profile Window

This consists of a Score Profile section which lists the total score for the entity within the window (and a comparison with previous windows through the sparkline beneath), a radial indicator displaying where that score sits percentile wise in comparison to all of the entities within the entity group for the given window and a table showing the raw number of entities with scores below, equal to and above the chosen entity - providing context around the entity's behavioural profile.

Additionally there is a Score By Rule section which contains a pie chart showing the split of the total score attributed between the various scoring rules and a timeline showing the frequency and attribution particular rules triggered. Clicking on any given rule in either chart will open a panel beneath showing both the raw events and attribution events for the selected rule and entity in the given profile window

Managing Existing Behavioral Indicators/Scoring Rules:¶

Developing effective behavioural models is both an ongoing process and one with many moving parts on the Splunk backend. As such, there are views developed which enable you to monitor the health of your deployed searches, both Behavioral Indicators and Scoring Rules, and to drilldown to edit existing rules when required.

Review Indicators:¶

This dashboard provides a selectable table of all the indicator searches which are defined on the instance. Listed is their name, owner, schedule, status (enabled/disabled), specified base search and relevant fields. Additionally an edit button links to the underlying saved search where the user can edit the SPL, schedule or enable/disable the search.

Selecting a search from the table opens up two sections of panels:

Search Execution Monitoring: a selection of views displaying the searches which have run successfully/failed and of those which ran successfully produced events or didn't, The number of events returned by recent searches and the average run time of the searches. These panels are designed to highlight any issues with the search logic or failure by the search to run.
Indicator Data Analysis: a pair of panels summarising the content of the indicator fields. The first covers statistics on the numeric values of various fields within the indicator search and the second shows the cardinality of field values, which is useful for gauging the range of non-numeric values represented and can aid investigation into search execution failures.

Review Scoring Rules:¶

This dashboard provides a selectable table of all the scoring rule searches which are defined on the instance. Listed is their name, owner, schedule, status (enabled/disabled), specified base search and relevant fields. Additionally an edit button links to the underlying saved search where the user can edit the SPL, schedule or enable/disable the search.

Selecting a search from the table opens up four panels:

Search Execution Summary: This panel shows the history of the execution for the selected scoring rule.
Search Runtime: This panel shows statistics for the selected rule’s search run-time over time. If you observe significant increases in search runtime or large variations, you may wish to investigate the load on your platform to ensure that searches are sufficiently optimized.
Data Cardinality: This panel shows cardinality for important fields and entity fields indexed by the scoring rule. Significant and unexpected changes in these values may be cause for further investigation of changes to the underlying data.
Event Count: This panel shows the number of events generated by this rule’s search over time. This may increase or decrease as entities and data volumes change, but if significant changes are observed unexpectedly, you may need to investigate your data sources. This panel also shows a sum of total scores assigned to all entities associated with this rule.

Deleting Behavioral Indicator Searches/Anomaly Scoring Rules:¶

Developing effective behavioral models both an ongoing process and one with many moving parts on the Splunk backend. As such, during the iteration of your models there will undoubtedly be times in which you require to remove content from the Splunk instance. To achieve this there are two delete pages, one for Behavioral Indicator Searches and one for Anomaly Scoring Rules.

Delete Behavioral Indicators:¶

To delete a behavioral indicator search, click Behavioral Indicators under Deploy on the navigation bar and select Delete Behavioral Indicators. From here, to remove a previously created behavioral indicator from the application, select the delete action in the dispayed table. This will remove:

The scheduled saved search associated with the behavioral indicator
Metadata for the behavioral indicator from indicator_metadata KV store

This will NOT remove previously generated behavioral indicator values from the defined indicator_index or any of the displayed anomaly scoring rules which rely on the behavioral indicator search as a dependency.

Delete Anomaly Scoring Rules:¶

To delete an anomaly scoring rule, click Anomaly Scoring Rules under Deploy on the navigation bar and select Delete Anomaly Scoring Rule. From here, to remove a previously created anomaly scoring rule from the application, select the delete action in the dispayed table. This will remove:

The scheduled saved search associated with the anomaly scoring rule
Metadata for the behavioral indicator from scoring_search_metadata KV store
if applicable, any searches, collections.conf and transforms.conf stanzas associated with the baseline used by the search

This will NOT remove previously generated scored anomaly events from the defined scoring_index or any of the displayed behavioral indicator searches which contribute to the anomaly scoring rule.

Ended: Administration Guide

Release Notes¶

2.0.2¶

Bug Fix - search on "Manage Anomaly Scoring Rule" dashboard: The dashboard panel which provides the selectable list of anomaly scoring rules previously searched over an index called scoring_index as opposed to the scoring_index macro used by the application.
App leverages @splunk/splunk-utils@3.0.0: previous versions of the app leveraged older versions of the splunk-utils package which feature references to deprecated v1 search APIs, this has been upgraded to ensure Splunk Cloud compatibility.

2.0.0¶

Entirely new workflow for search creation: the workflows for building both behavioral indicator searches and anomaly scoring rules have both been re-engineered from scratch to utilise React based Splunk UI components as opposed to SimpleXML based Splunk classic dashboards. This introduces a number of improvements:
- Streamlined workflow: instead of dashboard panels, each stage of search creation is a different step in the navigation bar - improving readability and clarity of current progress.
- Consistent back-end searches: All searches deployed in the building of a behavioral indicator search or anomaly scoring rule leverage the Splunk SearchJob class. This avoids issues with previous releases where a mixture of JS based SearchJob searches and dashboard panel searches leveraging tokens made for a fragile architecture prone to bugs where users couldn't search certain data sets or move back through the workflow to make changes if desired. It also means the application now makes heavy use of post process searches, improving efficiency and reducing load times.
- Common UI with wider Splunk portfolio: the use of Splunk UI design components make the application familiar to users of the wider Splunk product portfolio and as a result increases comprehension of the Splunk App for Behavioral Profiling.
indicator_index and scoring_index replaced with macros: indexes used for storing output by behavioral indicator searches and anomaly scoring rules now leverage macros defined by the user, called "indicator_index" and "scoring_index" as opposed to being hardcoded to those index names - this allows usage of any visible indexes on the Splunk instance by the application.
Configure Indexes and Collections dashboard updates: dashboard updated to reflect new system for defining indexes leveraged by application searches as outlined above.
Deletion workflows: new workflows accessible from the "Behavioral Indicators" and "Anomaly Scoring Rules" pages support deletion of all backend configuration related to behavioral indicator searches and anomaly scoring rules through the UI.
Lookup file based baselines: the Entity Specific Statistical anomaly scoring rule method now supports lookup file based baseline creation in addition to the existing KV Store based option.
Improved event drilldown from Single Entity Profile: the click through drilldown action to raw events from the Single Entity Profile investigative dashboard no longer includes the _indicator_search macro in the search and so provides a clearer view of the events.
Indicator search macros usage removed: to improve output readibility of the behavioral indicator searches generated by the application, the searches no longer make use of the single_mode_indicator_search and timechart_mode_indicator_search search macros. These will be removed from macros.conf in a later version.
Teminology changes: scoring rules have been renamed to anomaly scoring rules within the application and supporting collateral.

1.2.2¶

"Today" search window: Both the Single Entity Profile and Entity Behavioral Scores dashboards now include a "Today" search window.
Bug Fix - Single Entity Profile score attributions drilldown: on the Single Entity Profile Score Attribution drilldown none of the fields in the search were passed with quotation marks. This meant any fields with whitespace would result in a search error.

1.2.0¶

Reviewer user displayed: the user who marks an entity as reviewed on the Entity Behavioral Scores dashboard is now stored and displayed.
Bug Fix - indexes.conf.txt: indexes.conf.txt now correctly provides a template for scoring_index as opposed to score_index.
Bug Fix - Quotation marks in indicator searches: the workflow to create a new indicator search previously didn't correctly escape quotation marks in the initial search query, this resulted in failure to build the associated KV store entry and prevented use of the indicator search in the scoring rule workflow.

1.1.0¶

New official app logo: new app logo added to app menus.
Streamlined new scoring rule indicator preview: table showing data populated by selected indicator rule on new scoring rule screen now shows a tighter subset of relevant fields.
Tweaked scheduling for scoring rules: hourly scoring rules now trigger on the 5th minute of the hour as opposed to the 15th by default.

Known Issues/Bugs:¶

"Today" time window doesn't support marking an entity as reviewed: Whilst using the "Today" time window on the Entity Behavioral Scores dashboard users are unable to mark an entity as reviewed due to issues with the application attempting to find the end timestamp being used.
Density doesn't scale correctly: when defining anomaly criteria as part of a new anomaly scoring rule, the distribution histogram calculates a density of each histogram bin - currently however this is displayed to a fixed Y axis as opposed to an automatically scaled one which can result in poor readability.

Submit a bug by emailing rtruman@splunk.com