The Splunk Operator automates the installation of Enterprise Security (ES) with support for other premium apps coming in the future releases. This page documents the prerequisites, installation steps, troublshooting steps, and limitations of deploying premium apps using the Splunk Operator.
2.2.0 which requires Splunk Enterprise 9.0.3-a2 or later. Per the Splunk Enterprise and Enterprise Security version compatibility matrix, Splunk ES versions 7.1.0, 7.0.2, 7.0.1, 7.0.0 or 6.6.2 are supported currently.The architectures that support automated deployment of ES by using the Splunk Operator include:
Notably, if deploying a distributed search environment, the use of indexer clustering is required to ensure that the necessary ES-specific configuration is pushed to the indexers via the Cluster Manager.
The Splunk Operator installs the necessary ES components depending on the architecture specified by the applied CRDs.
For Standalone Splunk instances and standalone search heads the Operator will install Splunk ES and all associated domain add-ons (DAs), and supporting add-ons (SAs).
When installing ES in a Search Head Cluster, the Operator will perform the following tasks:
1) Install the splunk enterprise app in Deployer’s etc/apps directory.
2) Run the ES post install command essinstall that stages the Splunk ES and all associated domain add-ons (DAs) and supporting add-ons (SAs) to the etc/shcluster/apps.
3) Push the Search Head Cluster bundle from the deployer to all the SHs.
When installing ES in an indexer clustering environment through the Splunk Operator it is necessary to manually extract and deploy the supplemental Splunk_TA_ForIndexers app from the ES package to the indexer cluster members via the cluster manager. This can be achieved using the AppFramework app deployment steps using appSources scope of “cluster”.
When crafting your Custom Resource(CR) to create a Splunk Enterprise Deployment it is necessary to take the following configurations into account.
When you install ES versions 6.3.0 or higher, you must supply a value for the parameter ssl_enablement that is required by the ES post installation command essinstall. By default, if you don’t set any value for ssl_enablement, the value of strict is used which requires Splunk to have SSL enabled in web.conf (refer to setting enableSplunkWebSSL). The below table can be used for reference of available values of ssl enablement.
| SSL mode | Description |
|---|---|
| strict | Default mode. Ensure that SSL is enabled in the web.conf configuration file to use this mode. Otherwise, the installer exists with an error. Note that for the SHC, the ES post install command essinstall is run on the deployer, and this command looks into web.conf files under etc/shcapps to validate that enableSplunkWebSSL is set to true or not. This is done with the assumption that you have already pushed a web.conf through etc/shcapps to all the SHC members. |
| auto | Enables SSL in the etc/system/local/web.conf configuration file. This mode is not supported by SHC |
| ignore | Ignores whether SSL is enabled or disabled. This option may be handy if you don’t want operator and essinstall to check SSL is enabled for Splunk web and continue ES installation without interruption. For example, you may have some processes outside of the operator where you are already checking that your Splunk deployment is web SSL enabled. |
The operator uses the following CR spec parameters to install the ES app on Splunk:
scope - use premiumApps
premiumAppsProps –> type: use enterpriseSecurity
esDefaults –> sslEnablement: possible value ignore, auto or strict – please see more details ssl_enablement values in the table above.
appSources:
- name: esApp
location: es_app/
scope: premiumApps
premiumAppsProps:
type: enterpriseSecurity
esDefaults:
sslEnablement: ignore
In the following examples, you can find the YAML files for installing ES in a Splunk Standalone, Splunk SHC, and Indexer Cluster.
This example sets sslEnablement=ignore for a standalone CR. Change the setting to suit your requirements, either auto or strict. Before you use this example, copy the ES app package into the folder “security-team-apps/es_app” in your S3 bucket
apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
name: example
finalizers:
- enterprise.splunk.com/delete-pvc
spec:
replicas: 1
appRepo:
appsRepoPollIntervalSeconds: 60
defaults:
volumeName: volume_app_repo
scope: local
appSources:
- name: esApp
location: es_app/
scope: premiumApps
premiumAppsProps:
type: enterpriseSecurity
esDefaults:
sslEnablement: ignore
volumes:
- name: volume_app_repo
storageType: s3
provider: aws
path: security-team-apps/
endpoint: https://s3-us-west-2.amazonaws.com
region: us-west-2
secretRef: splunk-s3-secret
In order to use strict mode of sslEnablement, you can enable SSL on splunkd using the extraEnv variable SPLUNK_HTTP_ENABLESSL as shown below:
apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
name: example
finalizers:
- enterprise.splunk.com/delete-pvc
spec:
extraEnv:
- name: SPLUNK_HTTP_ENABLESSL
value : "true"
replicas: 1
appRepo:
appsRepoPollIntervalSeconds: 60
defaults:
volumeName: volume_app_repo
scope: local
appSources:
- name: esApp
location: es_app/
scope: premiumApps
premiumAppsProps:
type: enterpriseSecurity
esDefaults:
sslEnablement: strict
volumes:
- name: volume_app_repo
storageType: s3
provider: aws
path: security-team-apps/
endpoint: https://s3-us-west-2.amazonaws.com
region: us-west-2
secretRef: splunk-s3-secret
Use the following steps to install ES on a Splunk deployment with an SHC integrated with Indexer Cluster:
The operator will poll this bucket after configured appsRepoPollIntervalSeconds and install the Splunk_TA_ForIndexers.
Following example shows creating a Splunk deployment with a SHC, ClusterManager and a IndexerCluster. Note the difference between the values of “scope” property for the SHC and ClusterManager. For the SHC, scope is set to “premiumApps” whereas for the ClusterManager, scope is set to “cluster”
apiVersion: enterprise.splunk.com/v4
kind: SearchHeadCluster
metadata:
name: shc-es
finalizers:
- enterprise.splunk.com/delete-pvc
spec:
appRepo:
appsRepoPollIntervalSeconds: 60
defaults:
volumeName: volume_app_repo
scope: local
appSources:
- name: esApp
location: es_app/
scope: premiumApps
premiumAppsProps:
type: enterpriseSecurity
esDefaults:
sslEnablement: ignore
volumes:
- name: volume_app_repo
storageType: s3
provider: aws
path: security-team-apps/
endpoint: https://s3-us-west-2.amazonaws.com
region: us-west-2
secretRef: splunk-s3-secret
clusterManagerRef:
name: cm-es
---
apiVersion: enterprise.splunk.com/v4
kind: ClusterManager
metadata:
name: cm-es
finalizers:
- enterprise.splunk.com/delete-pvc
spec:
appRepo:
appsRepoPollIntervalSeconds: 60
defaults:
volumeName: volume_app_repo
scope: local
appSources:
- name: esAppIndexer
location: es_app_indexer_ta/
scope: cluster
volumes:
- name: volume_app_repo
storageType: s3
provider: aws
path: security-team-apps/
endpoint: https://s3-us-west-2.amazonaws.com
region: us-west-2
secretRef: splunk-s3-secret
---
apiVersion: enterprise.splunk.com/v2
kind: IndexerCluster
metadata:
name: idc-es
finalizers:
- enterprise.splunk.com/delete-pvc
spec:
clusterManagerRef:
name: cm-es
replicas: 3
When using strict mode, the following additional steps are required so that the “splunk web ssl” is enabled for the SHC. These steps are necessary before you can install ES app with the strict mode of sslEnablement. Alternatively, if you are managing enableSplunkWebSSL setting outside of the Operator scope by pushing an app bundle through the deployer, then you can skip these steps.
Steps:
enableSplunkWebSSL=trueFollowing is an example to enable Splunk Web SSL through operator on SHC:
apiVersion: enterprise.splunk.com/v4
kind: SearchHeadCluster
metadata:
name: shcssl
finalizers:
- enterprise.splunk.com/delete-pvc
spec:
extraEnv:
- name: SPLUNK_HTTP_ENABLESSL
value : "true"
appRepo:
appsRepoPollIntervalSeconds: 60
defaults:
volumeName: volume_app_repo
scope: local
appSources:
- name: coreapps
scope: cluster
location: coreapps/
volumes:
- name: volume_app_repo
storageType: s3
provider: aws
path: security-team-apps/
endpoint: https://s3-us-west-2.amazonaws.com
region: us-west-2
secretRef: splunk-s3-secret
Ensure that the ES app package is present in the specified AppFramework S3 location with the correct appSources scope. Additionally, if configuring an indexer cluster, ensure that the Splunk_TA_ForIndexers app is present in the ClusterManager AppFramework S3 location with the appSources “cluster” scope.
Apply the specified custom resource(s), the Splunk Operator will handle installation and the environment will be ready to use once all pods are in the “Ready” state which may take upto 30 minutes. Please refer to the examples above for creating your YAML files.
After installing ES you have other steps to complete:
Deploy add-ons to Splunk ES - Technology add-ons (TAs) which need to be installed to indexers can be installed via AppFramework, while TAs that reside on forwarders will need to be installed manually or via third party configuration management.
Setup Integration with Splunk Stream (optional)
Configure and deploy indexes - The indexes associated with the packaged DAs and SAs will automatically be pushed to indexers when using indexer clustering, this step is only necessary if it is desired to configure any custom index configuration. Additionally, any newly installed technical ad-ons which are not included with the ES package may require index deployment.
To upgrade ES, move the new ES package into the specified AppFramework bucket. This will initiate a pod reset and begin the process of upgrading the new version. In indexer clustering environments, it is also necessary to move the new Splunk_TA_ForIndexers app to the Cluster Manager’s AppFramework bucket that deploys apps to cluster members.
The upgrade process will preserve any knowledge objects that exist in app local directories.
Be sure to check the ES upgrade notes for any version specific changes.
The following logs can be useful to check the ES app installation progress:
Splunk operator log:
kubectl logs <operator_pod_name>
Checking if there were any errors during ES install
ES post install failures: Log shows one or more entries with the following content: “premium scoped app package install failed” followed by (specific failure reasons show up here)
Example of error when you strict mode for sslEnablement and Splunk web is not SSL enabled
2022-12-07T00:17:36.780549729Z ERROR handleEsappPostinstall premium scoped app package install failed {"controller": "searchheadcluster", "controllerGroup": "enterprise.splunk.com", "controllerKind": "SearchHeadCluster", "SearchHeadCluster": {"name":"shc1","namespace":"default"}, "namespace": "default", "name": "shc1", "reconcileID": "83133a29-ca0d-46cc-9ae5-6f26385d4506", "name": "shc1", "namespace": "default", "pod": "splunk-shc1-deployer-0", "app name": "splunk-enterprise-security_702.spl", "stdout": "", "stderr": "FATAL: Error in 'essinstall' command: You must have SSL enabled to continue\n", "post install command": "/opt/splunk/bin/splunk search '| essinstall --ssl_enablement strict --deployment_type shc_deployer' -auth admin:`cat /mnt/splunk-secrets/password`", "failCount": 1, "error": "command terminated with exit code 17"}
To fix this error, you have one of the following options: 1) Enable splunk web SSL- details on the section Special consideration while using ssl enabled mode of strict in SHC 2) Or, use sslEnablement=ignore
Example of a connection timeout error:
2022-12-07T00:48:11.542927588Z ERROR handleEsappPostinstall premium scoped app package install failed {"controller": "searchheadcluster", "controllerGroup": "enterprise.splunk.com", "controllerKind": "SearchHeadCluster", "SearchHeadCluster": {"name":"shc1it","namespace":"default"}, "namespace": "default", "name": "shc1it", "reconcileID": "b34966a2-e716-428f-a0c6-7611812e6b24", "name": "shc1it", "namespace": "default", "pod": "splunk-shc1it-deployer-0", "app name": "splunk-enterprise-security_702.spl", "stdout": "", "stderr": "FATAL: Error in 'essinstall' command: (InstallException) \"install_apps\" stage failed - Splunkd daemon is not responding: ('Error connecting to /services/apps/shc/es_deployer: The read operation timed out',)\n", "post install command": "/opt/splunk/bin/splunk search '| essinstall --ssl_enablement ignore --deployment_type shc_deployer' -auth admin:`cat /mnt/splunk-secrets/password`", "failCount": 2, "error": "command terminated with exit code 17"}
To fix this error, check the following areas: 1) Check the splunkd log in the deployer pod for any issues. 2) Check the splunkdConnectionTimeout setting in web.conf.
Logs of the respective pods:
kubectl logs <pod_name>
Check the pod log (e.g deployer pod) in case you want to monitor the pod while it is coming up to ready state or it has gone to an error state.
Common issues that may be encountered are :