splunk-operator

Custom Resource Guide

The Splunk Operator provides a collection of custom resources you can use to manage Splunk Enterprise deployments in your Kubernetes cluster.

For examples on how to use these custom resources, please see Configuring Splunk Enterprise Deployments.

Metadata Parameters

All resources in Kubernetes include a metadata section. You can use this to define a name for a specific instance of the resource, and which namespace you would like the resource to reside within:

Key Type Description
name string Each instance of your resource is distinguished using this name.
namespace string Your instance will be created within this namespace. You must ensure that this namespace exists beforehand.

If you do not provide a namespace, you current context will be used.

apiVersion: enterprise.splunk.com/v3
kind: Standalone
metadata:
  name: s1
  namespace: splunk
  finalizers:
  - enterprise.splunk.com/delete-pvc

The enterprise.splunk.com/delete-pvc finalizer is optional, and may be used to tell the Splunk Operator that you would like it to remove all the Persistent Volumes associated with the instance when you delete it.

Common Spec Parameters for All Resources

apiVersion: enterprise.splunk.com/v3
kind: Standalone
metadata:
  name: example
spec:
  imagePullPolicy: Always
  livenessInitialDelaySeconds: 400
  readinessInitialDelaySeconds: 390
  extraEnv:
  - name: ADDITIONAL_ENV_VAR_1
    value: "test_value_1"
  - name: ADDITIONAL_ENV_VAR_2
    value: "test_value_2"
  resources:
    requests:
      memory: "512Mi"
      cpu: "0.1"
    limits:
      memory: "8Gi"
      cpu: "4"

The spec section is used to define the desired state for a resource. All custom resources provided by the Splunk Operator include the following configuration parameters:

Key Type Description
image string Container image to use for pod instances (overrides RELATED_IMAGE_SPLUNK_ENTERPRISE environment variable
imagePullPolicy string Sets pull policy for all images (either “Always” or the default: “IfNotPresent”)
livenessInitialDelaySeconds number Sets the initialDelaySeconds for Liveness probe (default: 300)
readinessInitialDelaySeconds number Sets the initialDelaySeconds for Readiness probe (default: 10)
extraEnv EnvVar Sets the extra environment variables to be passed to the Splunk instance containers. WARNING: Setting environment variables used by Splunk or Ansible will affect Splunk installation and operation
schedulerName string Name of Scheduler to use for pod placement (defaults to “default-scheduler”)
affinity Affinity Kubernetes Affinity rules that control how pods are assigned to particular nodes
resources ResourceRequirements The settings for allocating compute resource requirements to use for each pod instance. The default settings should be considered for demo/test purposes. Please see Hardware Resource Requirements for production values.
serviceTemplate Service Template used to create Kubernetes Services

Common Spec Parameters for Splunk Enterprise Resources

apiVersion: enterprise.splunk.com/v3
kind: Standalone
metadata:
  name: example
spec:
  etcVolumeStorageConfig:
    storageClassName: gp2
    storageCapacity: 15Gi
  varVolumeStorageConfig:
    storageClassName: customStorageClass
    storageCapacity: 25Gi
  volumes:
    - name: licenses
      configMap:
        name: splunk-licenses
  licenseMasterRef:
    name: example
  clusterMasterRef:
    name: example
  serviceAccount: custom-serviceaccount

The following additional configuration parameters may be used for all Splunk Enterprise resources, including: Standalone, LicenseMaster, SearchHeadCluster, ClusterMaster and IndexerCluster:

Key Type Description
etcVolumeStorageConfig StorageClassSpec Storage class spec for Splunk etc volume as described in StorageClass
varVolumeStorageConfig StorageClassSpec Storage class spec for Splunk var volume as described in StorageClass
volumes Volume List of one or more Kubernetes volumes. These will be mounted in all container pods as as /mnt/<name>
defaults string Inline map of default.yml overrides used to initialize the environment
defaultsUrl string Full path or URL for one or more default.yml files, separated by commas
licenseUrl string Full path or URL for a Splunk Enterprise license file
licenseMasterRef ObjectReference Reference to a Splunk Operator managed LicenseMaster instance (via name and optionally namespace) to use for licensing
clusterMasterRef ObjectReference Reference to a Splunk Operator managed ClusterMaster instance (via name and optionally namespace) to use for indexing
monitoringConsoleRef string Logical name assigned to the Monitoring Console pod. You can set the name before or after the MC pod creation.
serviceAccount ServiceAccount Represents the service account used by the pods deployed by the CRD

LicenseMaster Resource Spec Parameters

apiVersion: enterprise.splunk.com/v3
kind: LicenseMaster
metadata:
  name: example
spec:
  volumes:
    - name: licenses
      configMap:
        name: splunk-licenses
  licenseUrl: /mnt/licenses/enterprise.lic

Please see Common Spec Parameters for All Resources and Common Spec Parameters for All Splunk Enterprise Resources. The LicenseMaster resource does not provide any additional configuration parameters.

Standalone Resource Spec Parameters

apiVersion: enterprise.splunk.com/v3
kind: Standalone
metadata:
  name: standalone
  labels:
    app: SplunkStandAlone
    type: Splunk
  finalizers:
  - enterprise.splunk.com/delete-pvc

In addition to Common Spec Parameters for All Resources and Common Spec Parameters for All Splunk Enterprise Resources, the Standalone resource provides the following Spec configuration parameters:

Key Type Description
replicas integer The number of standalone replicas (defaults to 1)

SearchHeadCluster Resource Spec Parameters

apiVersion: enterprise.splunk.com/v3
kind: SearchHeadCluster
metadata:
  name: example
spec:
  replicas: 5

In addition to Common Spec Parameters for All Resources and Common Spec Parameters for All Splunk Enterprise Resources, the SearchHeadCluster resource provides the following Spec configuration parameters:

Key Type Description
replicas integer The number of search heads cluster members (minimum of 3, which is the default)

ClusterMaster Resource Spec Parameters

ClusterMaster resource does not have a required spec parameter, but to configure SmartStore, you can specify indexes and volume configuration as below -

apiVersion: enterprise.splunk.com/v3
kind: ClusterMaster
metadata:
  name: example-cm
spec:
  smartstore:
    defaults:
        remotePath: $_index_name
        volumeName: msos_s2s3_vol
    indexes:
      - name: salesdata1
        remotePath: $_index_name
        volumeName: msos_s2s3_vol
      - name: salesdata2
        remotePath: $_index_name
        volumeName: msos_s2s3_vol
      - name: salesdata3
        remotePath: $_index_name
        volumeName: msos_s2s3_vol
    volumes:
      - name: msos_s2s3_vol
        path: <remote path>
        endpoint: <remote endpoint>
        secretRef: s3-secret

IndexerCluster Resource Spec Parameters

apiVersion: enterprise.splunk.com/v3
kind: IndexerCluster
metadata:
  name: example
spec:
  replicas: 3
  clusterMasterRef: 
    name: example-cm

Note: clusterMasterRef is required field in case of IndexerCluster resource since it will be used to connect the IndexerCluster to ClusterMaster resource.

In addition to Common Spec Parameters for All Resources and Common Spec Parameters for All Splunk Enterprise Resources, the IndexerCluster resource provides the following Spec configuration parameters:

Key Type Description
replicas integer The number of indexer cluster members (defaults to 1)

MonitoringConsole Resource Spec Parameters

cat <<EOF | kubectl apply -f -
apiVersion: enterprise.splunk.com/v3
kind: MonitoringConsole
metadata:
  name: example_mc
  finalizers:
  - enterprise.splunk.com/delete-pvc
EOF

Use the Monitoring Console to view detailed topology and performance information about your Splunk Enterprise deployment. See What can the Monitoring Console do? in the Splunk Enterprise documentation.

The Splunk Operator now includes a CRD for the Monitoring Console (MC). This offers a number of advantages available to other CR’s, including: customizable resource allocation, app management, and license management.

The MC pod is referenced by using the monitoringConsoleRef parameter. There is no preferred order when running an MC pod; you can start the pod before or after the other CR’s in the namespace. When a pod that references the monitoringConsoleRef parameter is created or deleted, the MC pod will automatically update itself and create or remove connections to those pods.

Examples of Guaranteed and Burstable QoS

You can change the CPU and memory resources, and assign different Quality of Services (QoS) classes to your pods using the Kubernetes Quality of Service section. Here are some examples:

A Guaranteed QoS Class example:

Set equal requests and limits values for CPU and memory to establish a QoS class of Guaranteed.

Note: A pod will not start on a node that cannot meet the CPU and memory requests values.

Example: The minimum resource requirements for a Standalone Splunk Enterprise instance in production are 24 vCPU and 12GB RAM.

apiVersion: enterprise.splunk.com/v3
kind: Standalone
metadata:
  name: example
spec:
  imagePullPolicy: Always
  resources:
    requests:
      memory: "12Gi"
      cpu: "24"
    limits:
      memory: "12Gi"
      cpu: "24"  

A Burstable QoS Class example:

Set the requests value for CPU and memory lower than the limits value to establish a QoS class of Burstable.

Example: This Standalone Splunk Enterprise instance should start with minimal indexing and search capacity, but will be allowed to scale up if Kubernetes is able to allocate additional CPU and Memory up to the limits values.

apiVersion: enterprise.splunk.com/v3
kind: Standalone
metadata:
  name: example
spec:
  imagePullPolicy: Always
  resources:
    requests:
      memory: "2Gi"
      cpu: "4"
    limits:
      memory: "12Gi"
      cpu: "24"  

A BestEffort QoS Class example:

With no requests or limits values set for CPU and memory, the QoS class is set to BestEffort. The BestEffort QoS is not recommended for use with Splunk Operator.

Pod Resources Management

CPU Throttling

Kubernetes starts throttling CPUs if a pod’s demand for CPU exceeds the value set in the limits parameter. If your nodes have extra CPU resources available, leaving the limits value unset will allow the pods to utilize more CPUs.

POD Eviction - OOM

As oppose to throttling in case of CPU cycles starvation, Kubernetes will evict a pod from the node if the pod’s memory demands exceeds the value set in the limits parameter.