Validation Webhook

The Splunk Operator includes an optional validation webhook that validates Splunk Enterprise Custom Resource (CR) specifications before they are persisted to the Kubernetes API server. This provides immediate feedback when invalid configurations are submitted.

Overview

The validation webhook intercepts CREATE and UPDATE operations on Splunk Enterprise CRDs and validates the spec fields according to predefined rules. If validation fails, the request is rejected with a descriptive error message.

Supported CRDs

The webhook validates the following Custom Resource Definitions:

Standalone
IndexerCluster
SearchHeadCluster
ClusterManager
LicenseManager
MonitoringConsole

Enabling the Validation Webhook

The validation webhook is disabled by default and must be explicitly enabled. This is an opt-in feature for the v4 API.

Prerequisites

Before enabling the webhook, ensure you have:

cert-manager installed in your cluster (required for TLS certificate management)

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.0/cert-manager.yaml
kubectl wait --for=condition=Available --timeout=300s deployment/cert-manager -n cert-manager
kubectl wait --for=condition=Available --timeout=300s deployment/cert-manager-webhook -n cert-manager

Deployment Options

Option 1: Use the Webhook-Enabled Kustomize Overlay

Deploy using the config/default-with-webhook overlay which includes all necessary webhook components:

# Build and apply the webhook-enabled configuration
kustomize build config/default-with-webhook | kubectl apply -f -

Option 2: Enable Webhook on Existing Deployment

If you already have the operator deployed, you can enable the webhook by setting the ENABLE_VALIDATION_WEBHOOK environment variable:

kubectl set env deployment/splunk-operator-controller-manager \
  ENABLE_VALIDATION_WEBHOOK=true -n splunk-operator

Note: This option also requires the webhook service, ValidatingWebhookConfiguration, and TLS certificates to be deployed. Use Option 1 for a complete deployment.

Option 3: Modify Default Kustomization

Edit config/default/kustomization.yaml to uncomment the webhook-related sections:

Uncomment - ../webhook in the bases section
Uncomment - ../certmanager in the bases section
Uncomment - manager_webhook_patch.yaml in patchesStrategicMerge
Uncomment - webhookcainjection_patch.yaml in patchesStrategicMerge
Uncomment the vars section for certificate injection

Then deploy:

make deploy IMG=<your-image> SPLUNK_GENERAL_TERMS="--accept-sgt-current-at-splunk-com"

Validated Fields

The webhook validates the following spec fields:

Common Fields (All CRDs)

Field	Validation Rule	Error Message
`spec.etcVolumeStorageConfig.storageCapacity`	Must match format `^[0-9]+Gi$` (e.g., “10Gi”, “100Gi”)	must be in Gi format (e.g., ‘10Gi’, ‘100Gi’)
`spec.varVolumeStorageConfig.storageCapacity`	Must match format `^[0-9]+Gi$`	must be in Gi format (e.g., ‘10Gi’, ‘100Gi’)
`spec.etcVolumeStorageConfig.storageClassName`	Required when `ephemeralStorage=false` and `storageCapacity` is set	storageClassName is required when using persistent storage
`spec.varVolumeStorageConfig.storageClassName`	Required when `ephemeralStorage=false` and `storageCapacity` is set	storageClassName is required when using persistent storage

CRD-Specific Fields

CRD	Field	Validation Rule
Standalone	`spec.replicas`	Must be ≥ 0
IndexerCluster	`spec.replicas`	Must be ≥ 3
SearchHeadCluster	`spec.replicas`	Must be ≥ 3

SmartStore Validation (Standalone, ClusterManager)

SmartStore configuration is validated only when provided:

Field	Validation Rule
`spec.smartstore.volumes[*].name`	Required (non-empty)
`spec.smartstore.volumes[*]`	Either `endpoint` or `path` must be specified
`spec.smartstore.indexes[*].name`	Required (non-empty)
`spec.smartstore.indexes[*].volumeName`	Required (non-empty)

AppFramework Validation (Standalone, ClusterManager, SearchHeadCluster)

AppFramework configuration is validated only when provided:

Field	Validation Rule
`spec.appRepo.appSources[*].name`	Required (non-empty)
`spec.appRepo.appSources[*].location`	Required (non-empty)
`spec.appRepo.volumes[*].name`	Required (non-empty)

Example Validation Errors

Invalid Replicas

apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: example
spec:
  replicas: -1  # Invalid: negative value

Error:

The Standalone "example" is invalid: .spec.replicas: Invalid value: -1: should be a non-negative integer

Invalid Storage Configuration

apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: example
spec:
  etcVolumeStorageConfig:
    storageCapacity: "10GB"  # Invalid: must use Gi suffix

Error:

The Standalone "example" is invalid: spec.etcVolumeStorageConfig.storageCapacity: Invalid value: "10GB": must be in Gi format (e.g., '10Gi', '100Gi')

Missing SmartStore Volume Name

apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: example
spec:
  smartstore:
    volumes:
      - name: ""  # Invalid: empty name
        endpoint: "s3://bucket"

Error:

The Standalone "example" is invalid: spec.smartstore.volumes[0].name: Required value: volume name is required

Verifying Webhook Deployment

Check Webhook Pod is Running

kubectl get pods -n splunk-operator
# Expected: splunk-operator-controller-manager-xxx   1/1   Running

Check Certificate is Ready

kubectl get certificate -n splunk-operator
# Expected: splunk-operator-serving-cert   True   webhook-server-cert

Check Webhook is Registered

kubectl get validatingwebhookconfiguration splunk-operator-validating-webhook-configuration

Check Operator Logs

kubectl logs -n splunk-operator deployment/splunk-operator-controller-manager | grep -i webhook
# Look for: "Validation webhook enabled via ENABLE_VALIDATION_WEBHOOK=true"
# Look for: "Starting webhook server" {"port": 9443}

Troubleshooting

Webhook Not Being Called

Verify the ValidatingWebhookConfiguration exists:

kubectl get validatingwebhookconfiguration splunk-operator-validating-webhook-configuration -o yaml

Check that the CA bundle is injected:

kubectl get validatingwebhookconfiguration splunk-operator-validating-webhook-configuration \
  -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | head -1
# Should show: -----BEGIN CERTIFICATE-----

Verify webhook service endpoints:

kubectl get endpoints -n splunk-operator splunk-operator-webhook-service
# Should show an IP address

Certificate Issues

Check cert-manager logs:

kubectl logs -n cert-manager deployment/cert-manager

Check certificate status:

kubectl describe certificate -n splunk-operator splunk-operator-serving-cert

Check issuer:
```
kubectl get issuer -n splunk-operator
```

Webhook Disabled

If you see “Validation webhook disabled” in the logs, ensure:

The ENABLE_VALIDATION_WEBHOOK environment variable is set to true
You’re using the correct kustomize overlay (config/default-with-webhook)

Architecture

The validation webhook consists of:

Component	Description
Webhook Server	HTTP server listening on port 9443 with TLS
Validator Registry	Maps CRD types to their validation functions
ValidatingWebhookConfiguration	Kubernetes resource that registers the webhook
Certificate	TLS certificate managed by cert-manager
Service	Kubernetes service exposing the webhook endpoint

Request Flow

User submits a CREATE/UPDATE request for a Splunk CRD
Kubernetes API server intercepts the request
API server sends an AdmissionReview to the webhook service
Webhook server validates the spec fields
Webhook returns Allowed/Denied response
If allowed, the resource is persisted; if denied, user receives error

Disabling the Webhook

To disable the webhook after it has been enabled:

kubectl set env deployment/splunk-operator-controller-manager \
  ENABLE_VALIDATION_WEBHOOK=false -n splunk-operator

Or redeploy using the default kustomization (without webhook):

make deploy IMG=<your-image> SPLUNK_GENERAL_TERMS="--accept-sgt-current-at-splunk-com"