splunk-operator

SmartStore Resource Guide

NOTE: The below method is a temporary way of installing SmartStore configuration & indexes. Starting from the Splunk Operator release 1.0.2, an enhanced App installation framework is introduced which is the recommended method to install SmartStore indexes & configuration. The below method may still be used to specify the S3 access keys, which avoids storing them in the S3 buckets (via the App installation framework)

The Splunk Operator includes a method for configuring a SmartStore remote storage volume with index support using a Custom Resource. The SmartStore integration is not implemented as a StorageClass. This feature and its settings rely on support integrated into Splunk Enterprise. See SmartStore for information on the feature and implementation considerations.

• SmartStore configuration is supported on these Custom Resources: Standalone and ClusterManager.
• SmartStore support in the Splunk Operator is limited to Amazon S3 & S3-API-compliant object stores only if you are using the CRD configuration for S3 as described below.”
• For Amazon S3, if you are using interface VPC endpoints with DNS enabled to access AWS S3, please update the corresponding volume endpoint URL with one of the DNS names from the endpoint. Please ensure that the endpoint has access to the S3 buckets using the credentials configured. Similarly other endpoint URLs with access to the S3 buckets can also be used.
• Use of GCS with SmartStore is supported by using configuration via Splunk App.
• Specification allows definition of SmartStore-enabled indexes only.
• Already existing indexes data should be migrated from local storage to the remote store as a pre-requisite before configuring those indexes in the Custom Resource of the Splunk Operator. For more details, please see Migrate existing data on an indexer cluster to SmartStore.

SmartStore configuration involves indexes, volumes, and the volume credentials. Indexes and volume configurations are configured through the Custom Resource specification. The credentials for accessing the volume can either be done through IAM roles or static credential. For roles these can be configured via service accounts or annotations. For static keys the volume credentials are configured securely in a Kubernetes secret object, and that secret object is referred by the Custom Resource with SmartStore volume spec, through SecretRef

Storing Smartstore Secrets

Here is an example command to encode and load your static remote storage volume secret key and access key in the kubernetes secret object: kubectl create secret generic <secret_store_obj> --from-literal=s3_access_key=<access_key> --from-literal=s3_secret_key=<secret_key>

Example: kubectl create secret generic s3-secret --from-literal=s3_access_key=iRo9guRpeT2EWn18QvpdcqLBcZmW1SDg== --from-literal=s3_secret_key=ZXvNDSfRo64UelY7Y4JZTO1iGSZt5xaQ2

Creating a SmartStore-enabled Standalone instance

1. Configure remote store credentials by either:
   • Configure IAM role based credentials via service account or annotations, or
   • Create a Secret object with Secret & Access credentials, as explained in Storing SmartStore Secrets
2. Confirm your S3-based storage volume path and URL.
3. Confirm the name of the Splunk indexes being used with the SmartStore volume.
4. Create/Update the Standalone custom resource specification with volume and index configuration (see Example below)
5. Apply the custom resource specification: kubectl -f apply Standalone.yaml

Example. Standalone.yaml:

apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: s1
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  smartstore:
    defaults:
      volumeName: s2s3_vol
    indexes:
      - name: networkmonitor
        volumeName: s2s3_vol
        remotePath: $_index_name
      - name: salesdata
      - name: oslogs
    volumes:
      - name: s2s3_vol
        path: indexdata-s2-bucket/standaloneNodes/s1data/
        endpoint: https://s3-us-west-2.amazonaws.com
        secretRef: s3-secret

1. In the above example, indexdata-s2-bucket is the bucket name on remote storage, standaloneNodes/s1data is the relative path on that bucket in which the index data is stored.
2. There are 3 indexes defined in the above config example, i.e networkmonitor, salesdata and oslogs.
3. defaults: section is configured with the s3 volumeName parameter. An index can override with a specific volumeName, as shown for networkmonitor index. (Note: Unless multiple S3 volumes are used, specifying the volumeName in defaults: section makes it simple, and no need to repeat it across all the indexes)
4. If the remotePath is not explicitly specified, a default value of $_index_name is assumed. For the semantics of $_index_name, please see indexes.conf in Splunk docs.

The SmartStore parameters will be placed into the required .conf files in an app. The app is named as splunk-operator. In the case of a standalone deployment, the app is located at /opt/splunk/etc/apps/

Note: Custom apps with higher precedence can potentially overwrite the index and volume configuration in the splunk-operator app. Hence, care should be taken to avoid conflicting SmartStore configuration in custom apps. See Configuration file precedence order

Creating a SmartStore-enabled Indexer Cluster

1. Configure remote store credentials by either:
   • Configure IAM role based credentials via service account or annotations, or
   • Create a Secret object with Secret & Access credentials, as explained in Storing SmartStore Secrets
2. Confirm your S3-based storage volume path and URL.
3. Confirm the name of the Splunk indexes being used with the SmartStore volume.
4. Create/Update the Cluster Manager custom resource specification with volume and index configuration (see Example below)
5. Apply the custom resource specification: kubectl -f apply Clustermanager.yaml
6. Follow the rest of the steps to Create an Indexer Cluster. See Examples

Example. Clustermanager.yaml:

apiVersion: enterprise.splunk.com/v4
kind: ClusterManager
metadata:
  name: <name>
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  smartstore:
    volumes:
      - name: <remote_volume_name>
        path: <remote_volume_path>
        endpoint: https://s3-<region>.amazonaws.com
        secretRef: <secret_store_obj>
    indexes:
      - name: <index_name_1>
        remotePath: $_index_name
        volumeName: <remote_volume_name>
      - name: <index_name_2>
        remotePath: $_index_name
        volumeName: <remote_volume_name>
      - name: <index_name_3>
        remotePath: $_index_name
        volumeName: <remote_volume_name>

The SmartStore parameters will be placed into the required .conf files in an app. The app is named as splunk-operator. In case of a Indexer cluster deployment, the app is located on Cluster manager at /opt/splunk/etc/manager-apps/. Once the SmartStore configuration is populated to Cluster Manager’s splunk-operator app, Operator issues a bundle push command to Cluster Manager, so that the SmartStore configuration is distributed to all the peers in that indexer cluster

Note: Custom apps with higher precedence can potentially overwrite the index and volume configuration in the splunk-operator app. Hence, care should be taken to avoid conflicting SmartStore configuration in custom apps. See Configuration file precedence order

SmartStore Resource Spec Parameters

There are additional SmartStore settings available for tuning and storage management. The settings are equivalent to the SmartStore settings defined in indexes.conf and server.conf for Splunk Enterprise. The SmartStore resource applies to the Standalone and ClusterManager Custom Resources, and adds the following Spec configuration parameters:

smartstore:
  description:
    Splunk Smartstore configuration. Refer to indexes.conf.spec and
    server.conf.spec on docs.splunk.com
  properties:
    cacheManager:
      description: Defines Cache manager settings
      properties:
        evictionPadding:
          description: Additional size beyond 'minFreeSize' before eviction kicks in
          type: integer
        evictionPolicy:
          description: Eviction policy to use
          type: string
        hotlistBloomFilterRecencyHours:
          description:
            Time period relative to the bucket's age, during which the bloom
            filter file is protected from cache eviction
          type: integer
        hotlistRecencySecs:
          description:
            Time period relative to the bucket's age, during which the bucket is
            protected from cache eviction
          type: integer
        maxCacheSize:
          description: Max cache size per partition
          type: integer
        maxConcurrentDownloads:
          description:
            Maximum number of buckets that can be downloaded from remote storage
            in parallel
          type: integer
        maxConcurrentUploads:
          description: 
            Maximum number of buckets that can be uploaded to remote storage in
            parallel
          type: integer
      type: object
    defaults:
      description: Default configuration for indexes
      properties:
        maxGlobalDataSizeMB:
          description: 
            MaxGlobalDataSizeMB defines the maximum amount of space for warm and
            cold buckets of an index
          type: integer
        maxGlobalRawDataSizeMB:
          description: 
            MaxGlobalDataSizeMB defines the maximum amount of cumulative space
            for warm and cold buckets of an index
          type: integer
        volumeName:
          description: Remote Volume name
          type: string
      type: object
    indexes:
      description: List of Splunk indexes
      items:
        description: IndexSpec defines Splunk index name and storage path
        properties:
          hotlistBloomFilterRecencyHours:
            description: 
              Time period relative to the bucket's age, during which the bloom
              filter file is protected from cache eviction
            type: integer
          hotlistRecencySecs:
            description: 
              Time period relative to the bucket's age, during which the bucket
              is protected from cache eviction
            type: integer
          maxGlobalDataSizeMB:
            description: 
              MaxGlobalDataSizeMB defines the maximum amount of space for warm
              and cold buckets of an index
            type: integer
          maxGlobalRawDataSizeMB:
            description: 
              MaxGlobalDataSizeMB defines the maximum amount of cumulative space
              for warm and cold buckets of an index
            type: integer
          name:
            description: Splunk index name
            type: string
          remotePath:
            description: Index location relative to the remote volume path
            type: string
          volumeName:
            description: Remote Volume name
            type: string
        type: object
      type: array
    volumes:
      description: List of remote storage volumes
      items:
        description: VolumeSpec defines remote volume name and remote volume URI
        properties:
          endpoint:
            description: Remote volume URI
            type: string
          name:
            description: Remote volume name
            type: string
          path:
            description: Remote volume path
            type: string
          secretRef:
            description: Secret object name
            type: string
        type: object
      type: array
  type: object

Following is the table that maps the Custom Resource SmartStore spec to the Splunk docs.

See indexes.conf, and server.conf for more information about these configuration details.

Custom Resource Spec	Splunk Config	Splunk Stanza
volumeName + remotePath	remotePath	[<index name>], [default] in indexes.conf
maxGlobalDataSizeMB	maxGlobalDataSizeMB	[<index name>], [default] in indexes.conf
maxGlobalRawDataSizeMB	maxGlobalRawDataSizeMB	[<index name>], [default] in indexes.conf
hotlistRecencySecs	hotlist_recency_secs	[<index name>], [cachemanager]
hotlistBloomFilterRecencyHours	hotlist_bloom_filter_recency_hours	[<index name>], [cachemanager]
endpoint	remote.s3.endpoint	[volume:<name>]
path	path	[volume:<name>]
maxConcurrentUploads	max_concurrent_uploads	[cachemanager]
maxConcurrentDownloads	max_concurrent_downloads	[cachemanager]
maxCacheSize	max_cache_size	[cachemanager]
evictionPolicy	eviction_policy	[cachemanager]
evictionPadding	eviction_padding	[cachemanager]

Additional configuration

There are SmartStore/Index config settings that are not covered by the Custom Resource SmartStore spec. If there is a need to configure additional settings, this can be achieved by configuring the same via Apps:

1. Create an App with the additional configuration For example, in order to set the remote S3 encryption scheme as sse-s3, create an app with the config in indexes.conf file under default/local sub-directory as follows:

   [volume:\<remote_volume_name\>]]
   path = <remote_volume_path>
   remote.s3.encryption = sse-s3
   

2. Apply the CR with the necessary & supported Smartstore and Index related configs
3. Install the App created using the currently supported methods (Note: This can be combined with the previous step)

RepFactor for Internal Indexes

For Indexer Cluster, all Smartstore enabled indexes must have repFactor set to auto. However, by default, the Cluster Manager’s _cluster app(by default) sets the repFactor to 0 for internal indexes like _metrics, _introspection, _telemetry, _metrics_rollup, and _configtracker. If you want to replicate these indexes, please follow the instructions in the Additional Configuration section to set the repFactor to auto.

This site is open source. Improve this page.