Frequently Asked Questions

Issue: Some versions of Splunk had an issue with using the xpath command within SPL. This occurred in Splunk Enterprise version 8.1.4.

Resolution: Upgrade Splunk on the instance hosting the Monitoring Console to the latest version of 8.1.x (or upgrade to a later Splunk version). Note that upgrading to later versions will require additional planning time.

Issue: The KV Store is not running. This can be caused by several different reasons.

Resolutions:

• Expired TLS certificates: The TLS certificates associated with MongoDB have expired. Without valid certificates, the service will not start. Run the following command on the Monitoring Console instance to check the expiration date:

  cd /opt/splunk/bin/
  openssl x509 -enddate -noout -in ../etc/auth/server.pem

  If the certificate is expired, run:

  mv /opt/splunk/etc/auth/server.pem /opt/splunk/etc/auth/server-backup.pem
  /opt/splunk/bin/splunk restart

  When SSL does not exist and the instance is restarted, Splunk automatically creates a new SSL certificate valid for 1097 days (3 years). Perform this operation on all instances with expired or soon-to-expire SSL certificates. Perform this on all affected instances.
• MongoD lock file owned by root: A lock file is present and is owned by the root user rather than the Splunk user. Remove the lock file and change ownership of /opt/splunk to the Splunk user.
• KV Store path permissions: The KV Store path is in a directory the Splunk user does not own. Change the owner or adjust the configuration to use the default location: /opt/splunk/var/lib.
• KV Store disabled in server.conf: The customer has disabled the KV Store on the Monitoring Console. Remove the configuration setting. The default is:

  [kvstore]
  disabled = false

Option A — Automatic redaction

1. After installing the app, open savedsearches.conf in the default/ subfolder and perform a search/replace:
   • Search for and remove: `scma_comment("#REDACT
   • Search for and remove: #REDACT")`
   In vi:

   :%s/`scma_comment("#REDACT//g
   :%s/#REDACT")`//g

2. Save the file.
3. Run a debug/refresh by appending /en-US/debug/refresh to the URL, or restart the instance.
4. Open the SCMA App, go to Tools → Search, run the following command and immediately send it to the background:

   | scma order="5,9,10,16,17,19,28,33,34,37,38,39,40,41,43,44,45,47,50,51,52,54,57,59,61,67,71,77,99"

5. Re-run the data export. Download and check each JSON file for validation.

Option B — Manual redaction

1. Go to the Troubleshooting dashboard. Use filters to focus on checks that may contain sensitive information. Click each item to run it.
2. For any fields containing sensitive data, add an eval command using the Splunk md5() function just before scma_summarize_data. Separate multiple fields with commas.
3. Re-run the data export and validate that the sensitive data has been redacted.

Validation

/* Option 1: Search raw data */
earliest=-1d@d `scma_source_introspection_index`
sourcetype=scma:check check_name!=vv_customer_details
customer_name="Partners Demo"
| eval check_time=coalesce(check_time, now())
| eventstats latest(check_time) AS check_time by check_name
| eval diff= abs(check_time - _time)
| where diff < 1
| search app=*sensitive*

/* Option 2: Search JSON formatted export data */
| savedsearch "SCMA Data Export"
| search data="*sensitive*"
| table check_name data

Answer: As of today, the SCMA app is not supported in Splunk Cloud.

Customers who need similar functionality in a cloud environment can consider:

• Insights App for Splunk (IA4S): Provides functionality similar to SCMA, especially for health checks. Approved and self-service in both Classic and Victoria — no case required. Customers may open an ODS case for deployment assistance. The app ships with a repository of health searches and many dashboards for in-depth analysis.
• Splunk App for Chargeback: Helps customers understand SVC usage from a business perspective.
• Splunk Cloud Monitoring Console (CMC): Provides workload visibility and SVC usage analysis.

Answer: Most sensitive fields are excluded altogether from SCMA output. This includes fields such as:

• host
• savedsearches
• user
• And other personally identifiable or environment-specific values

For cases where additional redaction is needed, see FAQ-004 for manual and automatic redaction options.

Answer: Yes. In SCMA v5.2.5+, a new SCMA Review dashboard allows results to be reviewed directly within the app without exporting.

Steps:

1. Upgrade to SCMA version ≥ 5.2.5.
2. Launch the 1 - Setup dashboard and remediate any PreFlight checks highlighted in red.
3. Answer all questions in 1 - Set up (bottom of dashboard) and in 3 - Scoping and Planning.
4. Re-run the discovery job in 2 - Checks (Step 1) and remediate any issues.
5. Re-run the checks in 2 - Checks (Step 2).
6. Launch the SCMA Review dashboard from the Tools menu.

Check whether the License Manager logs are being forwarded to the production cluster being searched by the MC. Test with:

index=_internal source=*license_usage.log TERM(Usage)

If no data is returned, the LM instance is not forwarding its logs, or the logs are going to a different indexer or indexer cluster. This commonly occurs when a customer has multiple Splunk environments (prod, pre-prod, test) with a shared License Manager.

If the customer does not have a Monitoring Console, install the SCMA app on the Search Head with the most visibility into their environment. See Standard installation for details.

Ensure all relevant Splunk instances are added as search peers on that Search Head before running the checks.

The SCMA Review dashboard has a Diagnose icon that can be used to get more information about individual checks — including what the check is looking for and why it may have failed or produced unexpected results.

Also refer to the Checks Reference for detailed documentation on every check.

host_role_id (also called the name field in SCMA) is the redacted Splunk instance name. For example:

• An ES instance → es-01, es-02, …
• A clustered ES → es-shc-01
• A core SHC → shc-01
• Ad-hoc search heads → sh-01, sh-02, …
• Clustered indexers → idxc-01, …

label is used by individual checks to group results. The redacted version of the label is the host_role_id field.

If searches are not functioning within the SCMA App, investigate distsearch.conf. You may need to rebuild or correct it.

A reliable indicator is: adding splunk_server=local to a search returns data, while without it returns nothing. This confirms a distsearch.conf issue with the default stanza.

Example of a correctly configured distsearch.conf:

[distributedSearch:dmc_group_indexer]
default = true
servers = <indexer list>

Rebuild the distsearch.conf by reconfiguring the Monitoring Console under Settings → General Setup, selecting all instances, and clicking Apply Changes.