Troubleshooting Guide

Step 1 — Preflight checks

The first step of troubleshooting any SCMA issue is ensuring the PreFlight checks on the 1 - Set Up dashboard are not showing any errors (highlighted in red).

The customer must resolve all issues displayed in the Message field before proceeding. The PreFlight checks are self-documenting and explain exactly what needs to be remediated.

Step 2 — Verify all Splunk instances are in Distributed Search

Navigate to Settings → Distributed search → Search peers and validate that:

• Multiple instances are connected and healthy (no "Down" messages)
• Replication Status shows "Successful" for all peers

Sometimes the customer needs to add missing instances or fix existing ones by re-entering the admin username and password and saving the distributed search peers.

Once all instances are added and issues are resolved, launch the Monitoring Console app under Settings, open Settings / General Setup, select all instances, and click Apply Changes. If you see "New" in the State column, new instances were detected.

Step 3 — Ensure the Monitoring Console is configured properly

Launch the Monitoring Console (Settings → Monitoring Console icon). Confirm it looks like the Overview page for a distributed deployment — all expected instances should be visible and grouped correctly.

Under Settings → Server settings → General settings, ensure:

• The Splunk server name at the top matches the Default host name under "Index settings".

index=_introspection sourcetype=scma:check

Step 4 — SCMA Discovery output

In the 2 - Checks dashboard, click the SCMA Discovery button. Allow the job to run for a few minutes and review the results.

Pay attention to the num_searches field. A large number indicates the app discovered a Search Head or Search Head Cluster dispatching many searches that was not added to the Monitoring Console configuration. In that case, return to Step 2 and work with the customer to add the missing instances.

If the customer confirms those servers are test instances or are not part of the environment being migrated, they can be ignored and you can proceed to the next step.

Once SCMA Discovery completes successfully, click the Run Checks button to begin the full assessment.

Step 5 — Troubleshooting dashboard

A troubleshooting dashboard is available under Tools → Troubleshooting. This dashboard helps you understand what is occurring while searches are running.

The first panel shows a table of your current results. Click any row to run that search manually.

Scenario 1 — Many checks did not complete

If a large number of checks are still showing red (incomplete), do not re-run the 2-Checks dashboard. Instead:

1. Scroll to the bottom of the Troubleshooting dashboard.
2. Click the hyperlink at the bottom of the page — this launches the check runner search. Send it to the background immediately to extend the timeout to 24 hours (the default lifetime of a search is only 10 minutes).
3. Note the order of the checks. Some checks depend on the results of previous checks. For example, check 200 falls between checks 20 and 22 in the sort order.
4. Scroll back up and refresh panels periodically to confirm that more green (completed) statuses are appearing.

Scenario 2 — A few checks did not complete

If only a small number of checks remain incomplete, click on each incomplete check in the Troubleshooting dashboard to run the related search.

1. Click the incomplete check row in the Troubleshooting dashboard table.
2. Send each job to the background to extend its timeout to 24 hours.
3. Refresh the dashboard panel to confirm each check completes.