Install the Splunk Add-on for Cisco ASA in a distributed Splunk Enterprise deployment¶

Prerequisites¶

The Cisco ASA device can use TCP as the syslog transport, and can maintain an open TCP port with the syslog-ng server.
Do not place a load balancer between the ASA and the syslog server.
Implement the following DNS configurations:
- For each IP address assigned for management of the ASA, ensure both address (A) records and record route (R) records exist and match.
- For each egress NAT address assigned to the device, ensure address and record route records exist and match.
- For each ingress NAT address assigned to the device, ensure the record route record matches the internal destination address. The address record for this IP is not required.
Download the Splunk Add-on for Cisco ASA on Splunkbase.

Install the Splunk Add-on for Cisco ASA on search heads in a distributed Splunk Enterprise deployment¶

From the Splunk Web home screen, click the gear icon next to Apps.
Click Install app from file.
Locate the downloaded file and click Upload.
If Splunk Enterprise prompts you to restart, do so.
From the Splunk Web home screen, click the gear icon next to Apps.
Find the add-on and click Edit properties.
Change Visible to No.

Install an add-on on clustered indexers in a distributed Splunk Enterprise deployment¶

Use the master node to deploy add-ons to the peer nodes. Do not use a deployment server or any third-party deployment tool.

Prepare the configuration bundle¶

The set of subdirectories in the $SPLUNK_HOME/etc/master-apps directory constitute the configuration bundle.

Prepare the configuration bundle by making the following edits to the files you want to distribute to the peers. Try to combine all updates in a single bundle to reduce the impact on the work of the peer nodes:

Inspect the add-on for indexes.conf files. For each index defined in an add-on-specific indexes.conf file, set repFactor=auto so that the index is replicated across all peers.
Place the add-on in the $SPLUNK_HOME/etc/master-apps directory on the master node.

(Optional) Validate the bundle and check restart¶

Validate the bundle and test the files on a standalone test indexer to confirm that they are working correctly before distributing them to the set of peers. This helps ensure that the bundle applies across all peer nodes without problems. The validation process also provides information that is useful for debugging invalid bundles.

Use Splunk Web to validate the bundle and check restart¶

In Splunk Web for the master node instance, click Settings > Indexer Clustering.
Click Edit > Configuration Bundle Actions.
Click Validate and Check Restart > Validate and Check Restart. A message appears that indicates bundle validation and whether check restart succeeds. When bundle validation and check restart succeeds, then the bundle is acceptable for distribution to the peer nodes. Information about the validated bundle appears in Splunk Web, including whether you must restart the peer nodes.

If validation and check restart fails, then the bundle cannot be distributed to the peers. In this case, review the bundle details for information that might help you troubleshoot the issue.

Use the Command Line interface *CLI) to validate the bundle and check restart¶

Run splunk validate cluster-bundle: splunk validate cluster-bundle. This command returns a message confirming that bundle validation has started. Under certain failure conditions, the message indicates the cause of failure.

To validate the bundle and check whether you must restart Splunk, include the --check-restart parameter: splunk validate cluster-bundle --check-restart This version of the command first validates the bundle, and if validation succeeds, the command checks whether to restart the peer.

To view the status of bundle validation, run the splunk show cluster-bundle-status command. This command shows validation success or failure. If validation fails, the command provides information about the cause of failure and whether you should restart the peer.

The following example shows the output from the splunk show cluster-bundle-status command after a successful validation:

master
     cluster_status=None
     active_bundle
        checksum=576F6BBB187EA6BC99CE0615B1DC151F
        timestamp=1495569737 (in localtime=Tue May 23 13:02:17 2017)
    latest_bundle
        checksum=576F6BBB187EA6BC99CE0615B1DC151F
        timestamp=1495569737 (in localtime=Tue May 23 13:02:17 2017)
    last_validated_bundle
        checksum=1E0C4F0A7363611774E1E65C8B3932CF
        last_validation_succeeded=1
        timestamp=1495574646 (in localtime=Tue May 23 14:24:06 2017)
        last_check_restart_bundle
                checksum=1E0C4F0A7363611774E1E65C8B3932CF
                last_check_restart_result=restart required
                timestamp=1495574646 (in localtime=Tue May 23 14:24:06 2017)

Peer 1     1D00A8C2-026B-4CAF-90D6-5D5D39445569     default
    active_bundle=576F6BBB187EA6BC99CE0615B1DC151F
    latest_bundle=576F6BBB187EA6BC99CE0615B1DC151F
    last_validated_bundle=1E0C4F0A7363611774E1E65C8B3932CF
    last_bundle_validation_status=success
        last_bundle_checked_for_restart=1E0C4F0A7363611774E1E65C8B3932CF
        last_check_restart_result=restart required
    restart_required_apply_bundle=0
    status=Up
...

Where the settings are:

Notification field name	Description
`last_validated_bundle`	Identifies the newly validated bundle.
`last_validation_succeeded=1`	Indicates that validation succeeded.
`last_check_restart_result=restart required`	On the master, `last_check_restart_result=restart required` indicates that a restart is required on at least one of the cluster peers.
`last_check_restart_result=restart required`	On the peers, `last_check_restart_result=restart required` indicates that you must restart that peer.

Apply the bundle to the peers¶

To apply the configuration bundle to the peers, you can use Splunk Web or the CLI. You cannot initiate a configuration bundle push if a bundle push is currently in progress.

Use Splunk Web to apply the bundle to the peer nodes¶

To apply the configuration bundle to the peer nodes:

On the master node, in Splunk Web, click Settings > Indexer clustering.
Click Edit > Configuration Bundle Actions. The configuration bundle actions dashboard opens, and shows information on the last successful bundle push.
Click Push. A pop-up window warns you that the distribution might initiate a restart of all peer nodes.
Click Push Changes. The screen provides information on the distribution progress and whether distribution is successful.
- In the case of successful distribution, once each peer successfully validates the bundle, the master coordinates a rolling restart of all the peer nodes as needed.
- If distribution fails, the master indicates which peers could not receive the distribution so that you can resolve those peer issues. If any peer fails to accept the distribution, none of the peers will apply the bundle.
When the push is successful, the peers use their new set of configurations, now located in their local $SPLUNK_HOME/etc/slave-apps. Leave the files in this location.

Use the CLI to apply the bundle to the peer nodes¶

To apply the configuration bundle to the peers, run the following CLI command on the master: splunk apply cluster-bundle The warning message appears:

Caution: Under some circumstances, this command will initiate a rolling restart
of all peers. This depends on the contents of the configuration bundle. For
details, refer to the documentation. Do you wish to continue? [y/n]:

To proceed, type y.
- The master distributes the new configuration bundle to the peers, which then individually validate the bundle. After all peers successfully validate the bundle, the master coordinates a rolling restart of all the peer nodes, if necessary. The peers use their new set of configurations, located in their local $SPLUNK_HOME/etc/slave-apps. Leave the files in this location.
- If any peer is unable to validate the bundle, it sends a message to the master, and the master displays the error on its dashboard in Splunk Web. You must fix any problems noted by the master and rerun splunk apply cluster-bundle.

View the status of the bundle push¶

View the status of the bundle push using Splunk Web or the CLI.

Use Splunk Web to view the status of the bundle push¶

Once an app is distributed to the peers, launch and manage the app on each peer using Splunk Web. The apply cluster-bundle command takes an optional flag, --skip-validation, for use in cases where a problem exists in the validation process. Use this flag only under the direction of Splunk Support and after making sure that the bundle is valid. Do not use this flag to circumvent the validation process.

You can also validate the bundle without applying it. This is useful for debugging some validation issues.

Use the CLI to view the status of the bundle push¶

To see how the cluster bundle push is proceeding, run the following command from the master node:

splunk show cluster-bundle-status

This command tells you whether the bundle validation succeeded or failed. It also indicates the restart status of each peer.

Install an add-on on to your forwarders using a deployment server¶

Use your deployment server to distribute content and configurations (collectively called deployment apps) to deployment clients, grouped into server classes. Deployment apps can be full-fledged apps, such as those available on Splunkbase, or they can be just simple groups of configurations.

Deploy an add-on to your deployment clients¶

On your deployment server, navigate to $SPLUNK_HOME/etc/deployment-apps/.
Add your add-on to the /deployment-apps/ directory.
Extract the add-on.
Navigate to $SPLUNK_HOME/etc/deployment-apps/<APP NAME>/default/inputs.conf.
Add inputs for the data you want to collect.
Save your changes.
Restart the deployment server: /splunk restart.

View app deployment status¶

Go to the Apps tab. The tab provides information on the number of clients each app was deployed to. Click on an app to go to a detailed page for that app. The App Data Size field specifies the size of the app bundle. The bundle is a compressed file containing the app. Once a client receives a bundle, it uncompresses it and installs the app in its proper location.