Configure Azure Event Hub inputs for the Splunk Add-on for Microsoft Cloud Services¶

Configure your inputs on the Splunk platform instance responsible for collecting data for this add-on, usually a heavy forwarder. You can configure inputs using Splunk Web, which is a best practice, or by using the configuration files.

Add-on prerequisites¶

Configure an Active Directory Application in Azure Active Directory for the Splunk Add-on for Microsoft Cloud Services
Connect to your Azure App Account with Splunk Add-on for Microsoft Cloud Services

Azure Event Hub prerequisites¶

Perform the following prerequisites before configuring an Azure Event Hub input:

Configure an Azure Event Hub for each log category in Azure, such as Azure Active Directory, Resource, and Activity. For more information, see the Quickstart: Create an Event Hub using Azure portal topic in the Microsoft Azure documentation for more information.
Authorize access to Azure Event Hubs by giving Azure Event Hubs Data Receiver permissions to each applicable Azure application. See the Authorize access to Azure Event Hubs topic in the Microsoft Azure documentation for more information..
Splunk Cloud customers who are installing this add-on on the Inputs Data Manager (IDM) and want to collect Event Hub data, must use the Admin Configuration Service (ACS) to configure outbound ports 5671/tcp and 5672/tcp (Advanced Message Queuing Protocol (AMQP) specification) to connect to their target Azure address. By default IDM’s can only go out on port 443.

Scaling¶

On your Azure deployment, a scaling best practice is to configure a ratio of at least one Event Hub throughput unit for each partition. For example, if you have 20 throughput units, the best practice is to configure 20 partitions. For more information on Event Hub throughput scalability, see the https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-scalability#throughput-units in the Microsoft Azure documentation.

On the Splunk software side, the number of Event Hub inputs that you create as consumers on an Event Hub must be less than or equal to the number of partitions that you have on the Event Hub. For more information, see the https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features Event Hubs in the Microsoft Azure documentation.

Limitations¶

The Splunk Add-on for Microsoft Cloud Services does not support multiple Inputs Data Managers (IDMs) or heavy forwarders reading from a single Event Hub.

The Azure EventHubs input for the Splunk Add-on for Microsoft Cloud Services is not compatible with the Event Hubs input in the Splunk Add-on for Microsoft Azure, when listening to the same Event Hub namespace. The Event Hubs input in the Splunk Add-on for Microsoft Azure needs to be disabled for this input to run.

Horizontal Scaling Across Multiple Splunk Environment¶

Version 5.0.0 and higher of the Splunk Add-on for Microsoft Cloud Services supports multiple Eventhub inputs configuration across multiple Splunk environments to collect data from the same Azure Eventhub using the Storage Blob checkpoint store mechanism. To use the horizontal scaling, while creating the Eventhub input, enter “Enable Blob Checkpoint Store”, “Azure Storage Account” and “Container Name”.

Prerequisites¶

Create Storage Container in Azure which will be used during the Eventhub input configuration to store checkpoint details.
Create a Storage Account in the Splunk Add-on for Microsoft Cloud Services. See Connect to your Azure Storage account with the Splunk Add-on for Microsoft Cloud Services

Risks¶

There is a small chance of data duplication, up to 5%.

Configure inputs using Splunk Web¶

Configure your inputs using Splunk Web on the Splunk platform instance responsible for collecting data for this add-on, usually a heavy forwarder.

In the Splunk Add-on for Microsoft Cloud Services, select Inputs.
Select Create New Input and then select Azure Event Hub.
Enter the Name, Azure App Account, Event Hub namespace, Event Hub name, Consumer group, Max Wait Time, Max Batch Size, Transport Type, Interval and Index , “Enable Blob Checkpoint Store” then enter “Container Name”, and “Storage Account” using the information in the following Input parameters table.

Configure inputs using configuration files¶

Configure your inputs using the configuration files on the Splunk platform instance responsible for collecting data for this add-on, usually a heavy forwarder.

In your Splunk platform deployment, navigate to $SPLUNK_HOME/etc/apps/Splunk_TA_microsoft-cloudservices/local.
Create a file named inputs.conf , if it does not already exist.

Add the following stanza for the Event Hub input:

[<input_stanza_name>]
account = <value>
blob_checkpoint_enabled = <value>
storage_account = <value>
container_name = <value>
consumer_group = <value>
event_hub_name = <value>
event_hub_namespace = <value>
container_name = <value>
index = <value>
interval = <value>
max_batch_size = <value>
max_wait_time = <value>
use_amqp_over_websocket = 1
sourcetype = mscs:azure:eventhub

Save and restart the Splunk platform.

Input parameters¶

Each attribute in the following table corresponds to a field in Splunk Web.

Attribute	Corresponding field in Splunk Web	Description
`input_stanza_name`	Name	A friendly name for your input. Name cannot contain any whitespace.
`account`	Azure Account	The Azure App account from which you want to collect data. Name cannot contain any whitespace.
`consumer_group`	Consumer Group	The Azure Event Hub Consumer Group.
`event_hub_name`	Event Hub Name	The Azure Event Hub Name.
`event_hub_namespace`	Event Hub Namespace (Fully Qualified Domain Name (FQDN))	The Azure Event Hub Namespace (FQDN). On portal.azure.com, on your Event Hubs Namespace page, the `event_hub_namespace` is displayed as Host Name in the Essentials section. It has the following formatting: `.servicebus.windows.net`.
`index`	Index	The index in which to store Azure Event Hub data.
`interval`	Interval	The number of seconds to wait before the Splunk platform runs the command again. The default is 3600 seconds.
`max_batch_size`	Max Batch Size	The maximum number of events to retrieve in one batch. The default is 300.
`max_wait_time`	Max Wait Time	The maximum interval in seconds that the event processor will wait before processing. The default is 300 seconds.
`use_amqp_over_websocket`	Transport Type	The switch that allows use of Advanced Message Queuing Protocol (AMQP) over WebSocket. The default is AMQP over WebSocket. The Event Hub input does not support AMQP as the transport type in Splunk Cloud Platform.
`sourcetype`	Sourcetype	Select the source type based on the configured Event Hub. Supported source types are `mscs:azure:eventhub`, `azure:monitor:aad`, `azure:monitor:resource` and `azure:monitor:activity`. The default sourcetype is `mscs:azure:eventhub`
`blob_checkpoint_enabled`	Blob Checkpoint Store	Enable storage blob as checkpoint for eventhub input. It is important to note that if you use this option, there will be no backward compatibility for the File Checkpoint. If this option is checked once, and then disabled in future; there will be data duplication.
`storage_account`	Azure Storage Account	The Azure Storage account in which Container is created to store eventhub checkpoint.
`container_name`	Container Name	Enter the container name under the storage account. You can only add one container name for each input.
`ensure_ascii`	Enforce ASCII encoding (JSON)	If set to `Strict ASCII` the json events (and only those) are going to be encoded in ASCII. Native encoding doesn’t change the events encoding.