Skip to content

Configure Azure Event Hub inputs for the Splunk Add-on for Microsoft Cloud Services

Configure your inputs on the Splunk platform instance responsible for collecting data for this add-on, usually a heavy forwarder. You can configure inputs using Splunk Web, which is a best practice, or by using the configuration files.

Azure Service Bus SDK retirement

Microsoft is retiring legacy Azure Service Bus SDK libraries for .NET and Java. This retirement does not affect Event Hubs inputs because these inputs use the Azure Event Hubs Python SDK. No action is required.

The Python SDK communicates using AMQP or AMQP over WebSocket. It does not use the retiring .NET or Java Service Bus SDKs or the SBMP protocol.

Add-on prerequisites

Azure Event Hub prerequisites

Perform the following prerequisites before configuring an Azure Event Hub input:

  • Configure separate Azure Event Hubs for Microsoft Entra ID, Activity, and resource logs so that each input can use the appropriate sourcetype. Categories that use the same sourcetype can share an Event Hub. See Select a sourcetype.
  • Authorize access to Azure Event Hubs by giving Azure Event Hubs Data Receiver permissions to each applicable Azure application. See the Authorize access to Azure Event Hubs topic in the Microsoft Azure documentation for more information..
  • Splunk Cloud customers who are installing this add-on on the Inputs Data Manager (IDM) and want to collect Event Hub data, must use the Admin Configuration Service (ACS) to configure outbound ports 5671/tcp and 5672/tcp (Advanced Message Queuing Protocol (AMQP) specification) to connect to their target Azure address. By default IDM’s can only go out on port 443.

Select a sourcetype

Select the sourcetype according to the source and categories of the logs sent to the Event Hub. The input uses one configured sourcetype; it does not automatically select an Azure Monitor sourcetype from each event’s category.

Logs sent to the Event Hub Example event categories Sourcetype
Microsoft Entra ID sign-in, audit, provisioning, and risk logs SignInLogs, NonInteractiveUserSignInLogs, ServicePrincipalSignInLogs, ManagedIdentitySignInLogs, AuditLogs, ProvisioningLogs, UserRiskEvents, RiskyUsers azure:monitor:aad
Azure Activity logs exported through the subscription’s diagnostic settings Administrative, such as resource creation or deletion azure:monitor:activity
Azure resource logs exported through the resource’s diagnostic settings Cosmos DB categories such as DataPlaneRequests and ControlPlaneRequests; Azure Data Share categories such as RHC, Shares, SentShareSnapshots, and ReceivedShareSnapshots azure:monitor:resource
Generic Event Hub data that does not match the Azure Monitor log schemas above Custom event categories or payloads mscs:azure:eventhub

For Microsoft Entra sign-in and audit logs, select azure:monitor:aad to use the existing field extractions, event types, and Authentication, Change, and Alerts mappings for applicable events. The default, mscs:azure:eventhub, is a generic sourcetype. Events that retain this sourcetype have JSON field extraction but do not receive the data-specific CIM mappings available through the Azure Monitor sourcetypes.

The available mappings depend on the event category, operation, and schema. See Source types for the existing CIM data models and security alert and recommendation sourcetypes.

For security detection examples, see Splunk Security Content.

Match Azure diagnostic settings to the input

  1. Identify the source of the logs. For Entra logs, use Entra ID > Monitoring & health > Diagnostic settings in the Microsoft Entra admin center. For Azure Activity logs, configure the subscription’s Activity log diagnostic settings. For resource logs, configure diagnostic settings on the Azure resource. See Configure Microsoft Entra diagnostic settings, Export the Azure Activity log, and Diagnostic settings in Azure Monitor.
  2. Select the log categories to collect and Stream to an event hub. Select the destination namespace and Event Hub, and save the diagnostic setting. For example, route Entra SignInLogs and AuditLogs to an Event Hub that you will collect with azure:monitor:aad.
  3. Route log families that require different sourcetypes to separate Event Hubs. A consumer group receives the Event Hub’s stream; creating separate consumer groups on the same Event Hub does not filter events by category. See Consumer groups.
  4. Configure the add-on input with the destination Event Hub namespace, Event Hub name, and the matching Sourcetype. Confirm that the namespace and Event Hub match the diagnostic setting.

Scaling

On your Azure deployment, you should configure a ratio of at least one Event Hub throughput unit for each partition. For example, if you have 20 throughput units, configure 20 partitions. For more information on Event Hub throughput scalability, see the https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-scalability#throughput-units in the Microsoft Azure documentation.

On the Splunk software side, the number of Event Hub inputs that you create as consumers on an Event Hub must be less than or equal to the number of partitions that you have on the Event Hub. For more information, see the https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features Event Hubs in the Microsoft Azure documentation.

Limitations

The Splunk Add-on for Microsoft Cloud Services does not support multiple Inputs Data Managers (IDMs) or heavy forwarders reading from a single Event Hub.

Note

The Azure EventHubs input for the Splunk Add-on for Microsoft Cloud Services is not compatible with the Event Hubs input in the Splunk Add-on for Microsoft Azure, when listening to the same Event Hub namespace. The Event Hubs input in the Splunk Add-on for Microsoft Azure needs to be disabled for this input to run.

Horizontal Scaling Across Multiple Splunk Environment

Version 5.0.0 and higher of the Splunk Add-on for Microsoft Cloud Services supports multiple Eventhub inputs configuration across multiple Splunk environments to collect data from the same Azure Eventhub using the Storage Blob checkpoint store mechanism. To use the horizontal scaling, while creating the Eventhub input, enter “Enable Blob Checkpoint Store”, “Azure Storage Account” and “Container Name”.

Prerequisites

Risks

  • There is a small chance of data duplication, up to 5%.

Configure inputs using Splunk Web

Configure your inputs using Splunk Web on the Splunk platform instance responsible for collecting data for this add-on, usually a heavy forwarder.

  1. In the Splunk Add-on for Microsoft Cloud Services, select Inputs.
  2. Select Create New Input and then select Azure Event Hub.
  3. Enter the following:
  • Name
  • Azure App Account
  • Event Hub namespace
  • Event Hub name
  • Consumer group
  • Max Wait Time
  • Max Batch Size
  • Transport Type
  • Interval
  • Index
  • Sourcetype
  • Enable Blob Checkpoint Store
  • Container Name
  • Storage Account

Use the information from the following Input parameters table.

  1. Save the input with the Sourcetype selected in Select a sourcetype.

Configure inputs using configuration files

Configure your inputs using the configuration files on the Splunk platform instance responsible for collecting data for this add-on, usually a heavy forwarder.

  1. In your Splunk platform deployment, navigate to $SPLUNK_HOME/etc/apps/Splunk_TA_microsoft-cloudservices/local.
  2. Create a file named inputs.conf , if it does not already exist.
  3. Add the following stanza for the Event Hub input:
[mscs_azure_event_hub://<input_stanza_name>]
account = <value>
blob_checkpoint_enabled = <value>
storage_account = <value>
container_name = <value>
consumer_group = <value>
event_hub_name = <value>
event_hub_namespace = <value>
index = <value>
interval = <value>
max_batch_size = <value>
max_wait_time = <value>
use_amqp_over_websocket = 1
sourcetype = <selected_sourcetype>

Replace <selected_sourcetype> with the sourcetype for your logs, such as azure:monitor:aad for Microsoft Entra sign-in and audit logs. See Select a sourcetype.

  1. Save and restart the Splunk platform.

Input parameters

Each attribute in the following table corresponds to a field in Splunk Web.

Attribute Corresponding field in Splunk Web Description
input_stanza_name Name A friendly name for your input. Name cannot contain any whitespace.
account Azure Account The Azure App account from which you want to collect data. Name cannot contain any whitespace.
consumer_group Consumer Group The Azure Event Hub Consumer Group.
event_hub_name Event Hub Name The Azure Event Hub Name.
event_hub_namespace Event Hub Namespace (Fully Qualified Domain Name (FQDN)) The Azure Event Hub Namespace (FQDN). On portal.azure.com, on your Event Hubs Namespace page, the event_hub_namespace is displayed as Host Name in the Essentials section. It has the following formatting: .servicebus.windows.net.
index Index The index in which to store Azure Event Hub data.
interval Interval The number of seconds to wait before the Splunk platform runs the command again. The default is 3600 seconds.
max_batch_size Max Batch Size The maximum number of events to retrieve in one batch. The default is 300.
max_wait_time Max Wait Time The maximum interval in seconds that the event processor waits before processing. The default is 300 seconds.
use_amqp_over_websocket Transport Type The switch that allows use of Advanced Message Queuing Protocol (AMQP) over WebSocket. The default is AMQP over WebSocket.
The Event Hub input does not support AMQP as the transport type in Splunk Cloud Platform.
sourcetype Sourcetype Select the source type that matches the logs routed to the Event Hub. The default is mscs:azure:eventhub. See Select a sourcetype.
blob_checkpoint_enabled Blob Checkpoint Store Enable storage blob as checkpoint for eventhub input.
It is important to note that if you use this option, there is no backward compatibility for the File Checkpoint. If this option is checked once, and then disabled in future; there is data duplication.
storage_account Azure Storage Account The Azure Storage account in which Container is created to store eventhub checkpoint.
container_name Container Name Enter the container name under the storage account. You can only add one container name for each input.
ensure_ascii Enforce ASCII encoding (JSON) If set to Strict ASCII the json events (and only those) are going to be encoded in ASCII. Native encoding doesn’t change the events encoding.