Performance reference for the Azure Storage Blob input in the Splunk Add-on for Microsoft Cloud Services¶
The following tables show reference information about performance testing of the Azure Storage Blob input in the Splunk Add-on for Microsoft Cloud Services. Use this information to enhance the performance of your own Azure Storage Blob data collection tasks.
Note
Many factors impact performance results, including file size, file compression, event size, deployment architecture, and hardware. These results represent reference information and do not represent performance in all environments.
Version 5.2.1 and higher Blob input performance characteristics¶
| Common architecture setup | Scenario | Blob type | File size | Ingest rate | IDM CPU |
|---|---|---|---|---|---|
| CO2 Stack - Victoria Search Head Cluster - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (im4gn.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
1 input | AppendBlob | 1MB | 16324 kb/s | 10.95% |
| CO2 Stack - Victoria Search Head Cluster - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (im4gn.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
5 inputs | AppendBlob | 1MB | 39814 kb/s | 27.69% |
| CO2 Stack - Victoria Search Head Cluster - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (im4gn.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
1 input | BlockBlob | 1MB | 24364 kb/s | 9.42% |
| CO2 Stack - Victoria Search Head Cluster - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (im4gn.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
5 inputs | BlockBlob | 1MB | 40810 kb/s | 16.34% |
| CO2 Stack - Classic Cluster (1 IDM) - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (im4gn.4xlarge) - 16 CPU - 64 GiB Memory - 1 Inputs Data Manager (IDM) (m6i.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
1 input | AppendBlob | 1MB | 8717 kb/s | 8.44% |
| CO2 Stack - Classic Cluster (1 IDM) - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (im4gn.4xlarge) - 16 CPU - 64 GiB Memory - 1 Inputs Data Manager (IDM) (m6i.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
5 inputs | AppendBlob | 1MB | 39815 kb/s | 28.41% |
| CO2 Stack - Classic Cluster (1 IDM) - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (im4gn.4xlarge) - 16 CPU - 64 GiB Memory - 1 Inputs Data Manager (IDM) (m6i.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
1 input | BlockBlob | 1MB | 11577 kb/s | 6.4% |
| CO2 Stack - Classic Cluster (1 IDM) - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (im4gn.4xlarge) - 16 CPU - 64 GiB Memory - 1 Inputs Data Manager (IDM) (m6i.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
5 inputs | BlockBlob | 1MB | 41857 kb/s | 28% |
Version 5.1.2 and higher Blob input performance characteristics¶
| Common architecture setup | Scenario | Blob type | File size | Ingest rate | IDM CPU |
|---|---|---|---|---|---|
| CO2 Stack - Victoria Search Head Cluster - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (im4gn.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
1 input | AppendBlob | 1MB | 8277 kb/s | 25.34% |
| CO2 Stack - Victoria Search Head Cluster - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (im4gn.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
5 inputs | AppendBlob | 1MB | 37963 kb/s | 28.09% |
| CO2 Stack - Victoria Search Head Cluster - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (im4gn.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
1 input | BlockBlob | 1MB | 11532 kb/s | 18.45% |
| CO2 Stack - Victoria Search Head Cluster - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (im4gn.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
5 inputs | BlockBlob | 1MB | 40810 kb/s | 29.82% |
| CO2 Stack - Classic Cluster (1 IDM) - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (im4gn.4xlarge) - 16 CPU - 64 GiB Memory - 1 Inputs Data Manager (IDM) (m6i.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
1 input | AppendBlob | 1MB | 7405 kb/s | 9.45% |
| CO2 Stack - Classic Cluster (1 IDM) - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (im4gn.4xlarge) - 16 CPU - 64 GiB Memory - 1 Inputs Data Manager (IDM) (m6i.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
5 inputs | AppendBlob | 1MB | 39815 kb/s | 28.64% |
| CO2 Stack - Classic Cluster (1 IDM) - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (im4gn.4xlarge) - 16 CPU - 64 GiB Memory - 1 Inputs Data Manager (IDM) (m6i.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
1 input | BlockBlob | 1MB | 20030 kb/s | 10.12% |
| CO2 Stack - Classic Cluster (1 IDM) - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (im4gn.4xlarge) - 16 CPU - 64 GiB Memory - 1 Inputs Data Manager (IDM) (m6i.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
5 inputs | BlockBlob | 1MB | 40810 kb/s | 29.09% |
Version 5.0.0 and higher Blob input performance characteristics¶
| Common architecture setup | Scenario | Blob type | File size | Ingest rate | IDM CPU |
|---|---|---|---|---|---|
| CO2 Stack - Victoria Search Head Cluster - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (is4gen.4xlarge) - 16 CPU - 96 GiB Memory - Default TA configs |
1 input | AppendBlob | 1MB | 8155.26 kb/s | N/A |
| CO2 Stack - Victoria Search Head Cluster - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (is4gen.4xlarge) - 16 CPU - 96 GiB Memory - Default TA configs |
5 inputs | AppendBlob | 1MB | 21390.76 kb/s | N/A |
| CO2 Stack - Victoria Search Head Cluster - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (is4gen.4xlarge) - 16 CPU - 96 GiB Memory - Default TA configs |
1 input | BlockBlob | 1MB | 13100.21 kb/s | N/A |
| CO2 Stack - Victoria Search Head Cluster - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (is4gen.4xlarge) - 16 CPU - 96 GiB Memory - Default TA configs |
5 inputs | BlockBlob | 1MB | 24328.95 kb/s | N/A |
| CO2 Stack - Classic Cluster (1 IDM) - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (is4gen.4xlarge) - 16 CPU - 96 GiB Memory - 1 Inputs Data Manager (IDM) (m6i.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
1 input | AppendBlob | 1MB | 6850.66 kb/s | 55.85% |
| CO2 Stack - Classic Cluster (1 IDM) - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (is4gen.4xlarge) - 16 CPU - 96 GiB Memory - 1 Inputs Data Manager (IDM) (m6i.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
5 inputs | AppendBlob | 1MB | 8049.20 kb/s | 33.35% |
| CO2 Stack - Classic Cluster (1 IDM) - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (is4gen.4xlarge) - 16 CPU - 96 GiB Memory - 1 Inputs Data Manager (IDM) (m6i.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
1 input | BlockBlob | 1MB | 3808.87 kb/s | 44.55% |
| CO2 Stack - Classic Cluster (1 IDM) - 1 cluster master (c5.xlarge) - 3 search head (m6i.4xlarge) - 16 CPU - 64 GiB Memory - 3 indexer cluster (is4gen.4xlarge) - 16 CPU - 96 GiB Memory - 1 Inputs Data Manager (IDM) (m6i.4xlarge) - 16 CPU - 64 GiB Memory - Default TA configs |
5 inputs | BlockBlob | 1MB | 3838.31 kb/s | 32.00% |
| Continuous append scenario | |
|---|---|
| Append 6 files sequentially to 1GB. |
|
Version 4.1.4 Blob input performance characteristics¶
Block blob¶
| Common architecture setup | Blob type | File size | Ingest rate | Events per minute | Data lag | IDM CPU Cores % (Percentage of total IDM cores) |
|---|---|---|---|---|---|---|
| Scaled up stack - 20 containers - 20 add-on inputs - 1 cluster master (c5.xlarge) - 1 search head (c5.2xlarge) - 3 indexer cluster (i3.large) - 32 CPU cores - 244 GB memory - 1 input data manager (c5.9xlarge) - 32 CPU cores - 72 GB memory - 20 Splunk Add-on for Microsoft Cloud Services inputs - 120 seconds collection interval |
block blob | 10kb | 48GB/day | 3522 | ~120 seconds | 1.8% |
| Scaled up stack - 20 containers - 20 add-on inputs - 1 cluster master (c5.xlarge) - 1 search head (c5.2xlarge) - 3 indexer cluster (i3.large) - 32 CPU cores - 244 GB memory - 1 input data manager (c5.9xlarge) - 32 CPU cores - 72 GB memory - 20 Splunk Add-on for Microsoft Cloud Services inputs - 120 seconds collection interval |
block blob | 1MB | 50GB/day | 36 | ~121 seconds | 1.84% |
Append blob¶
| Common architecture setup | Blob type | File size | Ingest rate | Events per minute | Data lag | IDM CPU Cores % (Percentage of total IDM cores) |
|---|---|---|---|---|---|---|
| Scaled up stack - 20 containers - 20 add-on inputs - 1 cluster master (c5.xlarge) - 1 search head (c5.2xlarge) - 3 indexer cluster (i3.large) - 32 CPU cores - 244 GB memory - 1 input data manager (c5.9xlarge) - 32 CPU cores - 72 GB memory - 20 Splunk Add-on for Microsoft Cloud Services inputs - 120 seconds collection interval |
append blob | 1 kb | 33GB/day | 21522 | ~120 seconds | 3.7% |
| Scaled up stack - 20 containers - 20 add-on inputs - 1 cluster master (c5.xlarge) - 1 search head (c5.2xlarge) - 3 indexer cluster (i3.large) - 32 CPU cores - 244 GB memory - 1 input data manager (c5.9xlarge) - 32 CPU cores - 72 GB memory - 20 Splunk Add-on for Microsoft Cloud Services inputs - 120 seconds collection interval |
append blob | 10 kb | 281GB/day | 20496 | ~121 seconds | 2.67% |
| Scaled up stack - 20 containers - 20 add-on inputs - 1 cluster master (c5.xlarge) - 1 search head (c5.2xlarge) - 3 indexer cluster (i3.large) - 32 CPU cores - 244 GB memory - 1 input data manager (c5.9xlarge) - 32 CPU cores - 72 GB memory - 20 Splunk Add-on for Microsoft Cloud Services inputs - 120 seconds collection interval |
append blob | 100 kb | 2.69TB/day | 18402 | ~129 seconds | 6.5% |
Additional Blob input scenarios¶
| Scenario | Workflow | Result |
|---|---|---|
| Append file up to 4GB | - A single user creates an append blob. - Append 100 kb each time. - Append 40,000 times. - Pause up to 100 ms in each append. |
- All 40,000 append blocks pulled back. - Total event size matches original file size in Azure. - Minimal to no data lag. - Collection interval is 12 seconds. |
| Append 3 files sequentially to 100 MB | - A single user creates an append blob. - 100,000 append each time. - Pause up to 100 ms after each append. - Append 1,000 times to get to 100 MB for the file. - Move on to the next file and repeat. - In total, append 3 files to 100 MB sequentially. |
- All 3,000 append blocks pulled back. - Total event size matches original file size in Azure. |