Performance reference for the Azure Storage Blob input in the Splunk Add-on for Microsoft Cloud Services¶

The following tables show reference information about performance testing of the Azure Storage Blob input in the Splunk Add-on for Microsoft Cloud Services. Use this information to enhance the performance of your own Azure Storage Blob data collection tasks.

Note

Many factors impact performance results, including file size, file compression, event size, deployment architecture, and hardware. These results represent reference information and do not represent performance in all environments.

Version 5.2.1 and higher Blob input performance characteristics¶

Common architecture setup	Scenario	Blob type	File size	Ingest rate	IDM CPU
CO2 Stack - Victoria Search Head Cluster 1 cluster master (c5.xlarge) 3 search head (m6i.4xlarge) 16 CPU 64 GiB Memory 3 indexer cluster (im4gn.4xlarge) 16 CPU 64 GiB Memory Default TA configs
	1 input	AppendBlob	1MB	16324 kb/s	10.95%
	5 inputs	AppendBlob	1MB	39814 kb/s	27.69%
	1 input	BlockBlob	1MB	24364 kb/s	9.42%
	5 inputs	BlockBlob	1MB	40810 kb/s	16.34%
CO2 Stack - Classic Cluster (1 IDM) 1 cluster master (c5.xlarge) 3 search head (m6i.4xlarge) 16 CPU 64 GiB Memory 3 indexer cluster (im4gn.4xlarge) 16 CPU 64 GiB Memory 1 Inputs Data Manager (IDM) (m6i.4xlarge) 16 CPU 64 GiB Memory Default TA configs
	1 input	AppendBlob	1MB	8717 kb/s	8.44%
	5 inputs	AppendBlob	1MB	39815 kb/s	28.41%
	1 input	BlockBlob	1MB	11577 kb/s	6.4%
	5 inputs	BlockBlob	1MB	41857 kb/s	28%

Version 5.1.2 and higher Blob input performance characteristics¶

Common architecture setup	Scenario	Blob type	File size	Ingest rate	IDM CPU
CO2 Stack - Victoria Search Head Cluster 1 cluster master (c5.xlarge) 3 search head (m6i.4xlarge) 16 CPU 64 GiB Memory 3 indexer cluster (im4gn.4xlarge) 16 CPU 64 GiB Memory Default TA configs
	1 input	AppendBlob	1MB	8277 kb/s	25.34%
	5 inputs	AppendBlob	1MB	37963 kb/s	28.09%
	1 input	BlockBlob	1MB	11532 kb/s	18.45%
	5 inputs	BlockBlob	1MB	40810 kb/s	29.82%
CO2 Stack - Classic Cluster (1 IDM) 1 cluster master (c5.xlarge) 3 search head (m6i.4xlarge) 16 CPU 64 GiB Memory 3 indexer cluster (im4gn.4xlarge) 16 CPU 64 GiB Memory 1 Inputs Data Manager (IDM) (m6i.4xlarge) 16 CPU 64 GiB Memory Default TA configs
	1 input	AppendBlob	1MB	7405 kb/s	9.45%
	5 inputs	AppendBlob	1MB	39815 kb/s	28.64%
	1 input	BlockBlob	1MB	20030 kb/s	10.12%
	5 inputs	BlockBlob	1MB	40810 kb/s	29.09%

Version 5.0.0 and higher Blob input performance characteristics¶

Common architecture setup	Scenario	Blob type	File size	Ingest rate	IDM CPU
CO2 Stack - Victoria Search Head Cluster 1 cluster master (c5.xlarge) 3 search head (m6i.4xlarge) 16 CPU 64 GiB Memory 3 indexer cluster (is4gen.4xlarge) 16 CPU 96 GiB Memory Default TA configs
	1 input	AppendBlob	1MB	8155.26 kb/s	N/A
	5 inputs	AppendBlob	1MB	21390.76 kb/s	N/A
	1 input	BlockBlob	1MB	13100.21 kb/s	N/A
	5 inputs	BlockBlob	1MB	24328.95 kb/s	N/A
CO2 Stack - Classic Cluster (1 IDM) 1 cluster master (c5.xlarge) 3 search head (m6i.4xlarge) 16 CPU 64 GiB Memory 3 indexer cluster (is4gen.4xlarge) 16 CPU 96 GiB Memory 1 Inputs Data Manager (IDM) (m6i.4xlarge) 16 CPU 64 GiB Memory Default TA configs
	1 input	AppendBlob	1MB	6850.66 kb/s	55.85%
	5 inputs	AppendBlob	1MB	8049.20 kb/s	33.35%
	1 input	BlockBlob	1MB	3808.87 kb/s	44.55%
	5 inputs	BlockBlob	1MB	3838.31 kb/s	32.00%

Continuous append scenario
Append 6 files sequentially to 1GB.	A single user creates an append blob. Append 1MB each time in each file. Pause up to 1 second after each append. Append 1024 times to get to 1GB for the file. In total, append 6 files to 1GB in parallel. Total event size matches the original file size in Azure.

Version 4.1.4 Blob input performance characteristics¶

Block blob¶

Scaled up stack

20 containers
20 add-on inputs
1 cluster master (c5.xlarge)
1 search head (c5.2xlarge)
3 indexer cluster (i3.large)
- 32 CPU cores
- 244 GB memory
1 input data manager (c5.9xlarge)
- 32 CPU cores
- 72 GB memory
20 Splunk Add-on for Microsoft Cloud Services inputs
120 seconds collection interval

block blob

10kb

48GB/day

3522

~120 seconds

1.8%

1MB

50GB/day

36

~121 seconds

1.84%

Append blob¶

Scaled up stack

20 containers
20 add-on inputs
1 cluster master (c5.xlarge)
1 search head (c5.2xlarge)
3 indexer cluster (i3.large)
- 32 CPU cores
- 244 GB memory
1 input data manager (c5.9xlarge)
- 32 CPU cores
- 72 GB memory
20 Splunk Add-on for Microsoft Cloud Services inputs
120 seconds collection interval

append blob

1 kb

33GB/day

21522

~120 seconds

3.7%

10 kb

281GB/day

20496

~121 seconds

2.67%

100 kb

2.69TB/day

18402

~129 seconds

6.5%

Additional Blob input scenarios¶

Scenario	Workflow	Result
Append file up to 4GB	- A single user creates an append blob. - Append 100 kb each time. - Append 40,000 times. - Pause up to 100 ms in each append.	- All 40,000 append blocks pulled back. - Total event size matches original file size in Azure. - Minimal to no data lag. - Collection interval is 12 seconds.
Append 3 files sequentially to 100 MB	- A single user creates an append blob. - 100,000 append each time. - Pause up to 100 ms after each append. - Append 1,000 times to get to 100 MB for the file. - Move on to the next file and repeat. - In total, append 3 files to 100 MB sequentially.	- All 3,000 append blocks pulled back. - Total event size matches original file size in Azure.