Performance reference for the Splunk Add-on for Google Cloud Platform¶
This page provides reference information on performance testing of the pub/sub, pub/sub lite, and pub/sub based bucket inputs for the Splunk Add-on for Google Cloud Platform. Use this information to enhance the performance of your own Google Cloud Platform collection tasks.
Many factors impact performance results, including file size, file compression, event size, deployment architecture, and hardware. These results should be used as reference information and do not represent performance in all environments.
Version 4.5.0 Cloud Storage Bucket input performance statistics for JSON files¶
Common architecture setup |
Number of inputs |
Number of Threads |
Number of files |
File Size |
Events Count |
Data Collection Time (in minutes) |
Avg CPU (by input) |
Avg Memory (by input) |
|---|---|---|---|---|---|---|---|---|
CO2 Stack - Victoria Search Head Cluster
|
1 input |
1 |
10k |
~3MB |
6180001 |
53 |
3% |
0.1% |
1 input |
10 |
10k |
~3MB |
6180001 |
12 |
15% |
0.1% |
|
CO2 Stack - Classic Cluster (1 IDM)
|
1 input |
1 |
10k |
~3MB |
6180001 |
48 |
4% |
0.1% |
1 input |
10 |
10k |
~3MB |
6180001 |
15 |
13% |
0.1% |
|
On Prem
|
1 input |
1 |
10k |
~3MB |
6180001 |
108 |
2% |
0.1% |
1 input |
10 |
10k |
~3MB |
6180001 |
14 |
12% |
0.1% |
Version 4.5.0 Cloud Storage Bucket input performance statistics for CSV files¶
Common architecture setup |
Number of inputs |
Number of Threads |
Number of files |
File Size |
Events Count |
Data Collection Time (in minutes) |
Avg CPU (by input) |
Avg Memory (by input) |
|---|---|---|---|---|---|---|---|---|
CO2 Stack - Victoria Search Head Cluster
|
1 input |
1 |
100k |
~50KB |
86159944 |
225 |
6% |
0.1% |
1 input |
10 |
100k |
~50KB |
86159944 |
169 |
15% |
0.1% |
|
CO2 Stack - Classic Cluster (1 IDM)
|
1 input |
1 |
100k |
~50KB |
86159944 |
223 |
5% |
0.1% |
1 input |
10 |
100k |
~50KB |
86159944 |
167 |
15% |
0.1% |
|
On Prem
|
1 input |
1 |
100k |
~50KB |
86159944 |
445 |
3% |
0.1% |
1 input |
10 |
100k |
~50KB |
86159944 |
155 |
15% |
0.1% |
Version 4.2.0 Pub/Sub Lite input performance statistics¶
There following tests a configuration of 20 partitions for Version 4.2.0 Pub/Sub Lite input performance.
Common architecture setup |
Scenario |
Event type |
Event Size |
Number of threads |
Ingest Rate (KB/min) |
CPU |
RAM |
|---|---|---|---|---|---|---|---|
CO2 Stack - Victoria Search Head Cluster
|
1 input |
JSON |
1 KB |
10 |
212274 |
13.58% |
11.8% |
1 input |
JSON |
1 KB |
64 |
2202370 |
14.25% |
11.7% |
|
20 input |
JSON |
1 KB |
10 |
2020845 |
22.88% |
9.74% |
|
20 input |
JSON |
1 KB |
64 |
2067952 |
23.25% |
8.79% |
|
1 input |
JSON |
10KB |
10 |
3166630 |
14.32% |
6.77% |
|
20 input |
JSON |
10KB |
10 |
6197910 |
22.05% |
11.07% |
|
1 input |
Non-JSON |
1KB |
10 |
414976 |
13.69% |
8.05% |
|
1 input |
Non-JSON |
1KB |
64 |
379032 |
13.78% |
3.27% |
|
20 input |
Non-JSON |
1KB |
10 |
2069770 |
23.9% |
8.63% |
|
20 input |
Non-JSON |
1KB |
64 |
2064460 |
23.2% |
7.06% |
|
1 input |
Non-JSON |
10KB |
10 |
3795480 |
14.48% |
8.92% |
|
20 input |
Non-JSON |
10KB |
10 |
6129498 |
23.7% |
5.57% |
|
CO2 Stack - Classic Cluster (1 IDM)
|
|||||||
1 input |
JSON |
1KB |
10 |
231987 |
13.61% |
6.6% |
|
1 input |
JSON |
1KB |
64 |
237228 |
13.65% |
6.65% |
|
20 input |
JSON |
1KB |
10 |
1022357 |
22.4% |
13.1% |
|
20 input |
JSON |
1KB |
64 |
1019737 |
22.7% |
12.98% |
|
1 input |
JSON |
10KB |
10 |
1887750 |
14.21% |
12.9% |
|
20 input |
JSON |
10KB |
10 |
3228910 |
23.38% |
13.1% |
|
1 input |
Non-JSON |
1KB |
10 |
247171 |
13.6% |
12.96% |
|
1 input |
Non-JSON |
1KB |
64 |
241652 |
9.17% |
13.55% |
|
20 input |
Non-JSON |
1KB |
10 |
1021291 |
22.5% |
3.9% |
|
20 input |
Non-JSON |
1KB |
64 |
987298 |
23% |
4.02% |
|
1 input |
Non-JSON |
10KB |
10 |
2171011 |
14.35% |
4% |
|
20 input |
Non-JSON |
10KB |
10 |
291004 |
22% |
4.05% |
Version 4.2.0 Pub/Sub Based Bucket input performance statistics¶
The following tests file sizes of 1, 5, and 10 KB for Version 4.2.0 Pub/Sub Based Bucket input. The number of threads tested is 10.
Common architecture setup |
Scenario |
File type |
Message batch |
Ingest rate (KB/min) |
CPU |
RAM |
|---|---|---|---|---|---|---|
CO2 Stack - Victoria Search Head Cluster
|
||||||
1 input |
CSV |
10 |
1649 |
3.75% |
11.55% |
|
5 inputs |
CSV |
10 |
7869 |
3.7% |
11.54% |
|
10 inputs |
CSV |
10 |
16670 |
4.65% |
11.53% |
|
20 inputs |
CSV |
10 |
33327 |
3.75% |
11.49% |
|
40 inputs |
CSV |
10 |
64360 |
4.3% |
11.48% |
|
1 input |
CSV |
100 |
9783 |
3.9% |
11.56% |
|
5 inputs |
CSV |
100 |
49756 |
3.65% |
11.54% |
|
10 inputs |
CSV |
100 |
100888 |
3.85% |
11.54% |
|
20 inputs |
CSV |
100 |
192448 |
4.49% |
11.52% |
|
40 inputs |
CSV |
100 |
385078 |
7.46% |
11.54% |
|
1 input |
JSON |
10 |
1655 |
4.25% |
11.59% |
|
5 inputs |
JSON |
10 |
7853 |
4.5% |
11.58% |
|
10 inputs |
JSON |
10 |
16882 |
3.65% |
11.59% |
|
20 inputs |
JSON |
10 |
34611 |
4.65% |
11.57% |
|
40 inputs |
JSON |
10 |
65909 |
3.9% |
11.57% |
|
1 input |
JSON |
100 |
9813 |
3.85% |
11.57% |
|
5 inputs |
JSON |
100 |
48865 |
3.55% |
11.55% |
|
10 inputs |
JSON |
100 |
99252 |
4.4% |
11.52% |
|
20 inputs |
JSON |
100 |
201040 |
4.24% |
11.46% |
|
40 inputs |
JSON |
100 |
397136 |
7.89% |
11.39% |
|
CO2 Stack - Classic Cluster (1 IDM)
|
1 input |
CSV |
10 |
1596 |
4.5% |
9.87% |
5 inputs |
CSV |
10 |
7670 |
4.25% |
9.84% |
|
10 inputs |
CSV |
10 |
15129 |
4.75% |
9.48% |
|
20 inputs |
CSV |
10 |
30278 |
4.45% |
9.06% |
|
40 inputs |
CSV |
10 |
60220 |
6.5% |
8.68% |
|
1 input |
CSV |
100 |
8708 |
5.3% |
9.89% |
|
5 inputs |
CSV |
100 |
45657 |
4.45% |
9.46% |
|
10 inputs |
CSV |
100 |
92601 |
4.91% |
9.15% |
|
20 inputs |
CSV |
100 |
185263 |
6.57% |
8.86% |
|
40 inputs |
CSV |
100 |
365956 |
9.09% |
8.45% |
|
1 input |
JSON |
10 |
1591 |
5.7% |
6.98% |
|
5 inputs |
JSON |
10 |
7550 |
6.95% |
4.4% |
|
10 inputs |
JSON |
10 |
15376 |
4.75% |
6.88% |
|
20 inputs |
JSON |
10 |
30534 |
5.5% |
6.7% |
|
40 inputs |
JSON |
10 |
60647 |
8.05% |
6.23% |
|
1 input |
JSON |
100 |
11024 |
4.3% |
8.43% |
|
5 inputs |
JSON |
100 |
48298 |
4% |
7.97% |
|
10 inputs |
JSON |
100 |
93452 |
4.81% |
7.53% |
|
20 inputs |
JSON |
100 |
182911 |
6.92% |
7.44% |
|
40 inputs |
JSON |
100 |
372861 |
6.78% |
7.54% |
Version 1.3.0 Pub/Sub input performance statistics¶
Use the following information on version 1.3.0 Pub/Sub input performance statistics to enhance the performance of your own Google Cloud Platform collection tasks.
Testing architecture¶
The throughput data and conclusions provided here are based on performance testing using single-instance Splunk Enterprise 7.0.1 running on the following environment.
| Instance type | n1-standard-8 |
| Memory | 30 GB |
| vCPU | 8 |
| Cores | 4 CPU cores |
| Storage Type | standard persistent disk (2000GB) |
Measured performance data¶
The throughput data provided here is the average performance for different subscription numbers achieved in performance testing under specific operating conditions and is subject to change when any of the hardware and software variables changes. Use this data for a rough reference only.
| Subscriptions | Input numbers | Throughput (KB/s) | Throughput (GB/day) |
|---|---|---|---|
| 1 | 1 | 1800 | 150 |
| 4 | 4 | 9000 | 740 |
| 8 | 8 | 17000 | 1400 |
| 16 | 16 | 24000 | 1970 |
| 200 | 16 | 16320 | 1340 |
For heavy data ingestion scenarios, change the default Acknowledgment Deadline of your subscription from 10 seconds to 60 seconds for optimal performance and to avoid data duplication.