Skip to content

Performance reference for the Splunk Add-on for Google Cloud Platform

This page provides reference information on performance testing of the pub/sub, pub/sub lite, and pub/sub based bucket inputs for the Splunk Add-on for Google Cloud Platform. Use this information to enhance the performance of your own Google Cloud Platform collection tasks.

Many factors impact performance results, including file size, file compression, event size, deployment architecture, and hardware. These results should be used as reference information and do not represent performance in all environments.

Version 4.5.0 Cloud Storage Bucket input performance statistics for JSON files

Common architecture setup

Number of inputs

Number of Threads

Number of files

File Size

Events Count

Data Collection Time (in minutes)

Avg CPU (by input)

Avg Memory (by input)

CO2 Stack - Victoria Search Head Cluster

  • 3 search heads (m5.4xlarge)
    • 16 CPU
    • 64 GiB memory
  • 3 indexers (im4gn.large)
  • 1 cluster master (c5.xlarge)

1 input

1

10k

~3MB

6180001

53

3%

0.1%

1 input

10

10k

~3MB

6180001

12

15%

0.1%

CO2 Stack - Classic Cluster (1 IDM)

  • 1 cluster master (c5.xlarge)
  • 3 search heads (c5.2xlarge)
  • 3 indexers (i3.large)
  • 1 Input Data Manager (IDM) (m5.4xlarge)
    • 16 CPU
    • 64 GiB memory

1 input

1

10k

~3MB

6180001

48

4%

0.1%

1 input

10

10k

~3MB

6180001

15

13%

0.1%

On Prem

  • Heavy Forwarder (m5.4xlarge)
    • 16 CPU
    • 64 GiB memory

1 input

1

10k

~3MB

6180001

108

2%

0.1%

1 input

10

10k

~3MB

6180001

14

12%

0.1%

Version 4.5.0 Cloud Storage Bucket input performance statistics for CSV files

Common architecture setup

Number of inputs

Number of Threads

Number of files

File Size

Events Count

Data Collection Time (in minutes)

Avg CPU (by input)

Avg Memory (by input)

CO2 Stack - Victoria Search Head Cluster

  • 3 search heads (m5.4xlarge)
    • 16 CPU
    • 64 GiB memory
  • 3 indexers (im4gn.large)
  • 1 cluster master (c5.xlarge)

1 input

1

100k

~50KB

86159944

225

6%

0.1%

1 input

10

100k

~50KB

86159944

169

15%

0.1%

CO2 Stack - Classic Cluster (1 IDM)

  • 1 cluster master (c5.xlarge)
  • 3 search heads (c5.2xlarge)
  • 3 indexers (i3.large)
  • 1 Input Data Manager (IDM) (m5.4xlarge)
    • 16 CPU
    • 64 GiB memory

1 input

1

100k

~50KB

86159944

223

5%

0.1%

1 input

10

100k

~50KB

86159944

167

15%

0.1%

On Prem

  • Heavy Forwarder (m5.4xlarge)
    • 16 CPU
    • 64 GiB memory

1 input

1

100k

~50KB

86159944

445

3%

0.1%

1 input

10

100k

~50KB

86159944

155

15%

0.1%

Version 4.2.0 Pub/Sub Lite input performance statistics

There following tests a configuration of 20 partitions for Version 4.2.0 Pub/Sub Lite input performance.

Common architecture setup

Scenario

Event type

Event Size

Number of threads

Ingest Rate (KB/min)

CPU

RAM

CO2 Stack - Victoria Search Head Cluster

  • 1 cluster master (c5.xlarge)
  • 3 search heads (c6i.4xlarge)
    • 16 CPU
    • 16 GiB Memory
  • 3 indexer clusters (is4gen.4xlarge)
    • 16 CPU core
    • 96 GiB Memory
  • Default TA configs

1 input

JSON

1 KB

10

212274

13.58%

11.8%

1 input

JSON

1 KB

64

2202370

14.25%

11.7%

20 input

JSON

1 KB

10

2020845

22.88%

9.74%

20 input

JSON

1 KB

64

2067952

23.25%

8.79%

1 input

JSON

10KB

10

3166630

14.32%

6.77%

20 input

JSON

10KB

10

6197910

22.05%

11.07%

1 input

Non-JSON

1KB

10

414976

13.69%

8.05%

1 input

Non-JSON

1KB

64

379032

13.78%

3.27%

20 input

Non-JSON

1KB

10

2069770

23.9%

8.63%

20 input

Non-JSON

1KB

64

2064460

23.2%

7.06%

1 input

Non-JSON

10KB

10

3795480

14.48%

8.92%

20 input

Non-JSON

10KB

10

6129498

23.7%

5.57%

CO2 Stack - Classic Cluster (1 IDM)

  • 1 cluster master (c5.xlarge)
  • 3 search heads (c6i.xlarge)
    • 4 CPU core
    • 8 GB RAM
  • 3 indexers (is4gen.4xlarge)
    • 16 CPU
    • 96 GiB Memory
  • 1 Input Data Manager (IDM) (c6i.4xlarge)
    • 16 CPU core
    • 32 GB RAM
  • Default TA configs

1 input

JSON

1KB

10

231987

13.61%

6.6%

1 input

JSON

1KB

64

237228

13.65%

6.65%

20 input

JSON

1KB

10

1022357

22.4%

13.1%

20 input

JSON

1KB

64

1019737

22.7%

12.98%

1 input

JSON

10KB

10

1887750

14.21%

12.9%

20 input

JSON

10KB

10

3228910

23.38%

13.1%

1 input

Non-JSON

1KB

10

247171

13.6%

12.96%

1 input

Non-JSON

1KB

64

241652

9.17%

13.55%

20 input

Non-JSON

1KB

10

1021291

22.5%

3.9%

20 input

Non-JSON

1KB

64

987298

23%

4.02%

1 input

Non-JSON

10KB

10

2171011

14.35%

4%

20 input

Non-JSON

10KB

10

291004

22%

4.05%

Version 4.2.0 Pub/Sub Based Bucket input performance statistics

The following tests file sizes of 1, 5, and 10 KB for Version 4.2.0 Pub/Sub Based Bucket input. The number of threads tested is 10.

Common architecture setup

Scenario

File type

Message batch

Ingest rate (KB/min)

CPU

RAM

CO2 Stack - Victoria Search Head Cluster

  • 1 cluster master (c5.xlarge)
  • 3 search head (c6i.4xlarge)
    • 16 CPU
    • 32 GiB Memory
  • 3 indexer cluster (is4gen.4xlarge)
  • 16 CPU
  • 96 GiB Memory
  • Default TA configs

1 input

CSV

10

1649

3.75%

11.55%

5 inputs

CSV

10

7869

3.7%

11.54%

10 inputs

CSV

10

16670

4.65%

11.53%

20 inputs

CSV

10

33327

3.75%

11.49%

40 inputs

CSV

10

64360

4.3%

11.48%

1 input

CSV

100

9783

3.9%

11.56%

5 inputs

CSV

100

49756

3.65%

11.54%

10 inputs

CSV

100

100888

3.85%

11.54%

20 inputs

CSV

100

192448

4.49%

11.52%

40 inputs

CSV

100

385078

7.46%

11.54%

1 input

JSON

10

1655

4.25%

11.59%

5 inputs

JSON

10

7853

4.5%

11.58%

10 inputs

JSON

10

16882

3.65%

11.59%

20 inputs

JSON

10

34611

4.65%

11.57%

40 inputs

JSON

10

65909

3.9%

11.57%

1 input

JSON

100

9813

3.85%

11.57%

5 inputs

JSON

100

48865

3.55%

11.55%

10 inputs

JSON

100

99252

4.4%

11.52%

20 inputs

JSON

100

201040

4.24%

11.46%

40 inputs

JSON

100

397136

7.89%

11.39%

CO2 Stack - Classic Cluster (1 IDM)

  • 1 cluster master (c5.xlarge)
  • 3 search heads (c6i.xlarge)
    • 4 CPU core
    • 8 GB RAM
  • 3 indexers (is4gen.4xlarge)
    • 16 CPU
    • 96 GiB Memory
  • 1 Input Data Manager (IDM) (c6i.4xlarge)
  • 16 CPU core
  • 32 GB RAM
  • Default TA configs

1 input

CSV

10

1596

4.5%

9.87%

5 inputs

CSV

10

7670

4.25%

9.84%

10 inputs

CSV

10

15129

4.75%

9.48%

20 inputs

CSV

10

30278

4.45%

9.06%

40 inputs

CSV

10

60220

6.5%

8.68%

1 input

CSV

100

8708

5.3%

9.89%

5 inputs

CSV

100

45657

4.45%

9.46%

10 inputs

CSV

100

92601

4.91%

9.15%

20 inputs

CSV

100

185263

6.57%

8.86%

40 inputs

CSV

100

365956

9.09%

8.45%

1 input

JSON

10

1591

5.7%

6.98%

5 inputs

JSON

10

7550

6.95%

4.4%

10 inputs

JSON

10

15376

4.75%

6.88%

20 inputs

JSON

10

30534

5.5%

6.7%

40 inputs

JSON

10

60647

8.05%

6.23%

1 input

JSON

100

11024

4.3%

8.43%

5 inputs

JSON

100

48298

4%

7.97%

10 inputs

JSON

100

93452

4.81%

7.53%

20 inputs

JSON

100

182911

6.92%

7.44%

40 inputs

JSON

100

372861

6.78%

7.54%

Version 1.3.0 Pub/Sub input performance statistics

Use the following information on version 1.3.0 Pub/Sub input performance statistics to enhance the performance of your own Google Cloud Platform collection tasks.

Testing architecture

The throughput data and conclusions provided here are based on performance testing using single-instance Splunk Enterprise 7.0.1 running on the following environment.

Instance type n1-standard-8
Memory 30 GB
vCPU 8
Cores 4 CPU cores
Storage Type standard persistent disk (2000GB)

Measured performance data

The throughput data provided here is the average performance for different subscription numbers achieved in performance testing under specific operating conditions and is subject to change when any of the hardware and software variables changes. Use this data for a rough reference only.

Subscriptions Input numbers Throughput (KB/s) Throughput (GB/day)
1 1 1800 150
4 4 9000 740
8 8 17000 1400
16 16 24000 1970
200 16 16320 1340

For heavy data ingestion scenarios, change the default Acknowledgment Deadline of your subscription from 10 seconds to 60 seconds for optimal performance and to avoid data duplication.