Skip to content

Sizing, performance, and cost considerations for the Splunk Add-on for AWS

Before you configure the Splunk Add-on for Amazon Web Services (AWS), review these sizing, performance, and cost considerations.

General

See the following table for the recommended maximum daily indexing volume on a clustered indexer for different AWS source types. This information is based on a generic Splunk hardware configuration. Adjust the number of indexers in your cluster based on your actual system performance. Add indexers to a cluster to improve indexing and search retrieval performance. Remove indexers to a cluster to avoid within-cluster data replication traffic.

Source type Daily indexing volume per indexer (GB)
aws:cloudwatchlogs:vpcflow 25-30
aws:s3:accesslogs 80- 20
aws:cloudtrail 150-200
aws:billing 50- 00

These sizing recommendations are based on the Splunk platform hardware configurations in the following table. You can also use the System requirements for use of Splunk Enterprise on-premises in the Splunk Enterprise Installation Manual as a reference.

Splunk platform type CPU cores RAM EC2 instance type
Search head 8 16 GB c4.xlarge
Indexer 16 64 GB m4.4xlarge

Input configuration screens require data transfer from AWS to populate the services, queues, and buckets available to your accounts. If your network to AWS is slow, data transfers might be slow to load. If you encounter timeout issues, you can manually type in resource names.

Performance for the Splunk Add-on for AWS data inputs

The rate of data ingestion for this add-on depends on several factors: deployment topology, number of keys in a bucket, file size, file compression format, number of events in a file, event size, and hardware and networking conditions.

See the following tables for measured throughput data achieved under certain operating conditions. Use the information to optimize the Splunk Add-on for AWS add-on in your own production environment. Because performance varies based on user characteristics, application usage, server configurations, and other factors, specific performance results cannot be guaranteed. Contact Splunk Support for accurate performance tuning and sizing.

The Kinesis input for the Splunk Add-on for AWS has its own performance data. See Configure Kinesis inputs for the Splunk Add-on for AWS.

Reference hardware and software environment

Throughput data and conclusions are based on performance testing using Splunk platform instances (dedicated heavy forwarders and indexers) running on the following environment:
Instance type M4 Double Extra Large (m4.4xlarge)
Memory 64 GB
Compute Units (ECU) 53.5
vCPU 16
Storage (GB) 0 (EBS only)
Arch 64-bit
EBS optimized (max bandwidth) 2000 Mbps
Network performance High

The following settings are configured in the outputs.conf file on the heavy forwarder:

useACK = true

maxQueueSize = 15MB

Measured performance data

The throughput data is the maximum performance for each single input achieved in performance testing under specific operating conditions and is subject to change when any of the hardware and software variables changes. Use this data as a rough reference only.

Single-input max throughput

Data input

Source type

Max throughput (KBs)

Max EPS (events)

Max throughput (GB/day)

Generic S3

aws:elb:accesslogs
(plain text, syslog, event size 250 B, S3 key size 2 MB)

17,000

86,000

1,470

Generic S3

aws:cloudtrail
(gz, json, event size 720 B, S3 key size 2 MB)

11,000

35,000

950

Incremental S3

aws:elb:accesslogs
(plain text, syslog, event size 250 B, S3 key size 2 MB)

11,000

43,000

950

Incremental S3

aws:cloudtrail
(gz, json, event size 720 B, S3 key size 2 MB)

7,000

10,000

600

SQS-based S3

aws:elb:accesslogs
(plain text, syslog, event size 250 B, S3 key size 2 MB)

12,000

50,000

1,000

SQS-based S3

aws:elb:accesslogs
(gz, syslog, event size 250 B, S3 key size 2 MB)

24,000

100,000

2,000

SQS-based S3

aws:cloudtrail
(gz, json, event size 720 B, S3 key size 2 MB)

13,000

19,000

1,100

CloudWatch logs [1]

aws:cloudwatchlog:vpcflow

1,000

6,700

100

CloudWatch
(ListMetric, 10,000 metrics)

aws:cloudwatch

240 (Metrics)

NA

NA

CloudTrail

aws:cloudtrail
(gz, json, sqs=1,000, 9,000 events/key)

5,000

7,000

400

Kinesis

aws:cloudwatchlog:vpcflow
(json, 10 shards)

15,000

125,000

1,200

SQS

aws:sqs
(json, event size 2,800)

N/A

160

N/A

[1] API throttling error occurs if input streams are greater than 1,000.

Multi-inputs max throughput

The following throughput data was measured with multiple inputs configured on a heavy forwarder in an indexer cluster distributed environment.

Consolidate AWS accounts during add-on configuration to reduce CPU usage and increase throughput performance.

Data input

Source type

Max throughput (KBs)

Max EPS (events)

Max throughput (GB/day)

Generic S3

aws:elb:accesslogs
(plain text, syslog, event size 250 B, S3 key size 2 MB)

23,000

108,000

1,980

Generic S3

aws:cloudtrail
(gz, json, event size 720 B, S3 key size 2 MB)

45,000

130,000

3,880

Incremental S3

aws:elb:accesslogs
(plain text, syslog, event size 250 B, S3 key size 2 MB)

34,000

140,000

2,930

Incremental S3

aws:cloudtrail
(gz, json, event size 720 B, S3 key size 2 MB)

45,000

65,000

3,880

SQS-based S3 [1]

aws:elb:accesslogs
(plain text, syslog, event size 250 B, S3 key size 2 MB)

35,000

144,000

3,000

SQS-based S3 [1]

aws:elb:accesslogs
(gz, syslog, event size 250 B, S3 key size 2 MB)

42,000

190,000

3,600

SQS-based S3 [1]

aws:cloudtrail
(gz, json, event size 720 B, S3 key size 2 MB)

45,000

68,000

3,900

CloudWatch logs

aws:cloudwatchlog:vpcflow

1,000

6,700

100

CloudWatch (ListMetric)

aws:cloudwatch
(10,000 metrics)

240 (metrics/s)

NA

NA

CloudTrail

aws:cloudtrail
(gz, json, sqs=100, 9,000 events/key)

20,000

15,000

1,700

Kinesis

aws:cloudwatchlog:vpcflow
(json, 10 shards)

18,000

154,000

1,500

SQS

aws:sqs
(json, event size 2.8K)

N/A

670

N/A

[1] Performance testing of the SQS-based S3 input indicates that optimal performance throughput is reached when running four inputs on a single heavy forwarder instance. To achieve higher throughput performance beyond this bottleneck, you can further scale out data collection by creating multiple heavy forwarder instances each configured with up to four SQS-based S3 inputs to concurrently ingest data by consuming messages from the same SQS queue.

Max inputs benchmark per heavy forwarder

The following input number ceiling was measured with multiple inputs configured on a heavy forwarder in an indexer cluster distributed environment. CPU and memory resources were utilized to their fullest.

It is possible to configure more inputs than the maximum number indicated in the table if you have a smaller event size, fewer keys per bucket, or more available CPU and memory resources in your environment.

Data input Sourcetype Format Number of keys/bucket Event size Max inputs
S3 aws:s3 zip, syslog 100,000 100 B 300
S3 aws:cloudtrail gz, json 1,300,000 1 KB 30
Incremental S3 aws:cloudtrail gz, json 1,300,000 1 KB 20
SQS-based S3 aws:cloudtrail, aws:config gz, json 1,000,000 1 KB 50

Memory usage benchmark for generic S3 inputs

Event size Number of events per key Total number of keys Archive type Number of inputs Memory used
1,000 1,000 10,000 zip 20 20 G
1,000 1,000 1,000 zip 20 12 G
1,000 1,000 10,000 zip 10 18 G
100 B 1,000 10,000 zip 10 15 G

If you do not achieve the expected AWS data ingestion throughput, see Troubleshoot the Splunk Add-on for AWS.

Billing

The following table provides general guidance on sizing, performance, and cost considerations for the Billing data input:

Consideration

Notes

Sizing and performance

Detailed billing reports can be very large in size, depending on your environment. If you configure the add-on to collect detailed reports, it collects all historical reports available in the bucket by default. In addition, for each newly finalized monthly and detailed report, the add-on collects new copies of the same report once per interval until the etag is unchanged.
Configure separate inputs for each billing report type that you want to collect. Use the regex and interval parameters in the input configuration page of the add-on to limit the number of reports that you collect with each input.

AWS cost

Billing reports themselves do not incur charges, but standard S3 charges apply.
See https://aws.amazon.com/s3/pricing/.

Billing Cost and Usage Report

The following table provides general guidance on sizing, performance, and cost considerations for the Billing Cost and Usage Report data input. Testing was conducted using version 7.1.0 of the Splunk Add-on for AWS.

Splunk Platform Environment Architecture setup Number of inputs Event Count Data Collection time Max CPU % Max RAM %
Customer Managed Platform (CMP) m5.4xlarge (vcpu 16 / 64 GiB ram) 1 2,828,618 ~8mins 12.13% 0.20%
Classic Inputs Data Manager (IDM) m5.4xlarge (vcpu 16 / 64 GiB ram) 1 2,828,618 ~11mins 12.09% 0.21%
Victoria Search Head Cluster (SHC) m5.4xlarge (vcpu 16 / 64 GiB ram) 1 2,828,618 ~6mins 12.01% 0.20%

CloudTrail

The following table provides general guidance on sizing, performance, and cost considerations for the CloudTrail data input:

Consideration

Notes

Sizing and performance

None.

AWS cost

Using CloudTrail itself does not incur charges, but standard S3, SNS, and SQS charges apply.
See https://aws.amazon.com/pricing/services/.

Config

The following table provides general guidance on sizing, performance, and cost considerations for the Config data input:

Consideration

Notes

Sizing and performance

None.

AWS cost

Using Config incurs charges from AWS. See http://aws.amazon.com/config/pricing/.
In addition, standard S3, SNS, and SQS charges apply. See http://aws.amazon.com/pricing/services/.

Config Rules

The following table provides general guidance on sizing, performance, and cost considerations for the Config Rules data input:

Consideration Notes
Sizing and performance None.
AWS cost None.

CloudWatch

The following table provides general guidance on sizing, performance, and cost considerations for the CloudWatch data input:

Consideration

Notes

Sizing and performance

The smaller the granularity you configure, the more events you collect.
Create separate inputs that match your needs for different regions, services, and metrics. For each input, configure a granularity that matches the precision that you require, setting a larger granularity value in cases where indexing fewer, less-granular events is acceptable. You can increase granularity temporarily when a problem is detected. AWS rate-limits the number of free API calls against the CloudWatch API. With a period of 300 and a polling interval of 1,800, collecting data for 2 million metrics does not, by itself, exceed the current default rate limit, but collecting 4 million metrics does exceed it. If you have millions of metrics to collect in your environment, consider paying to have your API limit raised, or remove less essential metrics from your input and configure larger granularities in order to make fewer API calls.

AWS cost

Using CloudWatch and making requests against the CloudWatch API incurs charges from AWS.
See https://aws.amazon.com/cloudwatch/pricing/.

CloudWatch Logs (VPC Flow Logs)

The following table provides general guidance on sizing, performance, and cost considerations for the CloudWatch Logs (VPC Flow Logs) data input:

Consideration

Notes

Sizing and performance

AWS limits each account to 10 requests per second, each of which returns no more than 1 MB of data. In other words, the data ingestion and indexing rate is no more than 10 MB/s. The add-on modular input can process up to 4,000 events per second in a single log stream.
Best practices:

  • If volume is a concern, configure the only_after parameter to limit the amount of historical data you collect.
  • If you have high volume VPC Flow Logs, configure one or more Kinesis inputs to collect them instead of using the CloudWatch Logs input.

AWS cost

Using CloudWatch Logs incurs charges from AWS. See https://aws.amazon.com/cloudwatch/pricing/.
Transferring data out of CloudWatch Logs incurs charges from AWS. See https://aws.amazon.com/ec2/pricing/.

CloudWatch Metrics

The following table provides general guidance on sizing, performance, and cost considerations for the CloudWatch Metrics data input. Testing was conducted using version 7.1.0 of the Splunk Add-on for AWS.

The Number of API calls means m*n where m is the number of unique metric dimensions and n is the number of unique metric names.

Splunk Platform Environment Architecture setup Number of inputs Number of API calls Event Count Data Collection time Max CPU % Max RAM %
Customer Managed Platform (CMP) m5.4xlarge (vcpu 16 / 64 GiB ram) 1 200000 400000 ~28mins 16.03% 1.67%
Classic Inputs Data Manager (IDM) m5.4xlarge (vcpu 16 / 64 GiB ram) 1 200000 400000 ~35mins 14.89% 1.83%
Victoria Search Head Cluster (SHC) m5.4xlarge (vcpu 16 / 64 GiB ram) 1 200000 400000 ~32mins 15.09% 1.70%

Incremental S3

The following table provides general guidance on sizing, performance, and cost considerations for the Incremental S3 data input. Testing was conducted using version 7.1.0 of the Splunk Add-on for AWS.

Splunk Platform Environment Architecture setup Number of inputs Event count Data Collection time Max CPU % Max RAM %
Customer Managed Platform (CMP) m5.4xlarge (vcpu 16 / 64 GiB RAM) 1 10491968 ~104mins 0.05% 0.10%
Classic Inputs Data Manager (IDM) m5.4xlarge (vcpu 16 / 64 GiB RAM) 1 10491968 ~105mins 0.21% 0.11%
Victoria Search Head Cluster (SHC) m5.4xlarge (vcpu 16 / 64 GiB RAM) 1 10491968 ~104mins 0.02% 0.11%

Inspector

The following table provides general guidance on sizing, performance, and cost considerations for the Inspector data input:

Consideration Notes
Sizing and performance None.
AWS cost Using Amazon Inspector incurs charges from AWS. See https://aws.amazon.com/inspector/pricing/.

Kinesis

The following table provides general guidance on sizing, performance, and cost considerations for the Kinesis data input:

Consideration Notes
Sizing and performance See Performance reference for the Kinesis input in the Splunk Add-on for AWS.
AWS cost Using Amazon Kinesis incurs charges from AWS. See https://aws.amazon.com/kinesis/streams/pricing/.

S3

The following table provides general guidance on sizing, performance, and cost considerations for the S3 data input:

Consideration

Notes

Sizing and performance

AWS throttles S3 data collection at the bucket level, so expect some delay before all data arrives in your Splunk platform.
You can configure multiple S3 inputs for a single S3 bucket to improve performance. The Splunk platform dedicates one process for each data input, so provided that your system has sufficient processing power, performance improves with multiple inputs. See Performance reference for the S3 input in the Splunk Add-on for AWS.

AWS cost

Using S3 incurs charges from AWS. See https://aws.amazon.com/s3/pricing/.

Security Lake

The following tables provide general guidance on sizing, performance, and cost considerations for the Amazon Security Lake data input. Files ranging in size from 20KB to 200MB were used to collect the performance stats.

Splunk Platform Environment

Architecture setup

Number of indexers

Number of inputs

Batch size

Heavy forwarder/IDM CPU %

Heavy forwarder/IDM RAM %

Expected Average Throughput Indexed

Customer Managed Platform (CMP)
  • Heavy Forwarder (m5.4xlarge)
    • 16 CPU
    • 64 GB Memory

N/A

1

5

11.77%

3.37%

3.33 GB/h

Splunk Cloud Classic
  • IDM (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory
  • Indexers (is4gen.4xlarge)
    • 16 CPU
    • 96 GB Memory
  • Search heads (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory

3

1

5

99.90%

22.06%

2.82 GB/h

Splunk Cloud Victoria
  • Indexer (is4gen.4xlarge)
    • 16 CPU
    • 96 GB Memory
  • IDM (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory
  • Heavy Forwarder (m5.4xlarge)
    • 16 CPU
    • 64 GB Memory

3

1

5

54.28%

22.78%

2.58 GB/h

Customer Managed Platform (CMP)
  • Heavy Forwarder (m5.4xlarge)
    • 16 CPU
    • 64 GB Memory

N/A

2

5

9.93%

3.20%

7.72 GB/h

Splunk Cloud Classic
  • IDM (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory
  • Indexers (is4gen.4xlarge)
    • 16 CPU
    • 96 GB Memory
  • Search heads (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory

6

2

5

99.95%

22.71%

5.60 GB/h

Splunk Cloud Victoria
  • Indexer (is4gen.4xlarge)
    • 16 CPU
    • 96 GB Memory
  • IDM (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory
  • Heavy Forwarder (m5.4xlarge)
    • 16 CPU
    • 64 GB Memory

6

2

5

55.68%

24.13%

5.28 GB/h

Customer Managed Platform (CMP)
  • Heavy Forwarder (m5.4xlarge)
    • 16 CPU
    • 64 GB Memory

N/A

5

5

85.42%

13.65%

277 GB/h

Splunk Cloud Classic
  • IDM (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory
  • Indexers (is4gen.4xlarge)
    • 16 CPU
    • 96 GB Memory
  • Search heads (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory

9

5

5

99.95%

27.29%

96.93 GB/h

Splunk Cloud Victoria
  • Indexer (is4gen.4xlarge)
    • 16 CPU
    • 96 GB Memory
  • Search heads (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory

9

5

5

66.45%

21.18%

214 GB/h

Customer Managed Platform (CMP)
  • Heavy Forwarder (m5.4xlarge)
    • 16 CPU
    • 64 GB Memory

N/A

1

10

10.03%

3.07%

5.07 GB/h

Splunk Cloud Classic
  • IDM (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory
  • Indexers (is4gen.4xlarge)
    • 16 CPU
    • 96 GB Memory
  • Search heads (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory

3

1

10

99.95%

23.20%

5.32 GB/h

Splunk Cloud Victoria
  • Indexer (is4gen.4xlarge)
    • 16 CPU
    • 96 GB Memory
  • Search heads (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory

3

1

10

54.69%

21.14%

5.22 GB/h

Customer Managed Platform (CMP)
  • Heavy Forwarder (m5.4xlarge)
    • 16 CPU
    • 64 GB Memory

N/A

2

10

15.02%

3.31%

8.99 GB/h

Splunk Cloud Classic
  • IDM (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory
  • Indexers (is4gen.4xlarge)
    • 16 CPU
    • 96 GB Memory
  • Search heads (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory

6

2

10

99.95%

25.89%

10.78 GB/h

Splunk Cloud Victoria
  • Indexer (is4gen.4xlarge)
    • 16 CPU
    • 96 GB Memory
  • Search heads (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory

6

2

10

57.58%

20.83%

9.58 GB/h

Customer Managed Platform (CMP)
  • Heavy Forwarder (m5.4xlarge)
    • 16 CPU
    • 64 GB Memory

N/A

5

10

82.09%

16%

278 GB/h

Splunk Cloud Classic
  • IDM (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory
  • Indexers (is4gen.4xlarge)
    • 16 CPU
    • 96 GB Memory
  • Search heads (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory

9

5

10

99.93%

22.96%

100 GB/h

Splunk Cloud Victoria
  • Indexer (is4gen.4xlarge)
    • 16 CPU
    • 96 GB Memory
  • Search heads (m6i.4xlarge)
    • 16 CPU
    • 64 GB Memory

9

5

10

61.59%

19.63%

325 GB/h

Performance reference notes:

  • The Amazon Security Lake data input is stateless, so multiple inputs can be configured against the same SQS queue.
  • The following configuration settings are to scale:
    • Batch Size: The number of threads spawned by a single input. For example, n=10 will process 10 messages in parallel).
    • Number of Amazon Security Lake inputs.
  • If you have horizontally scaled the SQS-based S3 input by configuring multiple inputs with the same SQS queue, and your file size in the S3 bucket is not consistent, then the best practice is to decrease batch size (minimum up to 1), as batches are processed sequentially.

Transit Gateway Flow Logs

The following tables provide general guidance on sizing, performance, and cost considerations for the Transit Gateway Flow Logs data input. Files with 1MB size were used to collect the performance stats. The batch size for all the inputs was 10.

Common architecture setup

Number of indexers

Number of inputs

Heavy forwarder/IDM CPU %

Heavy forwarder/IDM RAM %

Expected Average Throughput Indexed

Customer Managed Platform (CMP)

  • Heavy Forwarder (m5.4xlarge)
    • 16 CPU
    • 64 GB RAM

N/A

1

38.40%

7.26%

26578 KB/m

N/A

5

50.75%

6.98%

40116 KB/m

Splunk platform environment - Victoria Search Head Cluster

  • cluster master (c5.xlarge)
  • search heads (m6i.4xlarge)
    • 16 CPU
    • 64 GB RAM
  • indexer cluster (im4gn.4xlarge)
    • 16 CPU
    • 64 GB RAM
  • parallelIngestionPipelines is set to 2

3

1

24.34%

9.20%

40483 KB/m

9

5

41.12%

18.05%

61498 KB/m

Splunk platform environment - Classic Cluster (1 IDM)

  • cluster master (c5.xlarge)
  • search heads (m6i.4xlarge)
    • 16 CPU
    • 64 GB RAM
  • indexer cluster (im4gn.4xlarge)
    • 16 CPU
    • 64 GB RAM
  • Inputs Data Manager (IDM) (m6i.4xlarge)
    • 16 CPU
    • 64 GB RAM

3

1

22.37%

7.52%

45048 KB/m

9

5

29.05%

20.40%

53792 KB/m

SQS

The following table provides general guidance on sizing, performance, and cost considerations for the SQS data input:

Consideration Notes
Sizing and performance None.
AWS cost Using SQS incurs charges from AWS. See https://aws.amazon.com/sqs/pricing/.

SNS

The following table provides general guidance on sizing, performance, and cost considerations for the SNS data input:

Consideration Notes
Sizing and performance None.
AWS cost Using SNS incurs charges from AWS. See https://aws.amazon.com/sns/pricing/.