Skip to content

Configure CloudWatch Log inputs for the Splunk Add-on for AWS

Complete the steps to configure CloudWatch Log inputs for the Splunk Add-on for Amazon Web Services (AWS):

  1. You must manage accounts for the add-on as a prerequisite. See Manage accounts for the Splunk Add-on for AWS.
  2. Configure AWS services for the CloudWatch Log input.
  3. Configure AWS permissions for the CloudWatch Log input.
  4. (Optional) Configure VPC Interface Endpoints for STS and logs services from your AWS Console if you want to use private endpoints for data collection and authentication. For more information, see the Interface VPC endpoints (AWS PrivateLink) topic in the Amazon Virtual Private Cloud documentation.
  5. Configure CloudWatch Log inputs either through Splunk Web or configuration files.

Due to rate limitations, don’t use pull-based (API) input configurations to collect CloudWatch Log data which has the source type aws:cloudwatchlogs:*. Instead, use push-based (Amazon Kinesis Firehose) input configurations to collect CloudWatch Log and VPC Flow Logs. The push-based (Amazon Kinesis Firehose) input configurations for the Splunk Add-on for AWS include index-time logic to perform the correct knowledge extraction for these events through the Kinesis input as well.

Configure AWS permissions for the CloudWatch Log input

Required permissions for Logs:

  • DescribeLogGroups
  • DescribeLogStreams
  • GetLogEvents
  • s3:GetBucketLocation

See the following sample inline policy to configure CloudWatch Log input permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams",
        "logs:GetLogEvents",
        "s3:GetBucketLocation"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

You must also ensure that your role has a trust relationship that allows the flow logs service to assume the role. While viewing the IAM role, choose Edit Trust Relationship and replace that policy with this one:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "vpc-flow-logs.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Configure a CloudWatch Logs input using Splunk Web

To configure inputs using Splunk Web, click Splunk Add-on for AWS in the navigation bar on Splunk Web home, then choose one of the following menu paths depending on the data type you want to collect:

  • Create New Input > VPC Flow Logs > CloudWatch Logs
  • Create New Input > Custom Data Type > CloudWatch Logs

Fill out the fields as described in the table:

Argument in configuration file

Field in Splunk Web

Description

account

AWS Account

The AWS account or EC2 IAM role the Splunk platform uses to access your CloudWatch Logs data. In Splunk Web, select an account from the drop-down list. In aws_cloudwatch_logs_tasks.conf, enter the friendly name of one of the AWS accounts that you configured on the Configuration page or the name of the automatically discovered EC2 IAM role.

region

AWS Region

The AWS region that contains the data. In aws_cloudwatch_logs_tasks.conf, enter the region ID.

private_endpoint_enabled

Use Private Endpoints

Check the checkbox to use private endpoints of AWS Security Token Service (STS) and AWS Simple Cloud Storage (S3) services for authentication and data collection. In inputs.conf, enter 0 or 1 to respectively disable or enable use of private endpoints.

logs_private_endpoint_url

Private Endpoint (Logs)

Private Endpoint (Interface VPC Endpoint) of your logs service, which can be configured from your AWS console.
Supported Formats :
://vpce--.logs..vpce.amazonaws.com ://vpce---.logs..vpce.amazonaws.com

sts_private_endpoint_url

Private Endpoint (STS)

Private Endpoint (Interface VPC Endpoint) of your STS service, which can be configured from your AWS console.
Supported Formats :
://vpce--.sts..vpce.amazonaws.com ://vpce---.sts..vpce.amazonaws.com

groups

Log group

A comma-separated list of log group names.
Do not use wildcards.

only_after

Only After

GMT time string in '%Y-%m-%dT%H:%M:%S' format. If set, only events after this time are queried and indexed. Defaults to 1970-01-01T00:00:00.

stream_matcher

Stream Matching Regex

REGEX to strictly match stream names. Defaults to .*

interval

Interval

The number of seconds to wait before the Splunk platform runs the command again. The default is 600 seconds.

metric_index_flag

Use Metric Index?

Whether to use metric index or event index. The default value is No (use event index). This field is only visible when creating VPC Flow Logs -> CloudWatch Logs inputs.

sourcetype

Source type

A source type for the events.
If you are indexing VPC Flow Log data through CloudWatch Logs:

  1. If using event index, the sourcetype value is aws:cloudwatchlogs:vpcflow.
  2. If using metric index, the sourcetype value is aws:cloudwatchlogs:vpcflow:metric.
Enter aws:cloudwatchlogs if you are collecting any other types of CloudWatch Logs data.

index

Index

The index name where the Splunk platform puts the CloudWatch Logs data. The default is main.

query_window_size

Query Window Size (minutes)

Specify the interval of data to be collected in each request(in minutes). Default=10, Min=1 & Max=43200(30days).

For example, if the calculated start date is 2024-01-01T00:00:00 (midnight on January 1, 2024) and query window size is 60 minutes, then end date for the request will be 2024-01-01T00:01:00 (one hour after midnight). The time period will continue sliding by 60 minutes until no more recent logs are available..

Configure a CloudWatch Logs input using configuration files

To configure the input using configuration files, create $SPLUNK_HOME/etc/apps/Splunk_TA_aws/local/aws_cloudwatch_logs_tasks.conf using the following template:

[<name>]
account = <value>
groups = <value>
index = <value>
interval = <value>
only_after = <value>
region = <value>
private_endpoint_enabled = <value>
logs_private_endpoint_url = <value>
sts_private_endpoint_url = <value>
sourcetype = <value>
stream_matcher = <value>
metric_index_flag = <value>
query_window_size = <value>

Here is an example stanza that collects VPC Flow Log data from two log groups:

[splunkapp2:us-west-2]
account = splunkapp2
groups = SomeName/DefaultLogGroup, SomeOtherName/SomeOtherLogGroup
index = default
interval = 600
only_after = 1970-01-01T00:00:00
region = us-west-2
sourcetype = aws:cloudwatchlogs:vpcflow
stream_matcher = eni.*
metric_index_flag = 0
query_window_size = 10