Configure CloudWatch inputs for the Splunk Add-on for AWS¶

Starting in version 7.1.0 of the Splunk Add-on for AWS, the file based checkpoint mechanism was migrated to the Splunk KV Store for Cloudwatch Metrics inputs. The CloudWatch metrics inputs must be disabled whenever you restart the Splunk software. Otherwise, it will result in data duplication against your already configured inputs.

Complete the steps to configure CloudWatch and EventBridge inputs for the Splunk Add-on for Amazon Web Services (AWS):

You must manage accounts for the add-on as a prerequisite. See Manage accounts for the Splunk Add-on for AWS.
Configure AWS services for the CloudWatch input.
Configure AWS permissions for the CloudWatch input.
(Optional) Configure VPC Interface Endpoints for STS, monitoring, ELB, EC2, Autoscaling, Lambda and S3 services from your AWS Console if you want to use private endpoints for data collection and authentication. Configuration of all service endpoints is not required. Configure only those endpoints which are required for each specific metric. For more information, see the Interface VPC endpoints (AWS PrivateLink) topic in the Amazon Virtual Private Cloud documentation.
Configure CloudWatch inputs either through Splunk Web or configuration files.

Configure separate CloudWatch inputs for each metric or set of metrics that have different minimum granularities, based on the sampling period that AWS allows for that metric. For example, CPU Utilization has a sampling period of 5 minutes, whereas Billing Estimated Charge has a sampling period of 4 hours. If you configure a granularity that is smaller than the minimum sampling period available in AWS, the input wastes API calls.

For more information, see Sizing, performance, and cost considerations for the Splunk Add-on for AWS.

To improve the data collection of CloudWatch input, Configure the CloudWatch Max Threads parameter from Add-on Global Settings. For more information, see Add-on Global Settings.

Configure AWS services for the CloudWatch input¶

To enable AWS to produce billing metrics in CloudWatch, turn on Receive Billing Alerts in the Preferences section of the Billing and Cost Management console.

The CloudWatch service is automatically enabled to collect free metrics for your AWS services and requires no additional configuration for the Splunk Add-on for AWS.

Configure CloudWatch permissions¶

Required permissions for CloudWatch: Describe*, Get*, List*

Required permissions for Autoscaling: Describe*

Required permissions for EC2: Describe*

Required permissions for S3: List*

Required permissions for SQS: List*

Required permissions for SNS: List*

Required permissions for Lambda: List*

Required permissions for ELB: Describe*

See the following sample inline policy to configure CloudWatch input permissions:

{
    "Statement": [{
        "Action": [
            "cloudwatch:List*",
            "cloudwatch:Get*",
            "autoscaling:Describe*",
            "ec2:Describe*",
            "s3:List*",
            "sqs:List*",
            "sns:List*",
            "lambda:List*",
            "elasticloadbalancing:Describe*"
        ],
        "Effect": "Allow",
        "Resource": "*"
    }],
    "Version": "2012-10-17"
}

For more information and sample policies, see: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/UsingIAM.html

Configure a CloudWatch input using Splunk Web¶

To configure inputs in Splunk Web:

Click Splunk Add-on for AWS in the navigation bar on Splunk Web home.
Click Create New Input > CloudWatch.
Click Advanced to edit Metrics Configuration.
Use the following table to complete the fields for the new input in the .conf file or in Splunk Web:

Argument in configuration file	Field in Splunk Web	Description
`aws_account`	AWS Account	The AWS account or EC2 IAM role the Splunk platform uses to access your CloudWatch data. In Splunk Web, select an account from the drop-down list. In inputs.conf, enter the friendly name of one of the AWS accounts that you configured on the Configuration page or the name of the automatically discovered EC2 IAM role.
`aws_iam_role`	Assume Role	The IAM role to assume, see Manage accounts for the Splunk Add-on for AWS.
`aws_region`	AWS Regions	The AWS region name or names. In Splunk Web, select one or more regions from the drop-down list. In inputs.conf, enter one or more valid AWS region IDs, separated by commas. See the AWS service endpoints for more information.
`interval`	Interval	The number of seconds to wait before the Splunk platform runs the command again. Set polling interval using the interval parameter according to your requirements in the inputs.conf file. Default value is 300 or 5 mins
Metrics Configuration arguments
`metric_dimensions`	Dimensions	CloudWatch metric dimensions display as a JSON array, with strings as keys and regular expressions as values. Splunk Web automatically populates correctly formatted JSON objects to collect all metric dimensions in the namespace you have selected. If you want, you can customize the JSON object to limit the collection to just the dimensions you want to collect. For example, for the SQS namespace, you can collect only the metrics for queue names that start with “splunk” and end with “_current” by entering `[{"QueueName": ["\"splunk._current\\\\s\""]}]`. You can set multiple dimensions in one data input. If you use a JSON array, the dimension matched by the JSON object in the array is matched. A JSON object has strings as keys and values that are either a regex or an array of regexes. The Splunk Add-on for AWS supports one JSON object per JSON array. For example `[{"key1": "regex1"}, {"key2": "regex2"}]` is not supported. A dimension is matched to the object if it meets these two conditions: It has the same key set to the object In the value of each key, there is one or more elements matched by every regex in the value to the key in the JSON object. For example, `[{"key":["val.", ".*lue"]}]` matches `{"key":"value"}` and `{"key":["value"]}`, but not `{"key":"value", "key2":"value2"}`. The BucketName dimension does not support wildcards or arrays with length greater than one. When you collect metrics from the AWS S3 namespace, configure separate CloudWatch inputs for each S3 bucket. For example, `{"StorageType": ["StandardStorage"], "BucketName": ["my_favorite_bucket"]}`.
`metric_names`	Metrics	CloudWatch metric names in JSON array. For example, `["CPUUtilization","DiskReadOps","StatusCheckFailed_System"]`. Splunk Web automatically populates correctly formatted JSON objects for all metric names in the namespace you have selected. Edit the JSON object to remove any metrics you do not want to collect. Collecting metrics you do not need results in unnecessary API calls.
`metric_namespace`	Namespace	The metric namespace. For example, `AWS/EBS`. In Splunk Web, click + Add Namespace and select a namespace from the drop-down list or manually enter it. If you manually enter a custom namespace, you need to type in all your JSON objects manually for the remaining fields. In inputs.conf, enter a valid namespace for the region you specified. You can only specify one metric namespace per input.
`metric_expiration`	Metric Expiration	Duration of time the discovered metrics are cached for, measured in seconds.
`index`	Index	The index name where the Splunk platform puts the CloudWatch data. The default is main.
`period`	Period	The granularity, in seconds, of the returned data points. For metrics with regular resolution, a period can be as short as 60 seconds (1 minute) and must be a multiple of 60. Different AWS metrics can support different minimum granularity based on the sampling period that AWS allows for that metric. For example, CPUUtilization has a sampling period of 5 minutes, whereas Billing Estimated Charge has a sampling period of 4 hours. Do not configure a granularity that is less than the allowed sampling period for the selected metric, or else the reported granularity reflects the sampling granularity but is labeled with your configured granularity, resulting in inconsistent data. The smaller your granularity, the more precise your metrics data becomes. Configuring a small granularity is useful when you want to do precise analysis of metrics and you are not concerned about limiting your data volume. Configure a larger granularity when a broader view is acceptable or you want to limit the amount of data you collect from AWS.
`private_endpoint_enabled`	Use Private Endpoints	Check the checkbox to use private endpoints of AWS Security TokenService (STS) and AWS Simple Cloud Storage (S3) services for authentication and data collection. In inputs.conf, enter `0` or `1` to respectively disable or enable use of private endpoints.
`sts_private_endpoint_url`	Private Endpoint (STS)	Private Endpoint (Interface VPC Endpoint) of your STS service, which can be configured from your AWS console. Supported Formats: `<http/https>://vpce-<endpoint_id>-<unique_id>.sts.<region_id>.vpce.amazonaws.com` `<http/https>://vpce-<endpoint_id>-<unique_id>-<availability_zone>.sts.<region_id>.vpce.amazonaws.com`
`monitoring_private_endpoint_url`	Private Endpoint (Monitoring)	Private Endpoint (Interface VPC Endpoint) of your monitoring service, which can be configured from your AWS console. Supported Formats: `<http/https>://vpce-<endpoint_id>-<unique_id>.monitoring.<region_id>.vpce.amazonaws.com` `<http/https>://vpce-<endpoint_id>-<unique_id>-<availability_zone>.monitoring.<region_id>.vpce.amazonaws.com`
`elb_private_endpoint_url`	Private Endpoint (ELB)	Private Endpoint (Interface VPC Endpoint) of your Elastic Load Balancer (ELB) service, which can be configured from your AWS console. Supported Formats: `<http/https>://vpce-<endpoint_id>-<unique_id>.elasticloadbalancing.<region_id>.vpce.amazonaws.com` `<http/https>://vpce-<endpoint_id>-<unique_id>-<availability_zone>.elasticloadbalancing.<region_id>.vpce.amazonaws.com`
`ec2_private_endpoint_url`	Private Endpoint (EC2)	Private Endpoint (Interface VPC Endpoint) of your Elastic Compute Cloud (EC2) service, which can be configured from your AWS console. Supported Formats: `<http/https>://vpce-<endpoint_id>-<unique_id>.ec2.<region_id>.vpce.amazonaws.com` `<http/https>://vpce-<endpoint_id>-<unique_id>-<availability_zone>.ec2.<region_id>.vpce.amazonaws.com`
`autoscaling_private_endpoint_url`	Private Endpoint (Autoscaling)	Private Endpoint (Interface VPC Endpoint) of your Autoscaling service, which can be configured from your AWS console. Supported Formats: `<http/https>://vpce-<endpoint_id>-<unique_id>.autoscaling.<region_id>.vpce.amazonaws.com` `<http/https>://vpce-<endpoint_id>-<unique_id>-<availability_zone>.autoscaling.<region_id>.vpce.amazonaws.com`
`lambda_private_endpoint_url`	Private Endpoint (Lambda)	Private Endpoint (Interface VPC Endpoint) of your Lambda service, which can be configured from your AWS console. Supported Formats: `<http/https>://vpce-<endpoint_id>-<unique_id>.lambda.<region_id>.vpce.amazonaws.com` `<http/https>://vpce-<endpoint_id>-<unique_id>-<availability_zone>.lambda.<region_id>.vpce.amazonaws.com`
`s3_private_endpoint_url`	Private Endpoint (S3)	Private Endpoint (Interface VPC Endpoint) of your S3 service, which can be configured from your AWS console. Supported Formats: `<http/https>://bucket.vpce-<endpoint_id>-<unique_id>.s3.<region_id>.vpce.amazonaws.com` `<http/https>://bucket.vpce-<endpoint_id>-<unique_id>-<availability_zone>.s3.<region_id>.vpce.amazonaws.com`
`query_window_size`	Query Window Size	Window of time used to determine how far back in time to go in order to retrieve data points, measured in number of data points.
`statistics`	Metric statistics	The metric statistics you want to request. Choose from `Average, Sum, SampleCount, Maximum, Minimum`. In inputs.conf, this list must be JSON encoded. For example: `["Average","Sum","SampleCount","Maximum","Minimum"]`.
`sourcetype`	Source type	A source type for the events. Enter a value if you want to override the default of aws:cloudwatch. Event extraction relies on the default value of source type. If you change the default value, you must update props.conf as well.

Configure a CloudWatch input using configuration files¶

To configure inputs manually in inputs.conf, create a stanza using the following template and add it to $SPLUNK_HOME/etc/apps/Splunk_TA_aws/local/inputs.conf. If the file or path does not exist, create it.

[aws_cloudwatch://<name>]
aws_account = <value>
aws_iam_role=<value>
aws_region = <value>
metric_namespace = <value>
metric_names = <value>
metric_dimensions = <value>
private_endpoint_enabled = <value>
sts_private_endpoint_url = <value>
s3_private_endpoint_url = <value>
autoscaling_private_endpoint_url = <value>
ec2_private_endpoint_url = <value>
elb_private_endpoint_url = <value>
monitoring_private_endpoint_url = <value>
lambda_private_endpoint_url = <value>
statistics = <value>
period = <value>
sourcetype = <value>
index = <value>
metric_expiration = <value>
query_window_size = <value>

Some of these settings have default values that can be found in $SPLUNK_HOME/etc/apps/Splunk_TA_aws/default/inputs.conf:

[aws_cloudwatch]
start_by_shell = false
sourcetype = aws:cloudwatch
use_metric_format = false
metric_expiration = 3600
query_window_size = 24
interval = 300
python.version = python3

The previous values correspond to the default values in Splunk Web as well as some internal values that are not exposed in Splunk Web for configuration. If you choose to copy this stanza to /local and use it as a starting point to configure your inputs.conf manually, change the stanza title from aws_cloudwatch to aws_cloudwatch://<name>.

If you want to change the interval, copy the [aws_cloudwatch] stanza to the local/inputs.conf file then set the interval value as you want. It will override the default value set in default/inputs.conf

.

Send CloudWatch events to a metrics index¶

Configure the Splunk Add-on for AWS to collect CloudWatch events and send them to a metrics index.

Prerequisites¶

Splunk Enterprise version 7.2 and higher.
An existing metrics index. See Get started with metrics in the Splunk Enterprise Metrics manual to learn more about creating a metrics index.

In Splunk Web, click Splunk Add-on for AWS in the left navigation bar on Splunk Web home.
Click Create New Input > CloudWatch.
In the AWS Input Configuration section, populate the Name, AWS Account, Assume Role, and AWS Regions fields, using the previous table as a reference.
Navigate to the Splunk-related Configuration section.
In the Source Type field, type aws:cloudwatch:metric.
Click on the Index dropdown menu, and type the name of your metrics index.
Click Save.