Splunk APM, Lambda Functions & Traces

Let’s start by making sure that Splunk APM is aware of our Environment from the trace data it is receiving. This is the deployment.name we set as part of the OTEL_RESOURCE_ATTRIBUTES variable we set on our Lambda function definitions in main.tf. It was also one of the outputs from the terraform apply command we ran earlier.

In Splunk Observability Cloud:

• Click on the APM Button from the Main Menu on the left. This will take you to the Splunk APM Overview.
• Select your APM Environment from the Environment: dropdown.
  • Your APM environment should be in the PREFIX-lambda-shop format, where the PREFIX is obtained from the environment variable you set in the Prerequisites section

Once you’ve selected your Environment name from the Environment drop down, you can take a look at the Service Map for your Lambda functions.

• Click the Service Map Button on the right side of the APM Overview page. This will take you to your Service Map view.

You should be able to see the producer-lambda function and the call it is making to the Kinesis Stream to put your record.

Explore the Traces from your Lambda Functions #

• Click the Traces button to view the Trace Analyzer.

On this page, we can see the traces that have been ingested from the OpenTelemetry Lambda layer of your producer-lambda function.

• Select a trace from the list to examine by clicking on its hyperlinked Trace ID.

We can see that the producer-lambda function is putting a record into the Kinesis Stream. But the action of the consumer-lambda function is missing!

This is because the trace context is not being propagated. Trace context propagation is not supported out-of-the-box by Kinesis service at the time of this workshop. Our distributed trace stops at the Kinesis service, and because its context isn’t automatically propagated through the stream, we can’t see any further.

Not yet, at least…

Let’s see how we work around this in the next section of this workshop. But before that, let’s clean up after ourselves!

Stop the messages #

The resources we deployed as part of this auto-instrumenation exercise need to be cleaned. Likewise, the script that was generating traffice against our producer-lambda endpoint needs to be stopped, if it’s still running. Follow the below steps to clean up.

• If the send_message.py script is still running, stop it with the follwing commands:

fg

• This brings your background process to the foreground.
• Next you can hit [CONTROL-C] to kill the process.

Destroy all AWS resources #

Terraform is great at managing the state of our resources individually, and as a deployment. It can even update deployed resources with any changes to their definitions. But to start afresh, we will destroy the resources and redeploy them as part of the manual instrumentation portion of this workshop.

Please follow these steps to destroy your resources:

• Change to the auto directory:

cd ~/workshop/lambda/auto

• Destroy the Lambda functions and other AWS resources you deployed earlier:

terraform destroy

• respond yes when you see the Enter a value: prompt
• This will result in the resources being destroyed, leaving you with a clean environment

This process will leave you with the files and directories created as a result of our activity. Do not worry about those.