Auto-Instrumentation

Auto-Instrumentation

Navigate to the auto directory that contains auto-instrumentation code.

cd ~/o11y-lambda-lab/auto

Inspect the contents of the files in this directory. Take a look at the serverless.yml template.

cat serverless.yml

You should see the Splunk OpenTelemetry Lambda layer being added to each fuction.

layers:
      - arn:aws:lambda:us-east-1:254067382080:layer:splunk-apm:70

You can see the relevant layer ARNs (Amazon Resource Name) and latest versions for each AWS region here: https://github.com/signalfx/lambda-layer-versions/blob/main/splunk-apm/splunk-apm.md

You should also see a section where the Environment variables that are being set.

environment:
  AWS_LAMBDA_EXEC_WRAPPER: /opt/nodejs-otel-handler
  OTEL_RESOURCE_ATTRIBUTES: deployment.environment=${self:custom.prefix}-apm-lambda
  OTEL_SERVICE_NAME: consumer-lambda
  SPLUNK_ACCESS_TOKEN: ${self:custom.accessToken}
  SPLUNK_REALM: ${self:custom.realm}

Using the environment variables we are configuring and enriching our auto-instrumentation.

Here we provide minimum information, such as NodeJS wrapper location in the Splunk APM Layer, environment name, service name, and our Splunk Org credentials. We are sending trace data directly to Splunk Observability Cloud. You could alternatively export traces to an OpenTelemetry Collector set up in Gateway mode.

Take a look at the function code.

cat handler.js

Notice there is no mention of Splunk or OpenTelemetry in the code. We are adding the instrumentation using the Lambda layer and Environment Variables only.

Deploy your Lambdas

Run the following command to deploy your Lambda Functions:

sls deploy

Deploying hostname-lambda-lab to stage dev (us-east-1)
...
...
endpoint: POST - https://randomstring.execute-api.us-east-1.amazonaws.com/dev/producer
functions:
  producer: hostname-lambda-lab-dev-producer (1.6 kB)
  consumer: hostname-lambda-lab-dev-consumer (1.6 kB)

This command will follow the instructions in your serverless.yml template to create your Lambda functions and your Kinesis stream. Note it may take a 1-2 minutes to execute.

Check the details of your serverless functions:

sls info

Take note of your endpoint value:

Send some Traffic

Use the curl command to send a payload to your producer function. Note the command option -d is followed by your message payload.

Try changing the value of name to your name and telling the Lambda function about your superpower. Replace YOUR_ENDPOINT with the endpoint from your previous step.

curl -d '{ "name": "CHANGE_ME", "superpower": "CHANGE_ME" }' YOUR_ENDPOINT

For example:

curl -d '{ "name": "Kate", "superpower": "Distributed Tracing" }' https://xvq043lj45.execute-api.us-east-1.amazonaws.com/dev/producer

You should see the following output if your message is successful:

{"message":"Message placed in the Event Stream: hostname-eventSteam"}

If unsuccessful, you will see:

{"message": "Internal server error"}

If this occurs, ask one of the lab facilitators for assistance.

If you see a success message, generate more load: re-send that messate 5+ times. You should keep seeing a success message after each send.

Check the lambda logs output:

Producer function logs:

sls logs -f producer

Consumer function logs:

sls logs -f consumer

Examine the logs carefully.