How to connect to your workshop environment

1. How to retrieve the IP address of the AWS/EC2 instance assigned to you.
2. Connect to your instance using SSH, Putty¹ or your web browser.
3. Verify your connection to your AWS/EC2 cloud instance.
4. Using Putty (Optional)
5. Using Multipass (Optional)

1. AWS/EC2 IP Address

In preparation for the workshop, Splunk has prepared an Ubuntu Linux instance in AWS/EC2.

To get access to the instance that you will be using in the workshop please visit the URL to access the Google Sheet provided by the workshop leader.

Search for your AWS/EC2 instance by looking for your first and last name, as provided during registration for this workshop.

Find your allocated IP address, SSH command (for Mac OS, Linux and the latest Windows versions) and password to enable you to connect to your workshop instance.

It also has the Browser Access URL that you can use in case you cannot connect via ssh or Putty - see EC2 access via Web browser

2. SSH (Mac OS/Linux)

Most attendees will be able to connect to the workshop by using SSH from their Mac or Linux device, or on Windows 10 and above.

To use SSH, open a terminal on your system and type ssh splunk@x.x.x.x (replacing x.x.x.x with the IP address found in Step #1).

When prompted Are you sure you want to continue connecting (yes/no/[fingerprint])? please type yes.

Enter the password provided in the Google Sheet from Step #1.

Upon successful login, you will be presented with the Splunk logo and the Linux prompt.

3. SSH (Windows 10 and above)

The procedure described above is the same on Windows 10, and the commands can be executed either in the Windows Command Prompt or PowerShell. However, Windows regards its SSH Client as an “optional feature”, which might need to be enabled.

You can verify if SSH is enabled by simply executing ssh

If you are shown a help text on how to use the ssh-command (like shown on the screenshot below), you are all set.

If the result of executing the command looks something like the screenshot below, you want to enable the “OpenSSH Client” feature manually.

To do that, open the “Settings” menu, and click on “Apps”. While in the “Apps & features” section, click on “Optional features”.

Here, you are presented with a list of installed features. On the top, you see a button with a plus icon to “Add a feature”. Click it. In the search input field, type “OpenSSH”, and find a feature called “OpenSSH Client”, or respectively, “OpenSSH Client (Beta)”, click on it, and click the “Install”-button.

Now you are set! In case you are not able to access the provided instance despite enabling the OpenSSH feature, please do not shy away from reaching out to the course instructor, either via chat or directly.

At this point you are ready to continue and start the workshop

4. Putty (For Windows Versions prior to Windows 10)

If you do not have SSH pre-installed or if you are on a Windows system, the best option is to install putty, you can find here.

Open Putty and enter in the Host Name (or IP address) field the IP address provided in the Google Sheet.

You can optionally save your settings by providing a name and pressing Save.

To then login to your instance click on the Open button as shown above.

If this is the first time connecting to your AWS/EC2 workshop instance, you will be presented with a security dialogue, please click Yes.

Once connected, login in as splunk and the password is the one provided in the Google Sheet.

Once you are connected successfully you should see a screen similar to the one below:

At this point, you are ready to continue and start the workshop

5. Web Browser (All)

If you are blocked from using SSH (Port 22) or unable to install Putty you may be able to connect to the workshop instance by using a web browser.

Open your web browser and type http://x.x.x.x:6501 (where X.X.X.X is the IP address from the Google Sheet).

Once connected, login in as splunk and the password is the one provided in the Google Sheet.

Once you are connected successfully you should see a screen similar to the one below:

Unlike when you are using regular SSH, copy and paste does require a few extra steps to complete when using a browser session. This is due to cross browser restrictions.

When the workshop asks you to copy instructions into your terminal, please do the following:

Copy the instruction as normal, but when ready to paste it in the web terminal, choose Paste from browser as show below:

This will open a dialogue box asking for the text to be pasted into the web terminal:

Paste the text in the text box as shown, then press OK to complete the copy and paste process.

At this point you are ready to continue and start the workshop.

6. Multipass (All)

If you are unable to access AWS, but want to install software locally, follow the instructions for using Multipass.

Deploying the OpenTelemetry Collector in Kubernetes

• Use the Splunk Helm chart to install the OpenTelemetry Collector in K3s
• Explore your cluster in the Kubernetes Navigator

1. Installation using Helm

Install the OpenTelemetry Collector using the Splunk Helm chart. First, add the Splunk Helm chart repository to Helm and update:

helm repo add splunk-otel-collector-chart https://signalfx.github.io/splunk-otel-collector-chart && helm repo update

Install the OpenTelemetry Collector Helm chart with the following commands, do NOT edit this:

helm install splunk-otel-collector \
--set="splunkObservability.realm=$REALM" \
--set="splunkObservability.accessToken=$ACCESS_TOKEN" \
--set="clusterName=$INSTANCE-k3s-cluster" \
--set="splunkObservability.logsEnabled=false" \
--set="logsEngine=otel" \
--set="splunkObservability.profilingEnabled=true" \
--set="splunkObservability.infrastructureMonitoringEventsEnabled=true" \
--set="environment=$INSTANCE-workshop" \
--set="splunkPlatform.endpoint=$HEC_URL" \
--set="splunkPlatform.token=$HEC_TOKEN" \
--set="splunkPlatform.index=splunk4rookies-workshop" \
splunk-otel-collector-chart/splunk-otel-collector \
-f ~/workshop/k3s/otel-collector.yaml

Using ACCESS_TOKEN={REDACTED}
Using REALM=eu0
NAME: splunk-otel-collector
LAST DEPLOYED: Fri May  7 11:19:01 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

You can monitor the progress of the deployment by running kubectl get pods which should typically report a new pod is up and running after about 30 seconds.

Ensure the status is reported as Running before continuing.

kubectl get pods

NAME                                                          READY   STATUS    RESTARTS   AGE
splunk-otel-collector-agent-2sk6k                             0/1     Running   0          10s
splunk-otel-collector-k8s-cluster-receiver-6956d4446f-gwnd7   0/1     Running   0          10s

Ensure there are no errors by tailing the logs from the OpenTelemetry Collector pod. The output should look similar to the log output shown in the Output tab below.

Use the label set by the helm install to tail logs (You will need to press ctrl+c to exit). Or use the installed k9s terminal UI for bonus points!

kubectl logs -l app=splunk-otel-collector -f --container otel-collector

helm delete splunk-otel-collector

2. Validate metrics in the UI

In the Splunk UI, click the » bottom left and click on Infrastructure.

Under Containers click on Kubernetes to open the Kubernetes Navigator Cluster Map to ensure metrics are being sent in.

Validate that your cluster is discovered and reported by finding your cluster (in the workshop you will see many other clusters). To find your cluster name run the following command and copy the output to your clipboard:

echo $INSTANCE-k3s-cluster

Then in the UI, click on the “Cluster: - " menu just below the Splunk Logo, paste the Cluster name you just copied into the search box, click the box to select your cluster, and finally click off the menu into white space to apply the filter.

To examine the health of your node, hover over the pale blue background of your cluster, then click on the blue magnifying glass that appears in the top left-hand corner.

This will drill down to the node level. Next, open the Metrics sidebar by clicking on the sidebar button.

Once it is open, you can use the slider on the side to explore the various charts relevant to your cluster/node: CPU, Memory, Network, Events etc.

Working with Dashboards

• Introduction to the Dashboards and Charts
• Editing and creating charts
• Filtering and analytical functions
• Using formulas
• Saving charts in a dashboard
• Introduction to SignalFlow

1. Dashboards

Dashboards are groupings of charts and visualizations of metrics. Well-designed dashboards can provide useful and actionable insight into your system at a glance. Dashboards can be complex or contain just a few charts that drill down only into the data you want to see.

During this module, we are going to create the following charts and dashboard and connect them to your Team page.

2. Your Teams’ Page

Click on the from the navbar. As you have already been assigned to a team, you will land on the team dashboard. We use the Example Team as an example here. The one in your workshop will be different!

This page shows the total number of team members, how many active alerts for your team and all dashboards that are assigned to your team. Right now there are no dashboards assigned but as stated before, we will add the new dashboard that you will create to your Teams page later.

3. Sample Charts

To continue, click on All Dashboards in the top right corner of the screen. This brings you to the view that shows all the available dashboards, including the pre-built ones.

If you are already receiving metrics from a Cloud API integration or another service through the Splunk Agent you will see relevant dashboards for these services.

4. Inspecting the Sample Data

Among the dashboards, you will see a Dashboard group called Sample Data. Expand the Sample Data dashboard group by clicking on it, and then click on the Sample Charts dashboard.

In the Sample Charts dashboard, you can see a selection of charts that show a sample of the various styles, colors and formats you can apply to your charts in the dashboards.

Have a look through all the dashboards in this dashboard group (PART 1, PART 2, PART 3 and INTRO TO SPLUNK OBSERVABILITY CLOUD)

Working with Detectors

• Create a Detector from one of your charts
• Setting Alert conditions
• Running a pre-flight check
• Working with muting rules

1. Introduction

Splunk Observability Cloud uses detectors, events, alerts, and notifications to keep you informed when certain criteria are met. For example, you might want a message sent to a Slack channel or an email address for the Ops team when CPU Utilization has reached 95%, or when the number of concurrent users is approaching a limit that might require you to spin up an additional AWS instance.

These conditions are expressed as one or more rules that trigger an alert when the conditions in the rules are met. Individual rules in a detector are labeled according to criticality: Info, Warning, Minor, Major, and Critical.

2. Creating a Detector

In Dashboards click on your Custom Dashboard Group (that you created in the previous module) and then click on the dashboard name.

We are now going to create a new detector from a chart on this dashboard. Click on the bell icon on the Latency vs Load chart, and then click New Detector From Chart.

In the text field next to Detector Name, ADD YOUR INITIALS before the proposed detector name.

Click on Create Alert Rule

In the Detector window, inside Alert signal, the Signal we will alert on is marked with a (blue) bell in the Alert on column. The bell indicates which Signal is being used to generate the alert.

Click on Proceed to Alert Condition

3. Setting Alert condition

In Alert condition, click on Static Threshold and then on Proceed to Alert Settings

In Alert Settings, enter the value 290 in the Threshold field. In the same window change Time on top right to past day (-1d).

4. Alert pre-flight check

A pre-flight check will take place after 5 seconds. See the Estimated alert count. Based on the current alert settings, the amount of alerts we would have received in 1 day would have been 3.

Click on Proceed to Alert Message

5. Alert message

In Alert message, under Severity choose Major.

Click on Proceed to Alert Recipients

Click on Add Recipient and then on your email address displayed as the first option.

6. Alert Activation

Click on Proceed to Alert Activation

In Activate… click on Activate Alert Rule

If you want to get alerts quicker you edit the rule and lower the value from 290 to say 280.

If you change the Time to -1h you can see how many alerts you might get with the threshold you have chosen based on the metrics from the last 1 hour.

Click on the in the navbar and then click on Detectors. You can optionally filter for your initials. You will see you detector listed here. If you don’t then please refresh your browser.

Congratulations! You have created your first detector and activated it!

Monitoring as Code

• Use Terraform¹ to manage Observability Cloud Dashboards and Detectors
• Initialize the Terraform Splunk Provider².
• Run Terraform to create detectors and dashboards from code using the Splunk Terraform Provider.
• See how Terraform can also delete detectors and dashboards.

1. Initial setup

Monitoring as code adopts the same approach as infrastructure as code. You can manage monitoring the same way you do applications, servers, or other infrastructure components.

You can use monitoring as code to build out your visualizations, what to monitor, and when to alert, among other things. This means your monitoring setup, processes, and rules can be versioned, shared, and reused.

Full documentation for the Splunk Terraform Provider is available here.

Remaining in your AWS/EC2 instance, change into the o11y-cloud-jumpstart directory

cd ~/observability-content-contrib/integration-examples/terraform-jumpstart

Initialize Terraform and upgrade to the latest version of the Splunk Terraform Provider.

terraform init -upgrade

    Upgrading modules...
    - aws in modules/aws
    - azure in modules/azure
    - docker in modules/docker
    - gcp in modules/gcp
    - host in modules/host
    - kafka in modules/kafka
    - kubernetes in modules/kubernetes
    - parent_child_dashboard in modules/dashboards/parent
    - pivotal in modules/pivotal
    - rum_and_synthetics_dashboard in modules/dashboards/rum_and_synthetics
    - usage_dashboard in modules/dashboards/usage

    Initializing the backend...

    Initializing provider plugins...
    - Finding latest version of splunk-terraform/signalfx...
    - Installing splunk-terraform/signalfx v6.20.0...
    - Installed splunk-terraform/signalfx v6.20.0 (self-signed, key ID CE97B6074989F138)

    Partner and community providers are signed by their developers.
    If you'd like to know more about provider signing, you can read about it here:
    https://www.terraform.io/docs/cli/plugins/signing.html

    Terraform has created a lock file .terraform.lock.hcl to record the provider
    selections it made above. Include this file in your version control repository
    so that Terraform can guarantee to make the same selections by default when
    you run "terraform init" in the future.

    Terraform has been successfully initialized!

    You may now begin working with Terraform. Try running "terraform plan" to see
    any changes that are required for your infrastructure. All Terraform commands
    should now work.

    If you ever set or change modules or backend configuration for Terraform,
    rerun this command to reinitialize your working directory. If you forget, other
    commands will detect it and remind you to do so if necessary.

2. Create execution plan

The terraform plan command creates an execution plan. By default, creating a plan consists of:

• Reading the current state of any already-existing remote objects to make sure that the Terraform state is up-to-date.
• Comparing the current configuration to the prior state and noting any differences.
• Proposing a set of change actions that should, if applied, make the remote objects match the configuration.

The plan command alone will not actually carry out the proposed changes, and so you can use this command to check whether the proposed changes match what you expected before you apply the changes

terraform plan -var="api_token=$API_TOKEN" -var="realm=$REALM" -var="o11y_prefix=[$INSTANCE]"

Plan: 146 to add, 0 to change, 0 to destroy.

If the plan executes successfully, we can go ahead and apply:

3. Apply execution plan

The terraform apply command executes the actions proposed in the Terraform plan above.

The most straightforward way to use terraform apply is to run it without any arguments at all, in which case it will automatically create a new execution plan (as if you had run terraform plan) and then prompt you to provide the API Token, Realm (the prefix defaults to Splunk) and approve the plan, before taking the indicated actions.

Due to this being a workshop it is required that the prefix is to be unique so you need to run the terraform apply below.

terraform apply -var="api_token=$API_TOKEN" -var="realm=$REALM" -var="o11y_prefix=[$INSTANCE]"

Apply complete! Resources: 146 added, 0 changed, 0 destroyed.

Once the apply has been completed, validate that the detectors were created, under the Alerts & Detectors and click on the Detectors tab. They will be prefixed by the instance name. To check the prefix value run:

echo $INSTANCE

You will see a list of the new detectors and you can search for the prefix that was output from above.

3. Destroy all your hard work

The terraform destroy command is a convenient way to destroy all remote objects managed by your Terraform configuration.

While you will typically not want to destroy long-lived objects in a production environment, Terraform is sometimes used to manage ephemeral infrastructure for development purposes, in which case you can use terraform destroy to conveniently clean up all of those temporary objects once you are finished with your work.

Now go and destroy all the Detectors and Dashboards that were previously applied!

terraform destroy -var="api_token=$API_TOKEN" -var="realm=$REALM"

Destroy complete! Resources: 146 destroyed.

Validate all the detectors have been removed by navigating to Alerts → Detectors

Service Bureau

• How to keep track of the usage of Observability Cloud in your organization
• Learn how to keep track of spend by exploring the Subscription Usage interface
• Creating Teams
• Adding notification rules to Teams
• Controlling usage

1. Understanding engagement

To fully understand Observability Cloud engagement inside your organization, click on the » bottom left and select the Settings → Organization Overview, this will provide you with the following dashboards that show you how your Observability Cloud organization is being used:

You will see various dashboards such as Throttling, System Limits, Entitlements & Engagement. The workshop organization you’re using now may have less data to work with as this is cleared down after each workshop.

Take a minute to explore the various dashboards and charts in the Organization Overview of this workshop instance.

2. Subscription Usage

If you want to see what your usage is against your subscription you can select Subscription Usage.

This screen may take a few seconds to load whilst it calculates and pulls in the usage.

3. Understanding usage

You will see a screen similar to the one below that will give you an overview of the current usage, the average usage and your entitlement per category: Hosts, Containers, Custom Metrics and High Resolution Metrics.

For more information about these categories please refer to Monitor Splunk Infrastructure Monitoring subscription usage.

4. Examine usage in detail

The top chart shows you the current subscription levels per category (shown by the red arrows at the top in the screenshot below).

Also, your current usage of the four categories is displayed (shown in the red lines at the bottom of the chart).

In this example, you can see that there are 25 Hosts, 0 Containers, 100 Custom Metrics and 0 High Resolution Metrics.

In the bottom chart, you can see the usage per category for the current period (shown in the drop-down box on the top right of the chart).

The blue line marked Average Usage indicates what Observability Cloud will use to calculate your average usage for the current Subscription Usage Period.

To get a feel for the options you can change the metric displayed by selecting the different options from the Usage Metric drop-down on the left, or change the Subscription Usage Period with the drop-down on the right.

Please take a minute to explore the different time periods & categories and their views.

Finally, the pane on the right shows you information about your Subscription.

Setup

This lab will make a tracing superhero out of you!

In this lab you will learn how a distributed trace is constructed for a small serverless application that runs on AWS Lambda, producing and consuming your message via AWS Kinesis.

Pre-requisites

You should already have the lab content available on your EC2 lab host.

Ensure that this lab’s required folder o11y-lambda-lab is on your home directory:

cd ~ && ls

o11y-lambda-lab

git clone https://github.com/kdroukman/o11y-lambda-lab.git

Set Environment Variables

In your Splunk Observability Cloud Organisation (Org) obtain your Access Token and Realm Values.

Please reset your environment variables from the earlier lab. Take care that for this lab we may be using different names - make sure to match the Environment Variable names below.

export ACCESS_TOKEN=CHANGE_ME \
export REALM=CHANGE_ME \
export PREFIX=$INSTANCE

Update Auto-instrumentation serverless template

Update your auto-instrumentation Serverless template to include new values from the Enviornment variables.

cat ~/o11y-lambda-lab/auto/serverless_unset.yml | envsubst > ~/o11y-lambda-lab/auto/serverless.yml

Examine the output of the updated serverless.yml contents (you may need to scroll up to the relevant section).

cat ~/o11y-lambda-lab/auto/serverless.yml

# USER SET VALUES =====================              
custom: 
  accessToken: <updated to your Access Token>
  realm: <updated to your Realm>
  prefix: <updated to your Hostname>
#====================================== 

Update Manual instrumentation template

Update your manual instrumentation Serverless template to include new values from the Enviornment variables.

cat ~/o11y-lambda-lab/manual/serverless_unset.yml | envsubst > ~/o11y-lambda-lab/manual/serverless.yml

Examine the output of the updated serverless.yml contents (you may need to scroll up to the relevant section).

cat ~/o11y-lambda-lab/manual/serverless.yml

# USER SET VALUES =====================              
custom: 
  accessToken: <updated to your Access Token>
  realm: <updated to your Realm>
  prefix: <updated to your Hostname>
#====================================== 

Set your AWS Credentials

You will be provided with AWS Access Key ID and AWS Secret Access Key values - substitue these values in place of AWS_ACCESS_KEY_ID and AWS_ACCESS_KEY_SECRET in the bellow command:

sls config credentials --provider aws --key AWS_ACCCESS_KEY_ID --secret AWS_ACCESS_KEY_SECRET

This command will create a file ~/.aws/credentials with your AWS Credentials populated.

Note that we are using sls here, which is a Serverless framework for developing and deploying AWS Lambda functions. We will be using this command throughout the lab.

Now you are set up and ready go!

Auto-Instrumentation

Auto-Instrumentation

Navigate to the auto directory that contains auto-instrumentation code.

cd ~/o11y-lambda-lab/auto

Inspect the contents of the files in this directory. Take a look at the serverless.yml template.

cat serverless.yml

You should see the Splunk OpenTelemetry Lambda layer being added to each fuction.

layers:
      - arn:aws:lambda:us-east-1:254067382080:layer:splunk-apm:70

You can see the relevant layer ARNs (Amazon Resource Name) and latest versions for each AWS region here: https://github.com/signalfx/lambda-layer-versions/blob/main/splunk-apm/splunk-apm.md

You should also see a section where the Environment variables that are being set.

environment:
  AWS_LAMBDA_EXEC_WRAPPER: /opt/nodejs-otel-handler
  OTEL_RESOURCE_ATTRIBUTES: deployment.environment=${self:custom.prefix}-apm-lambda
  OTEL_SERVICE_NAME: consumer-lambda
  SPLUNK_ACCESS_TOKEN: ${self:custom.accessToken}
  SPLUNK_REALM: ${self:custom.realm}

Using the environment variables we are configuring and enriching our auto-instrumentation.

Here we provide minimum information, such as NodeJS wrapper location in the Splunk APM Layer, environment name, service name, and our Splunk Org credentials. We are sending trace data directly to Splunk Observability Cloud. You could alternatively export traces to an OpenTelemetry Collector set up in Gateway mode.

Take a look at the function code.

cat handler.js

Notice there is no mention of Splunk or OpenTelemetry in the code. We are adding the instrumentation using the Lambda layer and Environment Variables only.

Deploy your Lambdas

Run the following command to deploy your Lambda Functions:

sls deploy

Deploying hostname-lambda-lab to stage dev (us-east-1)
...
...
endpoint: POST - https://randomstring.execute-api.us-east-1.amazonaws.com/dev/producer
functions:
  producer: hostname-lambda-lab-dev-producer (1.6 kB)
  consumer: hostname-lambda-lab-dev-consumer (1.6 kB)

This command will follow the instructions in your serverless.yml template to create your Lambda functions and your Kinesis stream. Note it may take a 1-2 minutes to execute.

Check the details of your serverless functions:

sls info

Take note of your endpoint value:

Send some Traffic

Use the curl command to send a payload to your producer function. Note the command option -d is followed by your message payload.

Try changing the value of name to your name and telling the Lambda function about your superpower. Replace YOUR_ENDPOINT with the endpoint from your previous step.

curl -d '{ "name": "CHANGE_ME", "superpower": "CHANGE_ME" }' YOUR_ENDPOINT

For example:

curl -d '{ "name": "Kate", "superpower": "Distributed Tracing" }' https://xvq043lj45.execute-api.us-east-1.amazonaws.com/dev/producer

You should see the following output if your message is successful:

{"message":"Message placed in the Event Stream: hostname-eventSteam"}

If unsuccessful, you will see:

{"message": "Internal server error"}

If this occurs, ask one of the lab facilitators for assistance.

If you see a success message, generate more load: re-send that messate 5+ times. You should keep seeing a success message after each send.

Check the lambda logs output:

Producer function logs:

sls logs -f producer

Consumer function logs:

sls logs -f consumer

Examine the logs carefully.

Lambdas in Splunk APM

Lambdas in Splunk APM

Now it’s time to check how your Lambda traffic has been captured in Splunk APM.

Navigate to your Splunk Observability Cloud

Select APM from the Main Menu and then select your APM Environment. Your APM environment should be in the format $INSTANCE-apm-lambda where the hostname value is a four letter name of your lab host. (Check it by looking at your command prompt, or by running echo $INSTANCE).

Go to Explore the Service Map to see the Dependencies between your Lambda Functions.

You should be able to see the producer-lambda and the call it is making to Kinesis service.

Click into Traces and examine some traces that container procuder function calls and traces with consumer function calls.

We can see the producer-lambda putting a Record on the Kinesis stream. But the action of consumer-function is disconnected!

This is because the Trace Context is not being propagated.

This is not something that is supported automatically Out-of-the-Box by Kinesis service at the time of this lab. Our Distributed Trace stops at Kinesis inferred service, and we can’t see the propagation any further.

Not yet…

Let’s see how we work around this in the next section of this lab.

Manual Instrumentation

Manual Instrumentation

Navigate to the manual directory that contains manually instrumentated code.

cd ~/o11y-lambda-lab/manual

Inspect the contents of the files in this directory. Take a look at the serverless.yml template.

cat serverless.yml

You can try to compare them with a diff command:

diff ~/o11y-lambda-lab/auto/serverless.yml ~/o11y-lambda-lab/manual/serverless.yml 

19c19
< #======================================    
---
> #======================================   

There is no difference! (Well, there shouldn’t be. Ask your lab facilitator to assist you if there is)

Now compare handler.js it with the same file in auto directory using the diff command:

diff ~/o11y-lambda-lab/auto/handler.js ~/o11y-lambda-lab/manual/handler.js 

Look at all these differences!

You may wish to view the entire file with cat handler.js command and examine its content.

Notice how we are now importing some OpenTelemetry libraries directly into our function to handle some of the manual instrumenation tasks we require.

const otelapi  = require('@opentelemetry/api');
const otelcore = require('@opentelemetry/core');

We are using https://www.npmjs.com/package/@opentelemetry/api to manipulate the tracing logic in our functions. We are using https://www.npmjs.com/package/@opentelemetry/core to access the Propagator objects that we will use to manually propagate our context with.

Inject Trace Context in Producer Function

The below code executes the following steps inside the Producer function:

1. Get the current Active Span.
2. Create a Propagator.
3. Initialize a context carrier object.
4. Inject the context of the active span into the carrier object.
5. Modify the record we are about to put on our Kinesis stream to include the carrier that will carry the active span’s context to the consumer.

const activeSpan = otelapi.trace.getSpan(otelapi.context.active());
const propagator = new otelcore.W3CTraceContextPropagator();
let carrier = {};
propagator.inject(otelapi.trace.setSpanContext(otelapi.ROOT_CONTEXT, activeSpan.spanContext()),
    carrier,
    otelapi.defaultTextMapSetter
  );
const data = "{\"tracecontext\": " + JSON.stringify(carrier) + ", \"record\":" + event.body + "}";
console.log(`Record with Trace Context added: 
  ${data}`);

Extract Trace Context in Consumer Function

The bellow code executes the following steps inside the Consumer function:

1. Extract the context that we obtained from the Producer into a carrier object.
2. Create a Propagator.
3. Extract the context from the carrier object in Customer function’s parent span context.
4. Start a new span with the parent span context.
5. Bonus: Add extra attributes to your span, including custom ones with the values from your message!
6. Once completed, end the span.

const carrier = JSON.parse( message ).tracecontext;
const propagator = new otelcore.W3CTraceContextPropagator();
const parentContext = propagator.extract(otelapi.ROOT_CONTEXT, carrier, otelapi.defaultTextMapGetter);
const tracer = otelapi.trace.getTracer(process.env.OTEL_SERVICE_NAME);
const span = tracer.startSpan("Kinesis.getRecord", undefined, parentContext);
                         
span.setAttribute("span.kind", "server");
const body = JSON.parse( message ).record;
if (body.name) {
    span.setAttribute("custom.tag.name", body.name);
}
 if (body.superpower) {
    span.setAttribute("custom.tag.superpower", body.superpower);
}
  --- function does some work
 span.end();

Now let’s see the difference this makes.

Redeploy Lambdas

Re-deploy your Lambdas

While remaining in your manual directory, run the following commandd to re-deploy your Lambda Functions:

sls deploy -f producer

Deploying function producer to stage dev (us-east-1)

✔ Function code deployed (6s)
Configuration did not change. Configuration update skipped. (6s)

sls deploy -f consumer

Deploying function consumer to stage dev (us-east-1)

✔ Function code deployed (6s)
Configuration did not change. Configuration update skipped. (6s)

Note that this deployment now only updates the code changes within the function. Our configuration remains the same.

Check the details of your serverless functions:

sls info

You endpoint value should remain the same:

Send some Traffic again

Use the curl command to send a payload to your producer function. Note the command option -d is followed by your message payload.

Try changing the value of name to your name and telling the Lambda function about your superpower. Replace YOUR_ENDPOINT with the endpoint from your previous step.

curl -d '{ "name": "CHANGE_ME", "superpower": "CHANGE_ME" }' YOUR_ENDPOINT

For example:

curl -d '{ "name": "Kate", "superpower": "Distributed Tracing" }' https://xvq043lj45.execute-api.us-east-1.amazonaws.com/dev/producer

You should see the following output if your message is successful:

{"message":"Message placed in the Event Stream: hostname-eventSteam"}

If unsuccessful, you will see:

{"message": "Internal server error"}

If this occurs, ask one of the lab facilitators for assistance.

If you see a success message, generate more load: re-send that messate 5+ times. You should keep seeing a success message after each send.

Check the lambda logs output:

sls logs -f producer

sls logs -f consumer

Examine the logs carefully.

Note that we are logging our Record together with the Trace context that we have added to it. Copy one of the underlined sub-sections of your trace parent context, and save it for later.

Updated Lambdas in Splunk APM

Navigate back to APM in Splunk Observabilty Cloud

Go back to your Service Dependency map.

You should be able to see the consumer-lambda now clearly connected to the producer-lambda.

Remember the value you copied from your producer logs? You can run sls logs -f consumer command again on your EC2 lab host to fetch one.

Take that value, and paste it into trace search:

Click on Go and you should be able to find the logged Trace:

Notice that the Trace ID is something that makes up the trace context that we propagated.

You can read up on the two common propagation standards:

1. W3C: https://www.w3.org/TR/trace-context/#traceparent-header
2. B3: https://github.com/openzipkin/b3-propagation#overall-process

It should be self-explanatory from the Propagator we are creating in the Functions

Expand the consumer-lambda span.

Summary

Before you Go

Please kindly clean up your lab using the following command:

sls remove

Conclusion

Congratuations on finishing the lab. You have seen how we complement auto-instrumentation with manual steps to force Producer function’s context to be sent to Consumer function via a Record put on a Kinesis stream. This allowed us to build the expected Distributed Trace.

You can now build out a Trace manually by linking two different functions together. This is very powerful when your auto-instrumenation, or third-party systems, do not support context propagation out of the box.

Getting Started with O11y GDI - Real Time Enrichment Workshop

Please note to begin the following lab, you must have completed the prework:

• Obtain a Splunk Observability Cloud access key
• Understand cli commands

Follow these steps if using O11y Workshop EC2 instances

1. Verify yelp data files are present

ll /var/appdata/yelp*

2. Export the following variables

export ACCESS_TOKEN=<your-access-token>
export REALM=<your-o11y-cloud-realm>
export clusterName=<your-k8s-cluster>

3. Clone the following repo

cd /home/splunk 
git clone https://github.com/leungsteve/realtime_enrichment.git 
cd realtime_enrichment/workshop 
python3 -m venv rtapp-workshop 
source rtapp-workshop/bin/activate

Deploy Complex Environments and Capture Metrics

Objective: Learn how to efficiently deploy complex infrastructure components such as Kafka and MongoDB to demonstrate metrics collection with Splunk O11y IM integrations

Duration: 15 Minutes

Scenario

A prospect uses Kafka and MongoDB in their environment. Since there are integrations for these services, you’d like to demonstrate this to the prospect. What is a quick and efficient way to set up a live environment with these services and have metrics collected?

1. Where can I find helm charts?

• Google “myservice helm chart”
• https://artifacthub.io/ (Note: Look for charts from trusted organizations, with high star count and frequent updates)

2. Review Apache Kafka packaged by Bitnami

We will deploy the helm chart with these options enabled:

• replicaCount=3
• metrics.jmx.enabled=true
• metrics.kafka.enabled=true
• deleteTopicEnable=true

3. Review MongoDB(R) packaged by Bitnami

We will deploy the helm chart with these options enabled:

• version 12.1.31
• metrics.enabled=true
• global.namespaceOverride=default
• auth.rootUser=root
• auth.rootPassword=splunk
• auth.enabled=false

4. Install Kafka and MongoDB with helm charts

helm repo add bitnami https://charts.bitnami.com/bitnami

helm install kafka --set replicaCount=3 --set metrics.jmx.enabled=true --set metrics.kafka.enabled=true  --set deleteTopicEnable=true bitnami/kafka

helm install mongodb --set metrics.enabled=true bitnami/mongodb --set global.namespaceOverride=default --set auth.rootUser=root --set auth.rootPassword=splunk --set auth.enabled=false --version 12.1.31

Verify the helm chart installation

helm list

NAME    NAMESPACE   REVISION    UPDATED                                 STATUS      CHART           APP VERSION
kafka   default     1           2022-11-14 11:21:36.328956822 -0800 PST deployed    kafka-19.1.3    3.3.1
mongodb default     1           2022-11-14 11:19:36.507690487 -0800 PST deployed    mongodb-12.1.31 5.0.10

Verify the helm chart installation

kubectl get pods

NAME                              READY   STATUS              RESTARTS   AGE
kafka-exporter-595778d7b4-99ztt   0/1     ContainerCreating   0          17s
mongodb-b7c968dbd-jxvsj           0/2     Pending             0          6s
kafka-1                           0/2     ContainerCreating   0          16s
kafka-2                           0/2     ContainerCreating   0          16s
kafka-zookeeper-0                 0/1     Pending             0          17s
kafka-0                           0/2     Pending             0          17s

Use information for each Helm chart and Splunk O11y Data Setup to generate values.yaml for capturing metrics from Kafka and MongoDB.

• References:
  • Apache Kafka packaged by Bitnami
  • Configure application receivers for databases » Apache Kafka
  • Kafkametricsreceiver

4.1 Example kafka.values.yaml

otelAgent:
config:
    receivers:
    receiver_creator:
        receivers:
        smartagent/kafka:
            rule: type == "pod" && name matches "kafka"
            config:
                    #endpoint: '`endpoint`:5555'
            port: 5555
            type: collectd/kafka
            clusterName: sl-kafka
otelK8sClusterReceiver:
k8sEventsEnabled: true
config:
    receivers:
    kafkametrics:
        brokers: kafka:9092
        protocol_version: 2.0.0
        scrapers:
        - brokers
        - topics
        - consumers
    service:
    pipelines:
        metrics:
        receivers:
                #- prometheus
        - k8s_cluster
        - kafkametrics

4.2 Example mongodb.values.yaml

otelAgent:
    config:
        receivers:
        receiver_creator:
            receivers:
            smartagent/mongodb:
                rule: type == "pod" && name matches "mongo"
                config:
                type: collectd/mongodb
                host: mongodb.default.svc.cluster.local
                port: 27017
                databases: ["admin", "O11y", "local", "config"]
                sendCollectionMetrics: true
                sendCollectionTopMetrics: true

4.3 Example zookeeper.values.yaml

otelAgent:
    config:
        receivers:
        receiver_creator:
            receivers:
            smartagent/zookeeper:
                rule: type == "pod" && name matches "kafka-zookeeper"
                config:
                type: collectd/zookeeper
                host: kafka-zookeeper
                port: 2181

5. Install the Splunk OTEL helm chart

cd /home/splunk/realtime_enrichment/otel_yamls/ 

helm repo add splunk-otel-collector-chart https://signalfx.github.io/splunk-otel-collector-chart

helm repo update

helm install --set provider=' ' --set distro=' ' --set splunkObservability.accessToken=$ACCESS_TOKEN --set clusterName=$clusterName --set splunkObservability.realm=$REALM --set otelCollector.enabled='false' --set splunkObservability.logsEnabled='true' --set gateway.enabled='false' --values kafka.values.yaml --values mongodb.values.yaml --values zookeeper.values.yaml --values alwayson.values.yaml --values k3slogs.yaml --generate-name splunk-otel-collector-chart/splunk-otel-collector

6. Verify installation

Verify that the Kafka, MongoDB and Splunk OTEL Collector helm charts are installed, note that names may differ.

helm list

NAME                                NAMESPACE   REVISION    UPDATED                                 STATUS      CHART                           APP VERSION
kafka                               default     1           2021-12-07 12:48:47.066421971 -0800 PST deployed    kafka-14.4.1                    2.8.1
mongodb                             default     1           2021-12-07 12:49:06.132771625 -0800 PST deployed    mongodb-10.29.2                 4.4.10
splunk-otel-collector-1638910184    default     1           2021-12-07 12:49:45.694013749 -0800 PST deployed    splunk-otel-collector-0.37.1    0.37.1

kubectl get pods

NAME                                                              READY   STATUS    RESTARTS   AGE
kafka-zookeeper-0                                                 1/1     Running   0          18m
kafka-2                                                           2/2     Running   1          18m
mongodb-79cf87987f-gsms8                                          2/2     Running   0          18m
kafka-1                                                           2/2     Running   1          18m
kafka-exporter-7c65fcd646-dvmtv                                   1/1     Running   3          18m
kafka-0                                                           2/2     Running   1          18m
splunk-otel-collector-1638910184-agent-27s5c                      2/2     Running   0          17m
splunk-otel-collector-1638910184-k8s-cluster-receiver-8587qmh9l   1/1     Running   0          17m

7. Verify dashboards

Verify that out of the box dashboards for Kafka, MongoDB and Zookeeper are populated in the Infrastructure Monitor landing page. Drill down into each component to view granular details for each service.

Tip: You can use the filter k8s.cluster.name with your cluster name to find your instance.

• Infrastructure Monitoring Landing page:

• K8 Navigator:

• MongoDB Dashboard:

• Kafka Dashboard:

Code to Kubernetes - Python

Code to Kubernetes - Python

Objective: Understand activities to instrument a python application and run it on Kubernetes.

• Verify the code
• Containerize the app
• Deploy the container in Kubernetes

Note: these steps do not involve Splunk

Duration: 15 Minutes

1. Verify the code - Review service

Navigate to the review directory

cd /home/splunk/realtime_enrichment/flask_apps/review/

Inspect review.py (realtime_enrichment/flask_apps/review)

cat review.py

from flask import Flask, jsonify
import random
import subprocess

review = Flask(__name__)
num_reviews = 8635403
num_reviews = 100000
reviews_file = '/var/appdata/yelp_academic_dataset_review.json'

@review.route('/')
def hello_world():
    return jsonify(message='Hello, you want to hit /get_review. We have ' + str(num_reviews) + ' reviews!')

@review.route('/get_review')
def get_review():
    random_review_int = str(random.randint(1,num_reviews))
    line_num = random_review_int + 'q;d'
    command = ["sed", line_num, reviews_file] # sed "7997242q;d" <file>
    random_review = subprocess.run(command, stdout=subprocess.PIPE, text=True)
    return random_review.stdout

if __name__ == "__main__":
    review.run(host ='0.0.0.0', port = 5000, debug = True)

Inspect requirements.txt

Flask==2.0.2

Create a virtual environment and Install the necessary python packages

cd /home/splunk/realtime_enrichment/workshop/flask_apps_start/review/

pip freeze #note output
pip install -r requirements.txt
pip freeze #note output

Start the REVIEW service. Note: You can stop the app with control+C

python3 review.py

 * Serving Flask app 'review' (lazy loading)
 * Environment: production
         ...snip...
 * Running on http://10.160.145.246:5000/ (Press CTRL+C to quit)
 * Restarting with stat
127.0.0.1 - - [17/May/2022 22:46:38] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [17/May/2022 22:47:02] "GET /get_review HTTP/1.1" 200 -
127.0.0.1 - - [17/May/2022 22:47:58] "GET /get_review HTTP/1.1" 200 -

Verify that the service is working

• Open a new terminal and ssh into your ec2 instance. Then use the curl command in your terminal.

curl http://localhost:5000

• Or hit the URL http://{Your_EC2_IP_address}:5000 and http://{Your_EC2_IP_address}:5000/get_review with a browser

curl localhost:5000
{
  "message": "Hello, you want to hit /get_review. We have 100000 reviews!"
}

curl localhost:5000/get_review
{"review_id":"NjbiESXotcEdsyTc4EM3fg","user_id":"PR9LAM19rCM_HQiEm5OP5w","business_id":"UAtX7xmIfdd1W2Pebf6NWg","stars":3.0,"useful":0,"funny":0,"cool":0,"text":"-If you're into cheap beer (pitcher of bud-light for $7) decent wings and a good time, this is the place for you. Its generally very packed after work hours and weekends. Don't expect cocktails. \n\n-You run into a lot of sketchy characters here sometimes but for the most part if you're chilling with friends its not that bad. \n\n-Friendly bouncer and bartenders.","date":"2016-04-12 20:23:24"}

2. Create a REVIEW container

To create a container image, you need to create a Dockerfile, run docker build to build the image referencing the Docker file and push it up to a remote repository so it can be pulled by other sources.

• Create a Dockerfile
• Creating a Dockerfile typically requires you to consider the following:
  • Identify an appropriate container image
    • ubuntu vs. python vs. alpine/slim
    • ubuntu - overkill, large image size, wasted resources when running in K8
    • this is a python app, so pick an image that is optimized for it
    • avoid alpine for python
  • Order matters
    • you’re building layers.
    • re-use the layers as much as possible
    • have items that change often towards the end
  • Other Best practices for writing Dockerfiles

Dockerfile for review

FROM python:3.10-slim
WORKDIR /app
COPY requirements.txt /app
RUN pip install -r requirements.txt
COPY ./review.py /app
EXPOSE 5000
CMD [ "python", "review.py" ]

Create a container image (locally) Run ‘docker build’ to build a local container image referencing the Dockerfile

docker build -f Dockerfile -t localhost:8000/review:0.01 .

[+] Building 35.5s (11/11) FINISHED
 => [internal] load build definition from Dockerfile                              0.0s
         ...snip...
 => [3/5] COPY requirements.txt /app                                              0.0s
 => [4/5] RUN pip install -r requirements.txt                                     4.6s
 => [5/5] COPY ./review.py /app                                                   0.0s
 => exporting to image                                                            0.2s
 => => exporting layers                                                           0.2s
 => => writing image sha256:61da27081372723363d0425e0ceb34bbad6e483e698c6fe439c5  0.0s
 => => naming to docker.io/localhost:8000/review:0.1                                   0.0

Push the container image into a container repository Run ‘docker push’ to place a copy of the REVIEW container to a remote location

docker push localhost:8000/review:0.01

The push refers to repository [docker.io/localhost:8000/review]
02c36dfb4867: Pushed
         ...snip...
fd95118eade9: Pushed
0.1: digest: sha256:3651f740abe5635af95d07acd6bcf814e4d025fcc1d9e4af9dee023a9b286f38 size: 2202

Verify that the image is in Docker Hub. The same info can be found in Docker Desktop

curl -s http://localhost:8000/v2/_catalog

{"repositories":["review"]}

3. Run REVIEW in Kubernetes

Create K8 deployment yaml file for the REVIEW app

Reference: Creating a Deployment

review.deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: review
  labels:
    app: review
spec:
  replicas: 1
  selector:
    matchLabels:
      app: review
  template:
    metadata:
      labels:
        app: review
    spec:
      imagePullSecrets:
      - name: regcred
      containers:
      - image: localhost:8000/review:0.01
        name: review
        volumeMounts:
        - mountPath: /var/appdata
          name: appdata
      volumes:
      - name: appdata
        hostPath:
          path: /var/appdata

Notes regarding review.deployment.yaml:

• labels - K8 uses labels and selectors to tag and identify resources
  • In the next step, we’ll create a service and associate it to this deployment using the label
• replicas = 1
  • K8 allows you to scale your deployments horizontally
  • We’ll leverage this later to add load and increase our ingestion rate
• regcred provides this deployment with the ability to access your dockerhub credentials which is necessary to pull the container image.
• The volume definition and volumemount make the yelp dataset visible to the container

Create a K8 service yaml file for the review app.

Reference: Creating a service:

review.service.yaml

apiVersion: v1
kind: Service
metadata:
  name: review
spec:
  type: NodePort
  selector:
    app: review
  ports:
    - port: 5000
      targetPort: 5000
      nodePort: 30000

Notes about review.service.yaml:

• the selector associates this service to pods with the label app with the value being review
• the review service exposes the review pods as a network service
  • other pods can now ping ‘review’ and they will hit a review pod.
  • a pod would get a review if it ran curl http://review:5000
• NodePort service
  • the service is accessible to the K8 host by the nodePort, 30000
  • Another machine that has this can get a review if it ran curl http://<k8 host ip>:30000

Apply the review deployment and service

kubectl apply -f review.service.yaml -f review.deployment.yaml

Verify that the deployment and services are running:

kubectl get deployments

NAME                                                    READY   UP-TO-DATE   AVAILABLE   AGE
review                                                  1/1     1            1           19h

kubectl get services

NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
review                     NodePort    10.43.175.21    <none>        5000:30000/TCP                  154d

curl localhost:30000

{
  "message": "Hello, you want to hit /get_review. We have 100000 reviews!"
}

curl localhost:30000/get_review

{"review_id":"Vv9rHtfBrFc-1M1DHRKN9Q","user_id":"EaNqIwKkM7p1bkraKotqrg","business_id":"TA1KUSCu8GkWP9w0rmElxw","stars":3.0,"useful":1,"funny":0,"cool":0,"text":"This is the first time I've actually written a review for Flip, but I've probably been here about 10 times.  \n\nThis used to be where I would take out of town guests who wanted a good, casual, and relatively inexpensive meal.  \n\nI hadn't been for a while, so after a long day in midtown, we decided to head to Flip.  \n\nWe had the fried pickles, onion rings, the gyro burger, their special burger, and split a nutella milkshake.  I have tasted all of the items we ordered previously (with the exception of the special) and have been blown away with how good they were.  My guy had the special which was definitely good, so no complaints there.  The onion rings and the fried pickles were greasier than expected.  Though I've thought they were delicious in the past, I probably wouldn't order either again.  The gyro burger was good, but I could have used a little more sauce.  It almost tasted like all of the ingredients didn't entirely fit together.  Something was definitely off. It was a friday night and they weren't insanely busy, so I'm not sure I would attribute it to the staff not being on their A game...\n\nDon't get me wrong.  Flip is still good.  The wait staff is still amazingly good looking.  They still make delicious milk shakes.  It's just not as amazing as it once was, which really is a little sad.","date":"2010-10-11 18:18:35"}

Instrument REVIEWS for Tracing

1. Use Data Setup to instrument a Python application

Within the O11y Cloud UI:

Data Management -> Add Integration -> Monitor Applications -> Python (traces) -> Add Integration

Provide the following to the Configure Integration Wizard:

• Service: review

• Django: no

• collector endpoint: http://localhost:4317

• Environment: rtapp-workshop-[YOURNAME]

• Kubernetes: yes

• Legacy Agent: no

We are instructed to:

• Install the instrumentation packages for your Python environment.

pip install splunk-opentelemetry[all]
  
splunk-py-trace-bootstrap

• Configure the Downward API to expose environment variables to Kubernetes resources.

  For example, update a Deployment to inject environment variables by adding .spec.template.spec.containers.env like:

apiVersion: apps/v1
kind: Deployment
spec:
  selector:
    matchLabels:
      app: your-application
  template:
    spec:
      containers:
        - name: myapp
          env:
            - name: SPLUNK_OTEL_AGENT
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://$(SPLUNK_OTEL_AGENT):4317"
            - name: OTEL_SERVICE_NAME
              value: "review"
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: "deployment.environment=rtapp-workshop-stevel"

• Enable the Splunk OTel Python agent by editing your Python service command.

  splunk-py-trace python3 main.py --port=8000

• The actions we must perform include:

  • Update the Dockerfile to install the splunk-opentelemetry packages
  • Update the deployment.yaml for each service to include these environment variables which will be used by the pod and container.
  • Update our Dockerfile for REVIEW so that our program is bootstrapped with splunk-py-trace

2. Update the REVIEW container

• Generate a new container image

• Update the Dockerfile for REVIEW (/workshop/flask_apps_finish/review)

  FROM python:3.10-slim
  
  WORKDIR /app
  COPY requirements.txt /app
  RUN pip install -r requirements.txt
  RUN pip install splunk-opentelemetry[all]
  RUN splk-py-trace-bootstrap
  
  COPY ./review.py /app
  
  EXPOSE 5000
  ENTRYPOINT [ "splunk-py-trace" ]
  CMD [ "python", "review.py" ]

• Generate a new container image with docker build in the ‘finished’ directory
• Notice that I have changed the repository name from localhost:8000/review:0.01 to localhost:8000/review-splkotel:0.01

Ensure you are in the correct directory.

pwd
./workshop/flask_apps_finish/review

docker build -f Dockerfile.review -t localhost:8000/review-splkotel:0.01 .

[+] Building 27.1s (12/12) FINISHED
=> [internal] load build definition from Dockerfile                                                        0.0s
=> => transferring dockerfile: 364B                                                                        0.0s
=> [internal] load .dockerignore                                                                           0.0s
=> => transferring context: 2B                                                                             0.0s
=> [internal] load metadata for docker.io/library/python:3.10-slim                                         1.6s
=> [auth] library/python:pull token for registry-1.docker.io                                               0.0s
=> [1/6] FROM docker.io/library/python:3.10-slim@sha256:54956d6c929405ff651516d5ebbc204203a6415c9d2757aad  0.0s
=> [internal] load build context                                                                           0.3s
=> => transferring context: 1.91kB                                                                         0.3s
=> CACHED [2/6] WORKDIR /app                                                                               0.0s
=> [3/6] COPY requirements.txt /app                                                                        0.0s
=> [4/6] RUN pip install -r requirements.txt                                                              15.3s
=> [5/6] RUN splk-py-trace-bootstrap                                                                       9.0s
=> [6/6] COPY ./review.py /app                                                                             0.0s
=> exporting to image                                                                                      0.6s
=> => exporting layers                                                                                     0.6s
=> => writing image sha256:164977dd860a17743b8d68bcc50c691082bd3bfb352d1025dc3a54b15d5f4c4d                0.0s
=> => naming to docker.io/localhost:8000/review-splkotel:0.01                                              0.0s

• Push the image to Docker Hub with docker push command

docker push localhost:8000/review-splkotel:0.01

The push refers to repository [docker.io/localhost:8000/review-splkotel]
682f0e550f2c: Pushed
dd7dfa312442: Pushed
917fd8334695: Pushed
e6782d51030d: Pushed
c6b19a64e528: Mounted from localhost:8000/review
8f52e3bfc0ab: Mounted from localhost:8000/review
f90b85785215: Mounted from localhost:8000/review
d5c0beb90ce6: Mounted from localhost:8000/review
3759be374189: Mounted from localhost:8000/review
fd95118eade9: Mounted from localhost:8000/review
0.01: digest: sha256:3b251059724dbb510ea81424fc25ed03554221e09e90ef965438da33af718a45 size: 2412

3. Update the REVIEW deployment in Kubernetes

• review.deployment.yaml must be updated with the following changes:

  • Load the new container image on Docker Hub
  • Add environment variables so traces can be emitted to the OTEL collector

• The deployment must be replaced using the updated review.deployment.yaml

  • Update review.deployment.yaml (updates highlighted in bold)

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: review
      labels:
        app: review
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: review
      template:
        metadata:
          labels:
            app: review
        spec:
          imagePullSecrets:
            - name: regcred
          containers:
          - image: localhost:8000/review-splkotel:0.01
            name: review
            volumeMounts:
            - mountPath: /var/appdata
              name: appdata
            env:
            - name: SPLUNK_OTEL_AGENT
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: OTEL_SERVICE_NAME
              value: 'review'
            - name: SPLUNK_METRICS_ENDPOINT
              value: "http://$(SPLUNK_OTEL_AGENT):9943"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://$(SPLUNK_OTEL_AGENT):4317"
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: 'deployment.environment=rtapp-workshop-stevel'
          volumes:
          - name: appdata
            hostPath:
              path: /var/appdata

• Apply review.deployment.yaml. Kubernetes will automatically pick up the changes to the deployment and redeploy new pods with these updates

  • Notice that the review-* pod has been restarted

kubectl apply -f review.deployment.yaml

kubectl get pods

NAME                                                              READY   STATUS        RESTARTS   AGE
kafka-client                                                      0/1     Unknown       0          155d
curl                                                              0/1     Unknown       0          155d
kafka-zookeeper-0                                                 1/1     Running       0          26h
kafka-2                                                           2/2     Running       0          26h
kafka-exporter-647bddcbfc-h9gp5                                   1/1     Running       2          26h
mongodb-6f6c78c76-kl4vv                                           2/2     Running       0          26h
kafka-1                                                           2/2     Running       1          26h
kafka-0                                                           2/2     Running       1          26h
splunk-otel-collector-1653114277-agent-n4dfn                      2/2     Running       0          26h
splunk-otel-collector-1653114277-k8s-cluster-receiver-5f48v296j   1/1     Running       0          26h
splunk-otel-collector-1653114277-agent-jqxhh                      2/2     Running       0          26h
review-6686859bd7-4pf5d                                           1/1     Running       0          11s
review-5dd8cfd77b-52jbd                                           0/1     Terminating   0          2d10h

kubectl get pods
NAME                                                              READY   STATUS    RESTARTS   AGE
kafka-client                                                      0/1     Unknown   0          155d
curl                                                              0/1     Unknown   0          155d
kafka-zookeeper-0                                                 1/1     Running   0          26h
kafka-2                                                           2/2     Running   0          26h
kafka-exporter-647bddcbfc-h9gp5                                   1/1     Running   2          26h
mongodb-6f6c78c76-kl4vv                                           2/2     Running   0          26h
kafka-1                                                           2/2     Running   1          26h
kafka-0                                                           2/2     Running   1          26h
splunk-otel-collector-1653114277-agent-n4dfn                      2/2     Running   0          26h
splunk-otel-collector-1653114277-k8s-cluster-receiver-5f48v296j   1/1     Running   0          26h
splunk-otel-collector-1653114277-agent-jqxhh                      2/2     Running   0          26h
review-6686859bd7-4pf5d                                           1/1     Running   0          15s

Monitor System Logs with Splunk Universal Forwarder

Objective: Learn how to monitor Linux system logs with the Universal Forwarder sending logs to Splunk Enterprise

Duration: 10 Minutes

Scenario

You’ve been tasked with monitoring the OS logs of the host running your Kubernetes cluster. We are going to utilize a script that will autodeploy the Splunk Universal Forwarder. You will then configure the Universal Forwarder to send logs to the Splunk Enterprise instance assigned to you.

1. Ensure You’re in the Correct Directory

• we will need to be in /home/splunk/session-2

cd /home/splunk/session-2

2. Review the Universal Forwarder Install Script

• Let’s take a look at the script that will install the Universal Forwarder and Linux TA automatically for you.
  • This script is primarily used for remote instances.
  • Note we are not using a deployment server in this lab, however it is recommended in production we do that.
  • What user are we installing Splunk as?

#!/bin/sh  
# This EXAMPLE script shows how to deploy the Splunk universal forwarder
# to many remote hosts via ssh and common Unix commands.
# For "real" use, this script needs ERROR DETECTION AND LOGGING!!
# --Variables that you must set -----
# Set username using by splunkd to run.
  SPLUNK_RUN_USER="ubuntu"

# Populate this file with a list of hosts that this script should install to,
# with one host per line. This must be specified in the form that should
# be used for the ssh login, ie. username@host
#
# Example file contents:
# splunkuser@10.20.13.4
# splunkker@10.20.13.5
  HOSTS_FILE="myhost.txt"

# This should be a WGET command that was *carefully* copied from splunk.com!!
# Sign into splunk.com and go to the download page, then look for the wget
# link near the top of the page (once you have selected your platform)
# copy and paste your wget command between the ""
  WGET_CMD="wget -O splunkforwarder-9.0.3-dd0128b1f8cd-Linux-x86_64.tgz 'https://download.splunk.com/products/universalforwarder/releases/9.0.3/linux/splunkforwarder-9.0.3-dd0128b1f8cd-Linux-x86_64.tgz'"
# Set the install file name to the name of the file that wget downloads
# (the second argument to wget)
  INSTALL_FILE="splunkforwarder-9.0.3-dd0128b1f8cd-Linux-x86_64.tgz"

# After installation, the forwarder will become a deployment client of this
# host.  Specify the host and management (not web) port of the deployment server
# that will be managing these forwarder instances.
# Example 1.2.3.4:8089
#  DEPLOY_SERVER="x.x.x.x:8089"



# After installation, the forwarder can have additional TA's added to the 
# /app directory please provide the local where TA's will be. 
  TA_INSTALL_DIRECTORY="/home/splunk/session-2"

# Set the seed app folder name for deploymentclien.conf
#  DEPLOY_APP_FOLDER_NAME="seed_all_deploymentclient"
# Set the new Splunk admin password
  PASSWORD="buttercup"

REMOTE_SCRIPT_DEPLOY="
  cd /opt
  sudo $WGET_CMD
  sudo tar xvzf $INSTALL_FILE
  sudo rm $INSTALL_FILE
  #sudo useradd $SPLUNK_RUN_USER
  sudo find $TA_INSTALL_DIRECTORY -name '*.tgz' -exec tar xzvf {} --directory /opt/splunkforwarder/etc/apps \;
  sudo chown -R $SPLUNK_RUN_USER:$SPLUNK_RUN_USER /opt/splunkforwarder
  echo \"[user_info] 
  USERNAME = admin
  PASSWORD = $PASSWORD\" > /opt/splunkforwarder/etc/system/local/user-seed.conf   
  #sudo cp $TA_INSTALL_DIRECTORY/*.tgz /opt/splunkforwader/etc/apps/
  #sudo find /opt/splunkforwarder/etc/apps/ -name '*.tgz' -exec tar xzvf {} \;
  #sudo -u splunk /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --auto-ports --no-prompt
  /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --auto-ports --no-prompt
  #sudo /opt/splunkforwarder/bin/splunk enable boot-start -user $SPLUNK_RUN_USER
  /opt/splunkforwarder/bin/splunk enable boot-start -user $SPLUNK_RUN_USER
  #sudo cp $TA_INSTALL_DIRECTORY/*.tgz /opt/splunkforwarder/etc/apps/

  exit
 "

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"


#===============================================================================================
  echo "In 5 seconds, will run the following script on each remote host:"
  echo
  echo "===================="
  echo "$REMOTE_SCRIPT_DEPLOY"
  echo "===================="
  echo 
  sleep 5
  echo "Reading host logins from $HOSTS_FILE"
  echo
  echo "Starting."
  for DST in `cat "$DIR/$HOSTS_FILE"`; do
    if [ -z "$DST" ]; then
      continue;
    fi
    echo "---------------------------"
    echo "Installing to $DST"
    echo "Initial UF deployment"
    sudo ssh -t "$DST" "$REMOTE_SCRIPT_DEPLOY"
  done  
  echo "---------------------------"
  echo "Done"
  echo "Please use the following app folder name to override deploymentclient.conf options: $DEPLOY_APP_FOLDER_NAME"

3. Run the install script

We will run the install script now. You will see some Warnings at the end. This is totally normal. The script is built for use on remote machines, however for todays lab you will be using localhost.

./install.sh

You will be asked Are you sure you want to continue connecting (yes/no/[fingerprint])? Answer Yes.

Enter your ssh password when prompted.

4. Verify installation of the Universal Forwarader

• We need to verify that the Splunk Universal Forwarder is installed and running.
  • You should see a couple PID’s return and a “Splunk is currently running.” message.

/opt/splunkforwarder/bin/splunk status

5. Configure the Universal Forwarder to Send Data to Splunk Enterprise

• We will be able to send the data to our Splunk Enterprise environment easily by entering one line into the cli.
  • Universal Forwarder Config Guide

/opt/splunkforwarder/bin/splunk add forward-server <your_splunk_enterprise_ip>:9997

6. Verify the Data in Your Splunk Enterprise Environment

• We are now going to take a look at the Splunk Enterprise environment to verify logs are coming in.

  • Logs will be coming into index=main

• Open your web browser and navigate to: http://<your_splunk_enterprise_ip:8000

  • You will use the credentials admin:<your_ssh_password>

• In the search bar, type in the following:

• index=main host=<your_host_name>

• You should see data from your host. Take note of the interesting fields and the different data sources flowing in.

User Profile

Aim

The aim of this module is for you to configure your personal profile which controls how you will be notified by Splunk On-Call whenever you get paged.

1. Contact Methods

Switch to the Splunk On-Call UI and click on your login name in the top right hand corner and chose Profile from the drop down. Confirm your contact methods are listed correctly and add any additional phone numbers and e-mail address you wish to use.

2. Mobile Devices

To install the Splunk On-Call app for your smartphone search your phones App Store for Splunk On-Call to find the appropriate version of the app. The publisher should be listed as VictorOps Inc.

Apple Store

Google Play

Configuration help guides are available:

• Apple
• Android

Install the App and login, then refresh the Profile page and your device should now be listed under the devices section. Click the Test push notification button and confirm you receive the test message.

3. Personal Calendar

This link will enable you to sync your on-call schedule with your calendar, however as you do not have any allocated shifts yet this will currently be empty. You can add it to your calendar by copying the link into your preferred application and setting it up as a new subscription.

4. Paging Policies

Paging Polices specify how you will be contacted when on-call. The Primary Paging Policy will have defaulted to sending you an SMS assuming you added your phone number when activating your account. We will now configure this policy into a three tier multi-stage policy similar to the image below.

4.1 Send a push notification

Click the edit policy button in the top right corner for the Primary Paging Policy.

• Send a push notification to all my devices
• Execute the next step if I have not responded within 5 minutes

Click Add a Step

4.2 Send an e-mail

• Send an e-mail to [your email address]
• Execute the next step if I have not responded within 5 minutes

Click Add a Step

4.3 Call your number

• Every 5 minutes until we have reached you
• Make a phone call to [your phone number]

Click Save to save the policy.

When you are on-call or in the escalation path of an incident, you will receive notifications in this order following these time delays.

To cease the paging you must acknowledge the incident. Acknowledgements can occur in one of the following ways:

• Expanding the Push Notification on your device and selecting Acknowledge
• Responding to the SMS with the 5 digit code included
• Pressing 4 during the Phone Call
• Slack Button

For more information on Notification Types, see here.

5. Custom Paging Policies

Custom paging polices enable you to override the primary policy based on the time and day of the week. A good example would be to get the system to immediately phone you whenever you get a page during the evening or weekends as this is more likely to get your attention than a push notification.

Create a new Custom Policy by clicking Add a Policy and configure with the following settings:

5.1 Custom evening policy

Policy Name: Evening

• Every 5 minutes until we have reached you
  • Make a phone call to [your phone number]
  • Time Period: All 7 Days
  • Time zone
    • Between 7pm and 9am

Click Save to save the policy then add one more.

5.2 Custom weekend policy

Policy Name: Weekend

• Every 5 minutes until we have reached you
  • Make a phone call to [your phone number]
  • Time Period: Sat & Sun
  • Time zone
    • Between 9am and 7pm

Click Save to save the policy.

These custom paging policies will be used during the specified times in place of the Primary Policy. However, admins do have the ability to ignore these custom policies, and we will highlight how this is achieved in a later module.

The final option here is the setting for Recovery Notifications. These are typically low priority, will default to Push, but can also be email, sms or phone call. Your profile is now fully configured using these example configurations.

Organizations will have different views on how profiles should be configured and will typically issue guidelines for paging policies and times between escalations etc.

Please wait for the instructor before proceeding to the Teams module.

Incident Lifecycle

Aim

The aim of this module is for you to get more familiar with the Timeline Tab and the filtering features.

1. Timeline

The aim of Splunk On-Call is to make being on call more bearable, and it does this by getting the critical data, to the right people, at the right time.

The key to making it work for you is to centralize all your alerting sources, sending them all to the Splunk On-Call platform, then you have a single pane of glass in which to manage all of your alerting.

Login to the Splunk On-Call UI and select the Timeline tab on the main menu bar, you should have a screen similar to the following image:

2. People

On the left we have the People section with the Teams and Users sub tabs. On the Teams tab, click on All Teams then expand [Your Team name].

Users with the Splunk On-Call Logo against their name are currently on call. Here you can see who is on call within a particular Team, or across all Teams via Users → On-Call.

If you click into one of the currently on call users, you can see their status. It shows which Rotation they are on call for, when their current Shift ends and their next Shift starts (times are displayed in your time zone), what contact methods they have and which Teams they belong to (dummy users such as Hank do not have Contact Methods configured).

3. Timeline

In the centre Timeline section you get a realtime view of what is happening within your environment with the newest messages at the top. Here you can quickly post update messages to make your colleagues aware of important developments etc.

You can filter the view using the buttons on the top toolbar showing only update messages, GitHub integrations, or apply more advanced filters.

Lets change the Filters settings to streamline your view. Click the Filters button then within the Routing Keys tab change the Show setting from all routing keys to selected routing keys. Change the My Keys value to all and the Other Keys value to selected and deselect all keys under the Other Keys section.

Click anywhere outside of the dialogue box to close it.

You will probably now have a much simpler view as you will not currently have Incidents created using your Routing Keys, so you are left with the other types of messages that the Timeline can display.

Click on Filters again, but this time switch to the Message Types tab. Here you control the types of messages that are displayed.

For example, deselect On-call Changes and Escalations, this will reduce the amount of messages displayed.

4. Incidents

On the right we have the Incidents section. Here we get a list of all the incidents within the platform, or we can view a more specific list such as incidents you are specifically assigned to, or for any of the Teams you are a member of.

Select the Team Incidents tab you should find that the Triggered, Acknowledged & Resolved tabs are currently all empty as you have had no incidents logged.

Let’s change that by generating your first incident!

Continue with the Create Incidents module.