1. Open the Ingest Processor Cloud Stack URL provided in the Splunk Show event details.

2. In the Connection info click on the Stack URL link to open your Splunk Cloud stack.

3. Use the admin username and password to login to Splunk Cloud.

4. After logging in, if prompted, accept the Terms of Service and click OK

5. Navigate back to the Splunk Show event details and select the Ingest Processor SCS Tenant

6. Click on the Console URL to access the Ingest Processor SCS Tenant

1. Open your Ingest Processor Cloud Stack instance using the URL provided in the Splunk Show workshop details.

2. Navigate to Apps → Search and Reporting

3. In the search bar, enter the following SPL search string.

### Replace USER_ID with the User ID provided in your Splunk Show instance information
index=main sourcetype="kube:apiserver:audit:USER_ID"

4. Press Enter or click the green magnifying glass to run the search.

1. Open the Ingest Processor SCS Tenant using the connection details provided in the Splunk Show event.

2. From the Splunk Data Management console select Pipelines → New pipeline → Ingest Processor pipeline.

3. In the Get started step of the Ingest Processor configuration page select Blank Pipeline and click Next.

4. In the Define your pipeline’s partition step of the Ingest Processor configuration page select Partition by sourcetype. Select the = equals Operator and enter kube:apiserver:audit:USER_ID (Be sure to replace USER_ID with the User ID you were assigned) for the value. Click Apply.

5. Click Next

6. In the Add sample data step of the Ingest Processor configuration page select Capture new snapshot. Enter k8s_audit_USER_ID (Be sure to replace USER_ID with the User ID you were assigned) for the Snapshot name and click Capture.

7. Make sure your newly created snapshot (k8s_audit_USER_ID) is selected and then click Next.

8. In the Select a metrics destination step of the Ingest Processor configuration page select show_o11y_org. Click Next.

9. In the Select a data destination step of the Ingest Processor configuration page select splunk_indexer. Under Specify how you want your events to be routed to an index select Default. Click Done.

10. In the Pipeline search field replace the default search with the following.

/*A valid SPL2 statement for a pipeline must start with "$pipeline", and include "from $source" and "into $destination".*/
/* Import logs_to_metrics */
import logs_to_metrics from /splunk/ingest/commands
$pipeline =
| from $source
| thru [
        //define the metric name, type, and value for the Kubernetes Events
        //
        // REPLACE UNIQUE_FIELD WITH YOUR INITIALS
        //
        | logs_to_metrics name="k8s_audit_UNIQUE_FIELD" metrictype="counter" value=1 time=_time
        | into $metrics_destination
    ]
| eval index = "kube_logs"
| into $destination;

11. In the upper-right corner click the Preview button or press CTRL+Enter (CMD+Enter on Mac). From the Previewing $pipeline dropdown select $metrics_destination. Confirm you are seeing a preview of the metrics that will be sent to Splunk Observability Cloud.

12. In the upper-right corner click the Save pipeline button . Enter Kubernetes Audit Logs2Metrics USER_ID for your pipeline name and click Save.

13. After clicking save you will be asked if you would like to apply the newly created pipeline. Click Yes, apply.

1. Login to the Splunk Observability Cloud organization you were invited for the workshop. In the upper-right corner, click the + Icon → Chart to create a new chart.

2. In the Plot Editor of the newly created chart enter the metric name you used while configuring the Ingest Pipeline.

1. Navigate back to the configuration page for the Ingest Pipeline you created in the previous step.

2. To add dimensions to the metric from the raw Kubernetes audit logs update the SPL2 query you created for the pipeline by replacing the logs_to_metrics portion of the query with the following:

| logs_to_metrics name="k8s_audit_UNIQUE_FIELD" metrictype="counter" value=1 time=_time dimensions={"level": _raw.level, "response_status": _raw.responseStatus.code, "namespace": _raw.objectRef.namespace, "resource": _raw.objectRef.resource, "user": _raw.user.username, "action": _raw.verb}

The updated pipeline should now be the following:

/*A valid SPL2 statement for a pipeline must start with "$pipeline", and include "from $source" and "into $destination".*/
/* Import logs_to_metrics */
import logs_to_metrics from /splunk/ingest/commands
$pipeline =
| from $source
| thru [
        //define the metric name, type, and value for the Kubernetes Events
        //
        // REPLACE UNIQUE_FIELD WITH YOUR INITIALS
        //
        | logs_to_metrics name="k8s_audit_UNIQUE_FIELD" metrictype="counter" value=1 time=_time dimensions={"level": _raw.level, "response_status": _raw.responseStatus.code, "namespace": _raw.objectRef.namespace, "resource": _raw.objectRef.resource, "user": _raw.user.username, "action": _raw.verb}
        | into $metrics_destination
    ]
| eval index = "kube_logs"
| into $destination;

3. In the upper-right corner click the Preview button or press CTRL+Enter (CMD+Enter on Mac). From the Previewing $pipeline dropdown select $metrics_destination. Confirm you are seeing a preview of the metrics that will be sent to Splunk Observability Cloud.

4. Confirm you are seeing the dimensions in the dimensions column of the preview table. You can view the entire dimensions object by clicking into the table.

5. In the upper-right corner click the Save pipeline button . On the “You are editing an active pipeline modal” click Save.

1. If you closed the chart you created in the previous section, in the upper-right corner, click the + Icon → Chart to create a new chart.

2. In the Plot Editor of the newly created chart enter k8s_audit* in the metric name field. You will use a wildcard here so that you can see all the metrics that are being ingested.

3. Notice the change from one to many metrics, which is when you updated the pipeline to include the dimensions. Now that we have this metric available, let’s adjust the chart to show us if any of our actions have errors associated with them.

First you’ll filter the Kubernetes events to only those that were not successful using the HTTP response code which is available in the response_status field. We only want events that have a response code of 409, which indicates that there was a conflict (for example a trying to create a resource that already exists) or 503, which indicates that the API was unresponsive for the request.

4. In the plot editor of your chart click the Add filter, use response_status for the field and select 409.0 and 503.0 for the values.

Next, you’ll add a function to the chart which will calculate the total number of events grouped by the resource, action, and response status. This will allow us to see exactly which actions and the associated resources had errors. Now we are only looking at Kubernetes events that were not successful.

5. Click Add analytics → Sum → Sum:Aggregation and add resource, action, and response_status in the Group by field.

6. Using the chart type along the top buttons, change the chart to a heatmap. Next to the Plot editor, click Chart options. In the Group by section select response_status then action. Change the Color threshold from Auto to Fixed. Click the blue + button to add another threshold. Change the Down arrow to Yellow, the Middle to orange. Leave the Up arrow as red. Enter 5 for the middle threshold and 20 for the upper threshold.

7. In the upper right corner of the chart click the blue Save as… button. Enter a name for your chart (For Example: Kubernetes Audit Logs - Conflicts and Failures).

8. On the Choose a dashboard select New dashboard.

9. Enter a name for your dashboard that includes your initials, so you can easily find it later. Click Save.

10. Make sure the new dashboard you just created is selected and click Ok.

You should now be taken to your new Kubernetes Audit Events dashboard with the chart you created. You can add new charts from other metrics in your environment, such as application errors and response times from the applications running in the Kubernetes cluster, or other Kubernetes metrics such as pod phase, pod memory utilization, etc. giving you a correlated view of your Kubernetes environment from cluster events to application health.

Make a copy of this chart using the three dots ... in the top right of your chart’s visualization box

Paste into the same dashboard you’ve been working in using the + icon in the top right of the UI

Click into your pasted chart and change the visualization to a Column chart.

Change SUM to just resource, namespace (our filters filter down to just problem codes)

In Chart options change title to Kubernetes Audit Logs - Conflicts by Namespace

Click Save and close

On your Conflicts by Namespace chart click the little bell icon and New detector from chart

Choose a name and click Create alert rule

For Alert condition click Static Threshold and click Proceed to Alert Settings

Enter a Threshold of 20

We wont choose any recipients for this alert so click into Activate and choose Activate Alert Rule and Save

Click Save one final time in the top right to save your detector

Navigate back to your dashboard and you will see a detector associated with the chart denoted by a lit up bell icon on the chart

Now that we have our time series metrics ingested to the Splunk Observability Cloud data store we can easily visualize these time series metrics in Splunk Cloud!

In your Splunk Cloud instance browse to Dashboards and select Create New Dashboard

Choose a Dashboard title, permissions and Dashboard Studio along with any Layout Mode. Click Create.

In Dashboard Studio click the chart icon and choose Column

In Select data source choose Create splunk observability cloud metric search

Choose a name for your new datasource and click the Content Import link under Search for metric or metadata

Copy and paste the URL for your chart into the Content URL field

Click Import

Size your chart to your dashboard

Expand Interactions in the right side of your charts Configuration and click Add Interaction

Copy the URL from your dashboard in Splunk Observability

In On click choose Link to custom URL and add the URL for your dashboard in Splunk Observability Cloud for easy navigation back to the source data. Also choose Open in new tab for friendly navigation.

Click Save in the top right to save your Dashboard.

Highlight and click a Column or name in your chart

You will be told you are navigating back to Splunk Observability. Click Continue

You’ve now navigated back to your corresponding Splunk Observability dashboard from Splunk Cloud.