Update Pipeline and Visualize Metrics

Context Matters

In the previous section, you reviewed the raw Kubernetes audit logs and created an Ingest Processor Pipeline to convert them to metrics and send those metrics to Splunk Observability Cloud.

Now that this pipeline is defined we are collecting the new metrics in Splunk Observability Cloud. This is a great start; however, you will only see a single metric showing the total number of Kubernetes audit events for a given time period. It would be much more valuable to add dimensions so that you can split the metric by the event type, user, response status, and so on.

In this section you will update the Ingest Processor Pipeline to include additional dimensions from the Kubernetes audit logs to the metrics that are being collected. This will allow you to further filter, group, visualize, and alert on specific aspects of the audit logs. After updating the metric, you will create a new dashboard showing the status of the different types of actions associated with the logs.

Update Ingest Pipeline

Exercise: Update Ingest Pipeline

1. Navigate back to the configuration page for the Ingest Pipeline you created in the previous step.

2. To add dimensions to the metric from the raw Kubernetes audit logs update the SPL2 query you created for the pipeline by replacing the logs_to_metrics portion of the query with the following:

Note

Be sure to update the metric name field (name="k8s_audit_UNIQUE_FIELD") to the name you provided in the original pipeline

| logs_to_metrics name="k8s_audit_UNIQUE_FIELD" metrictype="counter" value=1 time=_time dimensions={"level": _raw.level, "response_status": _raw.responseStatus.code, "namespace": _raw.objectRef.namespace, "resource": _raw.objectRef.resource, "user": _raw.user.username, "action": _raw.verb}

Note

Using the dimensions field in the SPL2 query you can add dimensions from the raw events to the metrics that will be sent to Splunk Observability Cloud. In this case you are adding the event response status, namespace, Kubernetes resource, user, and verb (action that was performed). These dimensions can be used to create more granular dashboards and alerts.

You should consider adding any common tags across your services so that you can take advantage of context propagation and related content in Splunk Observability Cloud.

The updated pipeline should now be the following:

/*A valid SPL2 statement for a pipeline must start with "$pipeline", and include "from $source" and "into $destination".*/
/* Import logs_to_metrics */
import logs_to_metrics from /splunk/ingest/commands
$pipeline =
| from $source
| thru [
        //define the metric name, type, and value for the Kubernetes Events
        //
        // REPLACE UNIQUE_FIELD WITH YOUR INITIALS
        //
        | logs_to_metrics name="k8s_audit_UNIQUE_FIELD" metrictype="counter" value=1 time=_time dimensions={"level": _raw.level, "response_status": _raw.responseStatus.code, "namespace": _raw.objectRef.namespace, "resource": _raw.objectRef.resource, "user": _raw.user.username, "action": _raw.verb}
        | into $metrics_destination
    ]
| eval index = "kube_logs"
| into $destination;

3. In the upper-right corner click the Preview button or press CTRL+Enter (CMD+Enter on Mac). From the Previewing $pipeline dropdown select $metrics_destination. Confirm you are seeing a preview of the metrics that will be sent to Splunk Observability Cloud.

4. Confirm you are seeing the dimensions in the dimensions column of the preview table. You can view the entire dimensions object by clicking into the table.

5. In the upper-right corner click the Save pipeline button . On the “You are editing an active pipeline modal” click Save.

Note

Because this pipeline is already active, the changes you made will take effect immediately. Your metric should now be split into multiple metric timeseries using the dimensions you added.

In the next step you will create a visualization using different dimensions from the Kubernetes audit events.

Visualize Kubernetes Audit Event Metrics

Now that your metric has dimensions you will create a chart showing the health of different Kubernetes actions using the verb dimension from the events.

Exercise: Visualize Kubernetes Audit Event Metrics

1. If you closed the chart you created in the previous section, in the upper-right corner, click the + Icon → Chart to create a new chart.

2. In the Plot Editor of the newly created chart enter k8s_audit* in the metric name field. You will use a wildcard here so that you can see all the metrics that are being ingested.

3. Notice the change from one to many metrics, which is when you updated the pipeline to include the dimensions. Now that we have this metric available, let’s adjust the chart to show us if any of our actions have errors associated with them.

First you’ll filter the Kubernetes events to only those that were not successful using the HTTP response code which is available in the response_status field. We only want events that have a response code of 409, which indicates that there was a conflict (for example a trying to create a resource that already exists) or 503, which indicates that the API was unresponsive for the request.

4. In the plot editor of your chart click the Add filter, use response_status for the field and select 409.0 and 503.0 for the values.

Next, you’ll add a function to the chart which will calculate the total number of events grouped by the resource, action, and response status. This will allow us to see exactly which actions and the associated resources had errors. Now we are only looking at Kubernetes events that were not successful.

5. Click Add analytics → Sum → Sum:Aggregation and add resource, action, and response_status in the Group by field.

6. Using the chart type along the top buttons, change the chart to a heatmap. Next to the Plot editor, click Chart options. In the Group by section select response_status then action. Change the Color threshold from Auto to Fixed. Click the blue + button to add another threshold. Change the Down arrow to Yellow, the Middle to orange. Leave the Up arrow as red. Enter 5 for the middle threshold and 20 for the upper threshold.

7. In the upper right corner of the chart click the blue Save as… button. Enter a name for your chart (For Example: Kubernetes Audit Logs - Conflicts and Failures).

8. On the Choose a dashboard select New dashboard.

9. Enter a name for your dashboard that includes your initials, so you can easily find it later. Click Save.

10. Make sure the new dashboard you just created is selected and click Ok.

You should now be taken to your new Kubernetes Audit Events dashboard with the chart you created. You can add new charts from other metrics in your environment, such as application errors and response times from the applications running in the Kubernetes cluster, or other Kubernetes metrics such as pod phase, pod memory utilization, etc. giving you a correlated view of your Kubernetes environment from cluster events to application health.

Make a copy of this chart using the three dots ... in the top right of your chart’s visualization box

Paste into the same dashboard you’ve been working in using the + icon in the top right of the UI

Click into your pasted chart and change the visualization to a Column chart.

Change SUM to just resource, namespace (our filters filter down to just problem codes)

In Chart options change title to Kubernetes Audit Logs - Conflicts by Namespace

Click Save and close

Exercise: Create a detector based on Kubernetes Audit Logs

On your Conflicts by Namespace chart click the little bell icon and New detector from chart

Choose a name and click Create alert rule

For Alert condition click Static Threshold and click Proceed to Alert Settings

Enter a Threshold of 20

We wont choose any recipients for this alert so click into Activate and choose Activate Alert Rule and Save

Click Save one final time in the top right to save your detector

Navigate back to your dashboard and you will see a detector associated with the chart denoted by a lit up bell icon on the chart

Exercise: Visualize your time series data in Splunk Cloud - Dashboard Studio

Now that we have our time series metrics ingested to the Splunk Observability Cloud data store we can easily visualize these time series metrics in Splunk Cloud!

In your Splunk Cloud instance browse to Dashboards and select Create New Dashboard

Choose a Dashboard title, permissions and Dashboard Studio along with any Layout Mode. Click Create.

In Dashboard Studio click the chart icon and choose Column

In Select data source choose Create splunk observability cloud metric search

Choose a name for your new datasource and click the Content Import link under Search for metric or metadata

Copy and paste the URL for your chart into the Content URL field

Click Import

Size your chart to your dashboard

Expand Interactions in the right side of your charts Configuration and click Add Interaction

Copy the URL from your dashboard in Splunk Observability

In On click choose Link to custom URL and add the URL for your dashboard in Splunk Observability Cloud for easy navigation back to the source data. Also choose Open in new tab for friendly navigation.

Click Save in the top right to save your Dashboard.

Highlight and click a Column or name in your chart

You will be told you are navigating back to Splunk Observability. Click Continue

You’ve now navigated back to your corresponding Splunk Observability dashboard from Splunk Cloud.

Update Pipeline and Visualize Metrics

Context Matters

Subsections of 4. Update Pipeline and Visualize Metrics

Update Ingest Pipeline

Visualize Kubernetes Audit Event Metrics