Logs

20 minutes

Persona

Remaining in your back-end developer role, you need to inspect the logs from your application to determine the root cause of the issue.

Using the content related to the APM trace (logs) we will now use Logs to drill down further to understand exactly what the problem is. Related Content is a powerful feature that allows you to jump from one component to another and is available for metrics, traces and logs.

1. Introduction to Logs

You’ve now navigated directly from an APM trace into Logs using the Related Content link. Logs is Splunk Observability Cloud’s no-code interface for exploring and analyzing log data.

The key advantage, just as with the RUM and APM integration, is that you’re viewing your logs in the context of your previous actions. In this case, that context includes the matching time range (1) from the trace and a filter (2) automatically applied to the trace_id.

This view will include all the log lines from all services that participated in the back-end transaction started by the end-user interaction with the Online Boutique.

Even in a small application such as our Online Boutique, the sheer amount of logs found can make it hard to see the specific log lines that matter to the actual incident we are investigating.

Before we go any further, let’s quickly recap what we have done so far and why we are here based on the 3 pillars of Observability:

Metrics	Traces	Logs
Do I have a problem?	Where is the problem?	What is the problem?

Using RUM metrics we identified we have a problem with our application. This was obvious from the duration metrics for the user sessions.
Using traces and span tags we found where the problem is. The paymentservice comprises two versions, v350.9 and v350.10, and the error rate was 100% for v350.10.
We did see that this error from the paymentservice v350.10 caused multiple retries and a long delay in the response back from the Online Boutique checkout.
From the trace, using the power of Related Content, we arrived at the log entries for the failing paymentservice version. Now, we can determine what the problem is.

2. Log Filtering

Exercise

We need to focus on just the Error messages in the logs:
Click on the Group By (1) drop-down box and use the filter to find severity.
Once selected click the Apply button (notice that the chart legend changes to show debug, error and info).

Selecting just the error logs can be done by either clicking on the word error (2) in the legend, followed by selecting Add to filter. Then click Run Search at the top of the page.

Next, we will look at log entries in detail.

3. Viewing Log Entries

Exercise

Click on an error entry in the log table (make sure it says hostname: "paymentservice-xxxx" in case there is a rare error from a different service in the list too).

Based on the message, what would you tell the development team to do to resolve the issue?

The development team needs to rebuild and deploy the container with a valid API Token or rollback to v350.9.

Click on the X in the log message pane to close it.

Congratulations

You have successfully used Splunk Observability Cloud to understand why you experienced a poor user experience whilst shopping at the Online Boutique. You used RUM, APM and logs to understand what happened in your service landscape and subsequently, found the underlying cause, all based on the 3 pillars of Observability, metrics, traces and logs.

You also learned how to use Splunk’s intelligent tagging and analysis with Tag Spotlight to detect patterns in your applications’ behavior and to use the full stack correlation power of Related Content to quickly move between the different components whilst keeping in context of the issue.

In the next part of the workshop, we will move from problem-finding mode into mitigation, prevention and process improvement mode.

Logs

Subsections of 5. Logs

1. Introduction to Logs

2. Log Filtering

3. Viewing Log Entries