• In Splunk Observability Cloud, from the main menu, hover over Digital Experience, then click on Overview (1) from the Real User Monitoring section as shown below.

• This will open up the Application Summary Dashboard. This section shows a quick overview of all the applications being monitored.

• The Real User Monitoring (RUM) Overview dashboard in Splunk Observability Cloud provides visibility into how real users experience your web applications. It captures browser-side performance metrics, JavaScript errors, and network request failures as they occur in actual user sessions. The dashboard surfaces Core Web Vitals (LCP, INP, CLS) to measure page load performance, displays error trends over time, and shows recent alerts, giving frontend teams the insights needed to identify and resolve issues affecting end-user experience.

• To ensure we are looking at the right data, please check the following settings (2):

  • The Time frame is set to -15m.
  • The Environment selected is [NAME OF WORKSHOP]-workshop.
  • The App selected is [NAME OF WORKSHOP]-store.
  • The Source is set to Browser.

• Next, click on the [NAME OF WORKSHOP]-store (3) above the Page Views / JavaScript Errors chart.

• You will now see a dashboard view breaking down the metrics by UX Metrics, Front-end Health, Back-end Health, Custom Events, Network Requests, Pages and a Map View comparing them to historic metrics (1 hour by default).

• The tabs available on this page include:

  • UX Metrics Page Views, Page Load and Web Vitals metrics
  • Front-end Health Breakdown of JavaScript Errors and Long Task duration and count
  • Back-end Health Network Errors and Requests and Time to First Byte
  • Custom Events RED metrics (Rate, Error & Duration) for custom events
  • Network Requests Network URL grouping and key metrics
  • Pages URL grouping and key metrics and web vitals
  • Map View Geographical requests by location

• Click through each of the tabs and examine the data.

• Make sure you are on the Custom Events tab.
• To identify problematic user sessions, we will use the latency spikes in the Custom Event Latency P75 chart.
• In the Custom Event Latency chart click on the see all (1) link under the chart title.

• This dashboard displays all tags associated with the RUM data, key-value pairs automatically generated by the RUM instrumentation. Tags are used to filter data and build charts and tables. The Tag Spotlight view lets you drill down into individual user sessions.

• Change the timeframe to Last 1 hour (1).

• Find the Operation chart, locate PlaceOrder in the list, click on it and select Add to filter (2).

• Click on the User Sessions tab (3).

• Click on the Duration heading twice to sort the sessions by duration (longest at the top) (4).

• We now have a User Session table sorted by longest duration (descending), showing users who have been shopping on the site. We could apply more filters to further narrow down the data, e.g. OS version, browser version, etc.

• In the User Sessions table, click on the Session ID with the longest Duration (over 15 seconds or longer). This will take you to the RUM Session view.
• Note the length of the span PlaceOrder, this is the time it took to complete the order. Not good!

• Look for the Fetch (1) which will be either above or below the PlaceOrder span.
  • It will look something like POST https://labob...y.com/cart/checkout.
• Hover over the blue APM (2), after a few seconds a popup will appear.
• You will see paymentservice and checkoutservice are in an error state as per the screenshot above.
• Under Workflow Name click on front-end:/cart/checkout (3), this will bring up the APM Service Map. Here we will investigate the backend services and their dependencies to identify the root cause of the issue.

• In the APM Service Map you can clearly see there is an issue with the paymentservice.

• Click on the paymentservice in the Service Map to select it.

• Splunk APM also provides built-in Service Centric Views to help you see problems occurring in real time and quickly determine whether the problem is associated with a service, a specific endpoint, or the underlying infrastructure. Let’s have a closer look.
• In the right-hand pane, click on paymentservice in blue (2).

• Check the Time box, you can see that the dashboards only show data relevant to the time it took for the APM trace we previously selected to complete (note that the charts are static).
• In the Time box change the timeframe to -1h (1).

• You can clearly see the Success rate is not 100%, this is because we have errors in our service.
• We need to understand if there is a pattern to this error rate. We have a handy tool for that, click on the Tag Spotlight tab (2).

• Once in Tag Spotlight ensure the toggle Show tags with no values is off.

• This view displays a series of cards, each representing an indexed tag (such as Endpoint, Environment, Version, or custom tags like tenant.level). Within each card, you can see the distribution of tag values along with key metrics including request count, error count, root cause errors, and latency percentiles (P50, P90, P99).

• Now that we have identified the tag that indicates the issue, let’s see if we can find out more information about the error.
• Click the APM link above paymentservice at the top of the page to return to the APM Overview.
• In APM Overview, click on Service Map in the right-hand pane.

• Select the paymentservice in the Service Map.
• In the right-hand pane click on the Breakdown.
• Select version in the list.

• Next, we need to drill down into a trace to see what is going on. Click on the red circle for v350.10 (1) in the paymentservice, then click on the Traces (2) tab in the right-hand pane.

• Find a trace with:
  • an error in the checkoutservice and the paymentservice (1)
  • and an Initiating Operation of frontend: POST /cart/checkout
  • then select the blue Trace ID (2) to continue
• This will open the Trace Waterfall for that specific trace.

• Click on the ! next to any of the paymentservice:grpc.hipstershop.PaymentService/Charge spans in the waterfall.

Now that we have identified the version of the paymentservice that is causing the issue, let’s see if we can find out more information about the error. This is where Related Logs come in.

Related Content relies on specific metadata that allows APM, Infrastructure Monitoring, and Log Observer to pass filters around Observability Cloud. For related logs to work, you need to have the following metadata in your logs:

• service.name

• deployment.environment

• host.name

• trace_id

• span_id

• At the very bottom of the Trace Waterfall click on Logs. This highlights that there are Related Logs for this trace.

• Click on the Logs for trace xxx entry in the pop-up, this will open the logs for the complete trace in Logs.

• Next, let’s find out more about the error in the logs.

• We need to focus on just the Error messages in the logs:
• Click on the Group By (1) drop-down box and use the filter to find severity.
• Once selected click the Apply button (notice that the chart legend changes to show debug, error and info).

• Selecting just the error logs can be done by either clicking on the word error (2) in the legend, followed by selecting Add to filter. Then click Run Search at the top of the page.

• Click on an error entry in the log table (make sure it says hostname: "paymentservice-xxxx" in case there is a rare error from a different service in the list too).

• Click on the X in the log message pane to close it.

• In Splunk Observability Cloud from the main menu, click on Digital Experience → Synthetics tests. Click on All or Browser tests to see the list of active tests.

• During our investigation in the RUM section, we found there was an issue with the Place Order Transaction. Let’s see if we can confirm this from the Synthetics test as well.

• In the Search box, enter [NAME OF WORKSHOP] to filter the tests for this workshop.

• Select the test.

• Click on Go to all run results.

• Change All to Failure (1).

  

• Click on one of the failed results.

• Now we are looking at the result of a single Synthetic Browser Test.

• By default, Splunk Synthetics provides screenshots and video capture of the test. This is useful for debugging issues. You can see, for example, the slow loading of large images, the slow rendering of a page, etc.

• Use your mouse to scroll left and right through the filmstrip to see how the site was being rendered during the test run.

• In the Video pane, press on the play button ▶ (1) to see the test playback.

• Using the filter above the filmstrip, under the heading Filter by a synthetic transaction, page, or step (2), click on Place Order under Synthetic transactions.

• In the waterfall find an entry that starts with POST checkout. If you don’t see that, go back and select another failed run result from the Run results page.

• Click on the > (1) to open the metadata section. Observe the metadata that is collected, and note the server-timing header under Response Headers. This header is what allows us to correlate the test run to a back-end trace.
• Click on the blue APM (2) link on the POST checkout line in the waterfall.

• Validate you see one or more errors for the paymentservice (1).
• To validate that it’s the same error, click on the related content for Logs (2).
• Repeat the earlier exercise to filter down to the errors only.
• View the error log to validate the failed payment due to an invalid token.

• Go to Digital Experience → Synthetics tests from the main menu.
• Select the workshop test [NAME OF WORKSHOP].
• Click on the test.
• Click Create Detector button at the top of the page:
• Change the alert criteria so that the metric is Run Duration (1) (instead of Uptime) and the condition is Static Threshold.
• Set the Trigger threshold (2) to be around 50000 ms.
• Set Split by location (3) to No.
• Note that there is now a row of red and white triangles appearing below the spikes in the chart.
• The red triangles let you know that your detector found that your test was above the given threshold and the white triangle indicates that the result returned below the threshold. Each red triangle will trigger an alert.

• This application is designed to fail constantly, hence the large number of alerts that would be generated. In a real-world scenario, you would want to fine-tune the threshold to avoid false positives.