Splunk Synthetics

15 minutes  
Persona

Putting your SRE hat back on, you have been asked to set up monitoring for the Online Boutique. You need to ensure that the application is available and performing well 24 hours a day, 7 days a week.

Wouldn’t it be great if we could have 24/7 monitoring of our application, and be alerted when there is a problem? This is where Synthetics comes in. We will show you a simple test that runs every 1 minute and checks the performance and availability of a typical user journey through the Online Boutique.

Last Modified Apr 3, 2024

Subsections of 8. Splunk Synthetics

1. Synthetics Dashboard

In Splunk Observability Cloud from the main menu, click on Synthetics. Click on All or Browser tests to see the list of active tests.

During our investigation in the RUM section, we found there was an issue with the Place Order Transaction. Let’s see if we can confirm this from the Synthetics test as well. We will be using the metric First byte time for the 4th page of the test, which is the Place Order step.

Exercise
  • In the Search box enter [WORKSHOP NAME] and select the test for your workshop (your instructor will advise as to which one to select).
  • Under Performance KPIs set the Time Picker to Last 1 hour and hit enter.
  • Click on Location and from the drop-down select Page. The next filter will be populated with the pages that are part of the test.
  • Click on Duration, deselect Duration and select First byte time. Transaction Filter Transaction Filter
  • Look at the legend and note the color of First byte time - Page 4.
  • Select the highest data point for First byte time - Page 4. You will now be taken to the Run results for this particular test run.
Last Modified Nov 7, 2024

2. Synthetics Test Detail

Right now we are looking at the result of a single Synthetic Browser Test. This test is split up into Business Transactions, think of this as a group of one or more logically related interactions that represent a business-critical user flow.

Info

The screenshot below doesn’t contain a red banner with an error in it however you might be seeing one in your run results. This is expected as in some cases the test run fails and does not impact the workshop.

waterfall waterfall

  1. Filmstrip: Offers a set of screenshots of site performance so that you can see how the page responds in real-time.
  2. Video: This lets you see exactly what a user trying to load your site from the location and device of a particular test run would experience.
  3. Browser test metrics: A View that offers you a picture of website performance.
  4. Synthetic transactions: List of the Synthetic transactions that made up the interaction with the site
  5. Waterfall chart The waterfall chart is a visual representation of the interaction between the test runner and the site being tested.

By default, Splunk Synthetics provides screenshots and video capture of the test. This is useful for debugging issues. You can see, for example, the slow loading of large images, the slow rendering of a page etc.

Exercise
  • Use your mouse to scroll left and right through the filmstrip to see how the site was being rendered during the test run.
  • In the Video pane, press on the play button ▶ to see the test playback. If you click the ellipses ⋮ you can change the playback speed, view it Picture in Picture and even Download the video.
  • In the Synthetic Transaction pane, under the header Business Transactions, click on the first button Home
  • The waterfall below will show all the objects that make up the page. The first line is the HTML page itself. The next lines are the objects that make up the page (HTML, CSS, JavaScript, Images, Fonts, etc.).
  • In the waterfall find the line GET splunk-otel-web.js.
  • Click on the > button to open the metadata section to see the Request/Response Header information.
  • In the Synthetic Transaction pane, click on the second Business Transaction Shop. Note that the filmstrip adjusts and moves to the beginning of the new transaction.
  • Repeat this for all the other Transactions, then finally select thePlaceOrder transaction.
Last Modified Dec 4, 2023

3. Synthetics to APM

We now should have a view similar to the one below.

Place Order Place Order

Exercise
  • In the waterfall find an entry that starts with POST checkout.
  • Click on the > button in front of it to drop open the metadata section. Observe the metadata that is collected, and note the Server-Timing header. This header is what allows us to correlate the test run to a back-end trace.
  • Click on the blue APM link on the POST checkout line in the waterfall.

APM trace APM trace

Exercise
  • Validate you see one or more errors for the paymentservice (1).
  • To validate that it’s the same error, click on the related content for Logs (2).
  • Repeat the earlier exercise to filter down to the errors only.
  • View the error log to validate the failed payment due to an invalid token.
Last Modified Dec 4, 2023

4. Synthetics Detector

Given you can run these tests 24/7, it is an ideal tool to get warned early if your tests are failing or starting to run longer than your agreed SLA instead of getting informed by social media, or Uptime websites.

Social media Social media

To stop that from happening let’s detect if our test is taking more than 1.1 minutes.

Exercise

  • Go back to the Synthetics home page via the menu on the left

  • Select the workshop test again and click the Create Detector button at the top of the page.
    synth detector synth detector

  • Edit the text New Synthetics Detector (1) and replace it with INITIALS - [WORKSHOPNAME]`.

  • Ensure that Run duration and Static threshold are selected.

  • Set the Trigger threshold (2) to be around 65,000 to 68,000 and hit enter to update the chart. Make sure you have more than one spike cutting through the threshold line as shown above (you may have to adjust the threshold value a bit to match your actual latency).

  • Leave the rest as default.

  • Note that there is now a row of red and white triangles appearing below the spikes (3). The red triangles let you know that your detector found that your test was above the given threshold & the white triangle indicates that the result returned below the threshold. Each red triangle will trigger an alert.

  • You can change the Alerts criticality (4) by changing the drop-down to a different level, as well as the method of alerting. Make sure you do NOT add a Recipient as this could lead to you being subjected to an alert storm!

  • Click Activate to deploy your detector.

  • To see your new created detector click Edit Test button

  • At the bottom of the page is a list of active detectors.

    list of detectors list of detectors

  • If you can’t find yours, but see one called New Synthetic Detector, you may not have saved it correctly with your name. Click on the New Synthetic Detector link, and redo the rename.

  • Click on the Close button to exit the edit mode.