APM Service Map

The above map shows all the interactions between all the services. The map may still be in an interim state as it will take the PetClinic Microservice application a few minutes to start up and fully synchronize. Reducing the time filter to a custom time of 2 minutes by entering -2m will help. You can click on the Refresh button (1) in the top right-hand corner of the screen. The initial startup-related errors indicated by red circles will eventually disappear.

Next, let’s examine the metrics that are available for each service that is instrumented by visiting the request, error, and duration (RED) metrics Dashboard.

For this exercise, we are going to use a common scenario you would use if your service operation was showing high latency or errors.

Click on customers-service in the Dependency map, then make sure the customers-service is selected in the Services dropdown box (1). Next, select GET /owners from the Operations dropdown (2) adjacent to the Service name.

This should give you the workflow with a filter on GET /owners as shown below:

APM Trace

To pick a trace, select a line in the Service Requests & Errors chart (1). A selection of related traces will appear.

Once you have the list of related traces, click on the blue (2) Trace ID Link, making sure the trace you select has the same three services mentioned in the Services Column.

This brings us to the selected Trace in the Waterfall view:

Here we find several sections:

• The Waterfall Pane (1), where you see the trace and all the instrumented functions visible as spans, with their duration representation and order/relationship showing.
• The Trace Info Pane (2), which shows the selected Span information (highlighted with a box around the Span in the Waterfall Pane).
• The Span Pane (3) where you can find all the Tags that have been sent in the selected Span. You can scroll down to see all of them.
• The process Pane, with tags related to the process that created the Span (scroll down to see as it is not in the screenshot).
• The Trace Properties, located at the top right-hand side of the pane is collapsed by default.

APM Span

While we examine our spans, let’s look at several out-of-the-box features that you get without code modifications when using automatic discovery and configuration on top of tracing:

First, in the Waterfall Pane, make sure the customers-service:SELECT petclinic or similar span is selected as shown in the screenshot below:

• The basic latency information is shown as a bar for the instrumented function or call. In our example above, it took 17.8 Milliseconds.
• Several similar Spans (1) are only visible if the span is repeated multiple times. In this case, there are 10 repeats in our example. You can show/hide them all by clicking on the 10x and all spans will show in order.
• Inferred Services: Calls made to external systems that are not instrumented show up as a grey ‘inferred’ span. The Inferred Service or span in our example is a call to the Mysql Database mysql:petclinic SELECT petclinic (2) as shown above.
• Span Tags: In the Tag Pane, we see standard tags produced by the automatic discovery and configuration. In this case, the span is calling a Database, so it includes the db.statement tag (3). This tag will hold the DB query statement and is used by the Database call performed during this span. This will be used by the DB-Query Performance feature. We look at DB-Query Performance in the next section.
• Always-on Profiling: IF the system is configured to capture Profiling data during a Span life cycle. It will show the number of Call Stacks captured in the Spans timeline. In our example above, we see 18 Call Stacks for the customer-service:GET /owners Span. (4)

We will look at Profiling in the next section.

Service Centric View

Splunk APM provide Service Centric Views that provide engineers a deep understanding of service performance in one centralized view. Now, across every service, engineers can quickly identify errors or bottlenecks from a service’s underlying infrastructure, pinpoint performance degradation from new deployments, and visualize the health of every third party dependency.

To see this dashboard for the api-gateway, click on APM from the left-hand menu and then click on the api-gateway service in the list. This will bring you to the Service Centric View dashboard:

This view, which is available for each of your instrumented services, offers an overview of Service metrics, Error breakdown, Runtime metrics (Java) and Infrastructure metrics.