Distributed Tracing and Bi-Directional Drilldowns

Distributed Tracing and Bi-Directional Drilldowns

Configure instrumentation

10 minutes

Summary of the changes we need to make

The ThousandEyes documentation , and specifically the page for Splunk Observability APM , shows what is needed for distributed tracing:

For Propagators:

In addition setting the sampler to parentbased_always_on ensures the trace continues once ThousandEyes starts the request.

Important Lesson

In our testing (at least at the time of this writing) we’ve confirmed that the order of the propagators makes a difference, and the default doesn’t work. So we will need to patch the correct order now.

We will make the following changes:

Step 1: Modify the OTel Collector

Let’s check the configuration of the instrumentation:

bash 
kubectl describe instrumentation splunk-otel-collector

Under Propagators you can see that the ones we need are set, but we will still patch it so it’s in the correct order.

Step 2: Patch the instrumentation (to resolve the defaults)

We’re going to patch this for now, but any future upgrades will lose this change. So the right way to do this would be to update a values.yaml file so that this is always applied.

bash 
kubectl patch instrumentation splunk-otel-collector \
  --type=merge \
  -p '{
    "spec": {
      "propagators": ["baggage", "b3", "tracecontext"],
      "sampler": {
        "type": "parentbased_always_on"
      }
    }
  }'

You can then verify the Propagators (in the right order) and added sampler are there:

bash 
kubectl describe instrumentation splunk-otel-collector

Step 3: Patch the application

First, let’s check which container images are deployed:

bash 
kubectl describe pods api-gateway | grep Image:

We can see there is only one container for the api-gateway. Once we patch the application we will see multiple container images (one for the api-gateway, and the other for instrumentation).

Let’s inject the java instrumentation. (NOTE: There will be no change for the config-server, discovery-server and admin-server as these have already been patched.):

bash 
kubectl get deployments -l app.kubernetes.io/part-of=spring-petclinic -o name | xargs -I % kubectl patch % -p "{\"spec\": {\"template\":{\"metadata\":{\"annotations\":{\"instrumentation.opentelemetry.io/inject-java\":\"default/splunk-otel-collector\"}}}}}"

For other runtimes

For other runtimes, use the annotation that matches the language; for example:

  • instrumentation.opentelemetry.io/inject-nodejs
  • instrumentation.opentelemetry.io/inject-python
  • instrumentation.opentelemetry.io/inject-dotnet

We can check that our instrumentation deployed with:

bash 
kubectl describe pods api-gateway | grep Image:

You can also see that this pod has the Java instrumentation enabled, and the propagators are including baggage, b3, and tracecontext in the right ordeer:

bash 
kubectl describe pods api-gateway | grep OTEL_PROPAGATORS

Restart all the pods

Since some of the pods were already injected, it’s important we restart them all to get the correct instrumentation.

To do that:

bash 
kubectl rollout restart deployment -l app.kubernetes.io/part-of=spring-petclinic

Now we can validate the in-cluster API path from the namespace where the ThousandEyes Enterprise Agent runs.

Try running:

bash 
kubectl run te-petclinic-curl \
  --rm -it \
  --restart=Never \
  --image=curlimages/curl \
  --command -- curl -sS http://api-gateway.default.svc.cluster.local:82/api/customer/owners

Be patient

This may take some time until you get the expected output.

Your deployment environment is:

bash 
echo "thousandeyes-$INSTANCE"

You should see the full environment showing in Splunk Observability Cloud (filter on your environment, thousandeyes-shw-xxxx)

ThousandEyes Service Map

Last Modified ·