Build & Deploy Application

Explore APM in Splunk

In this step, you’ll observe how disconnected traces appear in Splunk Observability Cloud. This is the “problem state” that the rest of the workshop fixes.

The APM Request Path #

When you clicked Place order, the request flowed through:

Browser (RUM span)
  → Frontend NGINX
    → Edge Gateway NGINX     ← break #1: trace headers dropped
      → Order API
        → Catalog API        ← direct HTTP 
        → Payment Gateway    ← break #2: strips headers to payment-api
          → Payment API
            → RabbitMQ       ← break #3: no trace context in message
              → Fulfillment Worker  ← orphan root trace

Three breaks occur:

1. HTTP break #1 at the edge NGINX gateway (browser → order API)
2. HTTP break #2 at the payment-gateway proxy (order API → payment API)
3. Messaging break at RabbitMQ (payment API → fulfillment worker)

Observe in Splunk APM #

Service Map #

1. Navigate to APM → Service Map
2. Filter environment: workshop-$INSTANCE
3. You should see services: order-api, payment-gateway, payment-api, fulfillment-worker, catalog-api

Trace Search #

1. Navigate to APM → Trace Analyzer
2. Filter:
   • Environment: workshop-$INSTANCE
   • Service: order-api
   • Operation: POST /api/orders (or storefront.place_order)
3. Open a recent trace

What you’ll see (broken state)

Knowledge Check #

Why does NGINX break propagation #

location /api/ {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_pass http://storefront_api;
}

When **any** `proxy_set_header` directive is present, NGINX stops automatically forwarding 
client headers to the upstream. Headers like `traceparent`, `tracestate`, and `baggage` 
are silently dropped unless explicitly configured.

This is one of the most common causes of broken trace correlation in production.

Why does RabbitMQ break propagation #

channel.sendToQueue(ordersQueue, Buffer.from(JSON.stringify(order)), {
  persistent: true,
  headers: { 'x-order-id': order.orderId },  // no traceparent
});

Unlike HTTP, message brokers don't participate in W3C Trace Context automatically. The
producer must **inject** trace context into message headers, and the consumer must
**extract** it. Without this, the consumer starts a new root trace.