ThousandEyes Integration with Splunk Observability Cloud
120 minutesAuthor
Alec Chamberlain
This workshop demonstrates integrating ThousandEyes with Splunk Observability Cloud to provide unified visibility across your synthetic monitoring and observability data.
What You’ll Learn
By the end of this workshop, you will:
Deploy a ThousandEyes Enterprise Agent as a containerized workload in Kubernetes
Integrate ThousandEyes metrics with Splunk Observability Cloud using OpenTelemetry
Configure distributed tracing so ThousandEyes and Splunk APM can link to the same requests
Create synthetic tests for internal Kubernetes services and external dependencies
Monitor test results in Splunk Observability Cloud dashboards
Move from ThousandEyes into Splunk APM traces and back to the originating ThousandEyes test
Sections
Core path
Overview - Understand ThousandEyes agent types and architecture
Deployment - Deploy the Enterprise Agent in Kubernetes
Splunk Integration - Stream ThousandEyes metrics into Splunk Observability Cloud
Distributed Tracing - Enable supported bi-directional drilldowns between ThousandEyes and Splunk APM
Scenario extensions
Kubernetes Testing - Create internal tests that are useful for both synthetic monitoring and trace correlation
RUM - Correlate ThousandEyes network signals with Splunk RUM for end-user investigations
Think of this scenario as two connected integrations: the OpenTelemetry stream gets ThousandEyes metrics into Splunk, and distributed tracing gives you the reverse path back into ThousandEyes from Splunk APM.
Prerequisites
A Kubernetes cluster (v1.16+)
RBAC permissions to deploy resources in your chosen namespace
A ThousandEyes account with access to Enterprise Agent tokens
A Splunk Observability Cloud account with ingest token access and permission to create an API token for APM lookups
Benefits of Integration
By connecting ThousandEyes to Splunk Observability Cloud, you gain:
🔗 Unified visibility: Correlate synthetic test results with RUM, APM traces, and infrastructure metrics
📊 Enhanced dashboards: Visualize ThousandEyes data alongside your existing Splunk observability metrics
🔄 Bi-directional drilldowns: Move from ThousandEyes Service Map to Splunk traces and from Splunk APM back to the ThousandEyes test that generated the request
🚨 Centralized alerting: Configure alerts based on ThousandEyes test results within Splunk
🔍 Root cause analysis: Quickly identify if issues are network-related (ThousandEyes) or application-related (APM)
Authentication: Splunk API token in the X-SF-Token header
Outcome: ThousandEyes can open related Splunk APM traces, and Splunk APM traces can link back to the originating ThousandEyes test
Observability Features:
Metrics: Real-time visualization of ThousandEyes data
Dashboards: Pre-built ThousandEyes dashboard with unified views
APM/RUM Integration: Correlate synthetic tests with application traces and real user monitoring
Alerting: Centralized alert management with correlation rules
6. Data Flow
Agent authenticates using token from Kubernetes Secret
Agent runs scheduled tests against internal and external targets
Test results sent to ThousandEyes Cloud
ThousandEyes streams metrics to Splunk via OpenTelemetry protocol
For HTTP Server and API tests with distributed tracing enabled, ThousandEyes injects b3, traceparent, and tracestate headers into the request
The instrumented application sends the resulting trace to Splunk APM
ThousandEyes can open the related Splunk trace, and Splunk APM can link back to the original ThousandEyes test
DevOps, network, and application teams collaborate across both views during an investigation
Testing Capabilities
With this deployment, you can:
✅ Test internal services: Monitor Kubernetes services, APIs, and microservices from within the cluster
✅ Test external dependencies: Validate connectivity to payment gateways, third-party APIs, and SaaS platforms
✅ Measure performance: Capture latency, availability, and performance metrics from your cluster’s perspective
✅ Troubleshoot issues: Identify whether problems originate from your infrastructure, network path, or instrumented application services
Note
This is not an officially supported ThousandEyes agent deployment configuration. However, it has been tested and works very well in production-like environments.
Deployment
20 minutes
This section guides you through deploying the ThousandEyes Enterprise Agent in your Kubernetes cluster.
Components
The deployment consists of two files:
1. Secrets File (credentialsSecret.yaml)
Contains your ThousandEyes agent token (base64 encoded). This secret is referenced by the deployment to authenticate the agent with ThousandEyes Cloud.
Create a file named thousandEyesDeploy.yaml with the deployment manifest shown above (customize the hostname and namespace as needed).
Apply the deployment:
kubectl apply -f thousandEyesDeploy.yaml
Step 5: Verify the Deployment
Verify the agent is running:
kubectl get pods -n te-demo
Expected output:
NAME READY STATUS RESTARTS AGE
thousandeyes-xxxxxxxxxx-xxxxx 1/1 Running 0 2m
Check the logs to ensure the agent is connecting:
kubectl logs -n te-demo -l app=thousandeyes
Step 6: Verify in ThousandEyes Dashboard
Verify in the ThousandEyes dashboard that the agent has registered successfully:
Navigate to Cloud & Enterprise Agents > Agent Settings to see your newly registered agent.
Success
Your ThousandEyes Enterprise Agent is now running in Kubernetes! Next, we’ll integrate it with Splunk Observability Cloud.
Background
ThousandEyes does not provide official Kubernetes deployment documentation. Their standard deployment method uses docker run commands, which makes it challenging to translate into reusable Kubernetes manifests. This guide bridges that gap by providing production-ready Kubernetes configurations.
Splunk Integration
15 minutes
About Splunk Observability Cloud
Splunk Observability Cloud is a real-time observability platform purpose-built for monitoring metrics, traces, and logs at scale. It ingests OpenTelemetry data and provides advanced dashboards and analytics to help teams detect and resolve performance issues quickly. This section explains how to integrate ThousandEyes data with Splunk Observability Cloud using OpenTelemetry.
Scope Of This Section
This section covers the metrics streaming path from ThousandEyes into Splunk Observability Cloud. The next section adds the separate distributed tracing workflow that creates bi-directional links between ThousandEyes and Splunk APM.
Step 1: Create a Splunk Observability Cloud Access Token
To send ThousandEyes metrics to Splunk Observability Cloud, you need an access token with the Ingest scope. Follow these steps:
In the Splunk Observability Cloud platform, go to Settings > Access Token
Click Create Token
Enter a Name
Select Ingest scope
Select Create to generate your access token
Copy the access token and store it securely
You need the access token to send telemetry data to Splunk Observability Cloud.
Step 2: Create an Integration
This integration is the one-way telemetry stream that gets ThousandEyes metrics into Splunk Observability Cloud dashboards and detectors.
Using the ThousandEyes UI
To integrate Splunk Observability Cloud with ThousandEyes:
Log in to your account on the ThousandEyes platform and go to Manage > Integration > Integration 1.0
Click New Integration and select OpenTelemetry Integration
Enter a Name for the integration
Set the Target to HTTP
Enter the Endpoint URL: https://ingest.{REALM}.signalfx.com/v2/datapoint/otlp
Replace {REALM} with your Splunk environment (e.g., us1, eu0)
For Preset Configuration, select Splunk Observability Cloud
For Auth Type, select Custom
Add the following Custom Headers:
X-SF-Token: {TOKEN} (Enter your Splunk Observability Cloud access token created in Step 1)
Content-Type: application/x-protobuf
For OpenTelemetry Signal, select Metric
For Data Model Version, select v2
Select a test
Click Save to complete the integration setup
You have now successfully integrated your ThousandEyes data with Splunk Observability Cloud.
Using the ThousandEyes API
For a programmatic integration, use the following API commands:
Replace streamEndpointUrl and X-SF-Token values with the correct values for your Splunk Observability Cloud instance.
Note
Make sure to replace {REALM} with your Splunk environment realm (e.g., us1, us2, eu0) and {TOKEN} with your actual Splunk access token.
What Comes Next
After you finish the metrics integration, continue to Distributed Tracing to add the reverse investigation path from ThousandEyes into Splunk APM and back again.
Step 3: ThousandEyes Dashboard in Splunk Observability Cloud
Once the integration is set up, you can view real-time monitoring data in the ThousandEyes Network Monitoring Dashboard within Splunk Observability Cloud. The dashboard includes:
HTTP Server Availability (%): Displays the availability of monitored HTTP servers
HTTP Throughput (bytes/s): Shows the data transfer rate over time
Client Request Duration (seconds): Measures the latency of client requests
Web Page Load Completion (%): Indicates the percentage of successful page loads
Page Load Duration (seconds): Displays the time taken to load pages
Your ThousandEyes data is now streaming to Splunk Observability Cloud. Next, add the distributed tracing connector so you can pivot between ThousandEyes and Splunk APM during troubleshooting.
Distributed Tracing and Bi-Directional Drilldowns
25 minutes
This section turns the ThousandEyes and Splunk integration into a true investigation workflow. In the previous section, ThousandEyes streamed synthetic metrics into Splunk Observability Cloud. In this section, you will enable the supported ThousandEyes <-> Splunk APM distributed tracing integration so network, platform, and application teams can pivot between both tools while looking at the same request.
Why This Matters
This is the piece that gives you bi-directional access between the two environments. ThousandEyes can open the related trace in Splunk APM, and Splunk APM can take you back to the originating ThousandEyes test.
What You Will Learn
By the end of this section, you will be able to:
Instrument an internal service so it sends traces to Splunk APM
Enable distributed tracing on a ThousandEyes HTTP Server or API test
Configure the ThousandEyes Generic Connector for Splunk APM
Open the ThousandEyes Service Map and jump directly into the corresponding Splunk trace
Use the ThousandEyes metadata in Splunk APM to jump back to the original ThousandEyes test
Supported Workflow
This learning scenario follows the supported workflow documented by ThousandEyes and Splunk:
ThousandEyes automatically injects b3, traceparent, and tracestate headers into HTTP Server and API tests when distributed tracing is enabled.
The monitored endpoint must accept headers, be instrumented with OpenTelemetry, propagate trace context, and send traces to your observability backend.
For Splunk APM, ThousandEyes uses a Generic Connector that points at https://api.<REALM>.signalfx.com and authenticates with an API-scope Splunk token.
Splunk APM enriches matching traces with ThousandEyes attributes such as thousandeyes.test.id and thousandeyes.permalink, which enables the reverse jump back to ThousandEyes.
What Those Headers Actually Mean
This part is easy to gloss over and it should not be. The trace correlation only works if the service understands the headers ThousandEyes injects and continues the trace correctly.
traceparent and tracestate are the W3C Trace Context headers.
b3 is the Zipkin B3 single-header format.
ThousandEyes injects both because real environments often contain a mix of proxies, meshes, gateways, and app runtimes that do not all prefer the same propagation format.
In OpenTelemetry terms, the important setting is the propagator list:
OTEL_PROPAGATORS=baggage,b3,tracecontext
That does two things:
It allows the service to extract either B3 or W3C context from the inbound ThousandEyes request.
It preserves the W3C tracestate by keeping tracecontext enabled.
Important Detail
You do not add tracestate as a separate OpenTelemetry propagator. The tracecontext propagator handles both traceparent and tracestate.
What “Properly Done” Looks Like
The collector is only one part of this setup. A correct ThousandEyes tracing deployment in Kubernetes has three layers:
Deployment annotation so the OpenTelemetry Operator injects the runtime-specific instrumentation.
Instrumentation resource so the injected SDK knows where to send traces and which propagators to use.
Collector trace pipeline so OTLP traces are actually received and exported to Splunk APM.
The most common mistake is to focus only on the collector. The collector never sees raw b3, traceparent, or tracestate request headers directly. Your application or auto-instrumentation library must extract those headers first, continue the span context, and then emit spans over OTLP to the collector.
Real-World Configuration From The Current Cluster
The examples below are trimmed from the live cluster currently running this workshop. They show the pattern that is actually working in Kubernetes today.
1. Deployment Annotation
In the live cluster, the teastore applications point at the teastore/default Instrumentation resource:
This is a useful validation checkpoint because it proves the propagators are being applied to the workload, not just declared in an abstract config object.
4. Agent Collector Trace Pipeline
The live agent collector in otel-splunk is receiving OTLP, Jaeger, and Zipkin traffic and forwarding traces upstream. This is a trimmed excerpt from the running ConfigMap:
For ThousandEyes, the important part is not a special B3 option in the collector. The important part is simply that the collector exposes OTLP on 4317 and 4318, and that your services are exporting their spans there.
5. Gateway Collector Export To Splunk APM
The live gateway collector then forwards traces to Splunk Observability Cloud. This is the relevant part of the running gateway ConfigMap:
This is the part that gets the spans to Splunk APM. If this pipeline is broken, ThousandEyes can still inject headers into the request, but no correlated trace will ever appear in Splunk.
Current Cluster Takeaway
In the live cluster, the teastore/default Instrumentation resource is the pattern to follow for ThousandEyes because it explicitly includes b3 together with tracecontext. That is the configuration you want to replicate for this scenario.
Important
Do not use a browser page URL for this section. ThousandEyes documents that browsers do not accept the custom trace headers required for this workflow. Use an instrumented backend endpoint behind an HTTP Server or API test instead.
Step 1: Make Sure the Workload Emits Traces to Splunk APM
If your application is already instrumented and traces are visible in Splunk APM, you can skip to Step 2. Otherwise, the fastest learning path in Kubernetes is to use the Splunk OpenTelemetry Collector with the Operator enabled for zero-code instrumentation.
Install the Splunk OpenTelemetry Collector with the Operator
For Java workloads, a generic example looks like this:
kubectl patch deployment api-gateway -n production -p '{"spec":{"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-java":"otel-splunk/splunk-otel-collector"}}}}}'
For other runtimes, use the annotation that matches the language:
instrumentation.opentelemetry.io/inject-nodejs
instrumentation.opentelemetry.io/inject-python
instrumentation.opentelemetry.io/inject-dotnet
If the collector is installed in the same namespace as the application, the official Splunk documentation also supports using "true" as the annotation value.
If you want to follow the live cluster pattern from this workshop environment, the annotation value is namespace-qualified and points at the teastore/default Instrumentation object:
In the current workshop cluster, a service such as http://teastore-webui.teastore.svc.cluster.local:8080/ is the right kind of target because it fronts several downstream application services and produces a more useful end-to-end trace than a simple health check.
Confirm that traces are arriving in Splunk APM before you continue.
Learning Tip
Use a business transaction, not a pure /health endpoint, for the tracing exercise. A multi-service request gives you a far better Service Map in ThousandEyes and a more useful trace in Splunk APM.
Step 2: Enable Distributed Tracing on the ThousandEyes Test
Create or edit an HTTP Server or API test that targets the instrumented backend endpoint from Step 1.
In ThousandEyes, create an HTTP Server or API test.
Open Advanced Settings.
Enable Distributed Tracing.
Save the test and run it against the same endpoint that is already sending traces to Splunk APM.
After the test runs, ThousandEyes injects the trace headers and captures the trace context for that request.
Step 3: Create the Splunk APM Connector in ThousandEyes
The metric streaming integration from the previous section uses an Ingest token. This step is different: ThousandEyes needs to query Splunk APM and build trace links, so it uses a Splunk API token instead.
In Splunk Observability Cloud, create an access token with the API scope.
In ThousandEyes, go to Manage > Integrations > Integrations 2.0.
Create a Generic Connector with:
Target URL: https://api.<REALM>.signalfx.com
Header: X-SF-Token: <your-api-scope-token>
Create a new Operation and select Splunk Observability APM.
Enable the operation and save the integration.
Step 4: Validate the Bi-Directional Investigation Loop
Once the test is running and the connector is enabled, validate the workflow in both directions.
Start in ThousandEyes
Open the test in ThousandEyes.
Navigate to the Service Map tab.
Confirm that you can see the trace path, service latency, and any downstream errors.
Use the ThousandEyes link into Splunk APM to inspect the full trace.
Continue in Splunk APM
Inside Splunk APM, verify that the trace contains ThousandEyes metadata such as:
thousandeyes.account.id
thousandeyes.test.id
thousandeyes.permalink
thousandeyes.source.agent.id
Use either the thousandeyes.permalink field or the Go to ThousandEyes test button in the trace waterfall view to navigate back to the originating ThousandEyes test.
Suggested Learning Scenario
Use the following flow during a workshop:
Create a ThousandEyes test against an internal API route that calls multiple services.
Let ThousandEyes surface the issue first, so the class starts from the network and synthetic-monitoring perspective.
Open the Service Map in ThousandEyes and identify where latency or errors begin.
Jump into Splunk APM for span-level analysis.
Jump back to ThousandEyes to inspect the test, agent, and network path again.
This is a strong teaching loop because it mirrors how different teams actually work:
Network and edge teams often start in ThousandEyes.
SRE and platform teams often start in Splunk dashboards or alerts.
Application teams usually want the trace in Splunk APM.
With this integration in place, everyone can pivot without losing context.
Common Pitfalls
A test might be visible in Splunk dashboards but still have no trace correlation. That usually means only the metrics stream is configured, not the Splunk APM Generic Connector.
A trace might exist in Splunk APM but not show up in ThousandEyes if the monitored endpoint does not propagate the trace headers downstream.
A shallow endpoint such as /health often produces limited trace value even when the configuration is correct.
AppDynamics offers a feature called “Test Recommendations” that automatically suggests synthetic tests for your application endpoints. With ThousandEyes deployed inside your Kubernetes cluster, you can replicate this capability by leveraging Kubernetes service discovery combined with Splunk Observability Cloud’s unified view.
Since the ThousandEyes Enterprise Agent runs inside the cluster, it can directly test internal Kubernetes services using their service names as hostnames. This provides a powerful way to monitor backend services that may not be exposed externally.
How It Works
Service Discovery: Use kubectl get svc to enumerate services in your cluster
Hostname Construction: Build test URLs using Kubernetes DNS naming convention: <service-name>.<namespace>.svc.cluster.local
Test Creation: Create both availability tests and trace-enabled transaction tests for internal services
Correlation in Splunk: View synthetic test results alongside APM traces and infrastructure metrics
Benefits of In-Cluster Testing
Internal Service Monitoring: Test backend services not exposed to the internet
Service Mesh Awareness: Monitor services behind Istio, Linkerd, or other service meshes
DNS Resolution Testing: Validate Kubernetes DNS and service discovery
Network Policy Validation: Ensure network policies allow proper communication
Pre-Production Testing: Test services before exposing them via Ingress/LoadBalancer
Step-by-Step Guide
1. Discover Kubernetes Services
List all services in your cluster or a specific namespace:
# Get all services in all namespaceskubectl get svc --all-namespaces
# Get services in a specific namespacekubectl get svc -n production
# Get services with detailed output including portskubectl get svc -n production -o wide
Example output:
NAMESPACE NAME TYPE CLUSTER-IP PORT(S) AGE
production api-gateway ClusterIP 10.96.100.50 8080/TCP 5d
production payment-svc ClusterIP 10.96.100.51 8080/TCP 5d
production auth-service ClusterIP 10.96.100.52 9000/TCP 5d
production postgres ClusterIP 10.96.100.53 5432/TCP 5d
2. Build Test Hostnames
Kubernetes services are accessible via DNS using the following naming pattern:
<service-name>.<namespace>.svc.cluster.local
For the services above:
api-gateway.production.svc.cluster.local:8080
payment-svc.production.svc.cluster.local:8080
auth-service.production.svc.cluster.local:9000
Shorthand within the same namespace:
If testing services in the same namespace as the ThousandEyes agent, you can use just the service name:
api-gateway:8080
payment-svc:8080
3. Create ThousandEyes Tests for Internal Services
For the best learning outcome, create two kinds of tests:
Availability tests against /health or /readiness endpoints to validate reachability and response time
Trace-enabled transaction tests against business endpoints that traverse multiple services
The first test teaches synthetic monitoring. The second teaches cross-tool troubleshooting with Splunk APM.
Via ThousandEyes UI
Navigate to Cloud & Enterprise Agents > Test Settings
For the trace-enabled version, switch the url to a business transaction endpoint and enable distributed tracing in the ThousandEyes test configuration.
Best Practice
If your goal is to teach distributed tracing, avoid using /health as the only example. Health checks are useful for uptime monitoring, but they rarely produce the multi-service traces that make the ThousandEyes and Splunk APM integration compelling.
4. Configure Alerting Rules
Set up alerts for common failure scenarios:
Availability Alert: Trigger when HTTP response is not 200
Performance Alert: Trigger when response time exceeds baseline
DNS Resolution Alert: Trigger when service DNS cannot be resolved
5. View Results in Splunk Observability Cloud
Once tests are running and integrated with Splunk:
Navigate to the ThousandEyes Dashboard in Splunk Observability Cloud
Filter by test name (e.g., [K8s] prefix) to see all Kubernetes internal tests
For trace-enabled tests, start in ThousandEyes first:
Open the Service Map
Inspect service-level latency and downstream errors
Follow the link into Splunk APM
Correlate with APM data:
View synthetic test failures alongside APM error rates
Identify if issues are network-related (ThousandEyes) or application-related (APM)
Use the Splunk trace metadata to jump back to the originating ThousandEyes test
Create custom dashboards combining:
ThousandEyes HTTP availability metrics
APM service latency and error rates
Kubernetes infrastructure metrics (CPU, memory, pod restarts)
While ThousandEyes is primarily for HTTP testing, you can test database proxies:
# Test PgBouncer or database HTTP management interfaceshttp://pgbouncer.production.svc.cluster.local:8080/stats
http://redis-exporter.production.svc.cluster.local:9121/metrics
Use Case 4: External Service Dependencies
One of the most valuable capabilities of the in-cluster ThousandEyes agent is monitoring your application’s external dependencies from the same network perspective as your services. This helps identify whether issues originate from your infrastructure, network path, or the external service itself.
Testing Payment Gateways
Create tests for critical payment gateway endpoints to ensure availability and performance:
Stripe API:
# Via ThousandEyes UITest Name: [External] Stripe API Health
URL: https://api.stripe.com/healthcheck
Interval: 2 minutes
Agents: Your Kubernetes Enterprise Agent
Expected Response: 200
PayPal API:
Test Name: [External] PayPal API Health
URL: https://api.paypal.com/v1/notifications/webhooks
Interval: 2 minutes
Agents: Your Kubernetes Enterprise Agent
Expected Response: 401(authentication required, but endpoint is reachable)
Cloud Storage: AWS S3, Google Cloud Storage, Azure Blob Storage
Third-Party APIs: Any critical business partner APIs
Best Practice
Use the [External] prefix in test names to easily distinguish between internal Kubernetes services and external dependencies in your dashboards.
Best Practices
Use Consistent Naming: Prefix test names with [K8s] or [Internal] for easy filtering
Test Health Endpoints First: Start with /health or /readiness endpoints before testing business logic
Set Appropriate Intervals: Use shorter intervals (1-2 minutes) for critical services
Tag Tests: Use ThousandEyes labels/tags to group tests by:
Environment (dev, staging, production)
Service type (API, database, cache)
Team ownership
Monitor Test Agent Health: Ensure the ThousandEyes agent pod is healthy and has sufficient resources
Use Both Test Types: Pair a simple availability test with a trace-enabled business transaction test for each critical service path
Correlate with APM: Create Splunk dashboards that show both synthetic and APM data side-by-side
Use Instrumented Backends for Trace Labs: Distributed tracing works best when the ThousandEyes target is an HTTP Server or API endpoint backed by OpenTelemetry-instrumented services
Tip
By testing internal services before they’re exposed externally, you can catch issues early and ensure your infrastructure is healthy before user traffic reaches it.
ThousandEyes and Splunk RUM
10 minutes
Integrate ThousandEyes with Splunk RUM to understand if network issues correlate to end user issues.
Requirements
Admin privilege to both Splunk Observability Cloud and ThousandEyes
At least one application sending data into Splunk RUM
At least one test of these types running in ThousandEyes, on the same domain as the app in Splunk RUM:
Select your username on the top-right corner, and then select Profile.
Under User API Tokens, select Create to generate an OAuth Bearer Token.
Copy or make a note of the token to use in the Observability Cloud data integration wizard
In Splunk Observability Cloud, open Data Management > Available Integrations > ThousandEyes Integration with RUM
Use the same Ingest token from the previous Splunk Integration, or create and select a dedicated Ingest token to better track the data usage of your RUM integration.
Enter the OAuth Bearer token from ThousandEyes
Review the test matches, change selections as desired, and review the estimated data ingestion before selecting Done
View the integration
Go to the RUM application where your ThousandEyes tests are running, and view the Map.
Hover over the locations where you also have ThousandEyes test running to see the preview of ThousandEyes metrics:
If you have active alerts in ThousandEyes, you will see the ThousandEyes icon over the relevant location bubble in RUM:
Click into a relevant region to see the Network metrics alongside other metrics from RUM, and open View ThousandEyes Tests to go to the relevant tests in ThousandEyes:
See RUM and ThousandEyes metrics in a custom dashboard
Now you can correlate other Observability Cloud KPIs with signals from your relevant ThousandEyes tests!
Go to Dashboards > search for “RUM” > click into one of the out-of-the-box RUM dashboards in the RUM applications group
Either copy charts with RUM KPIs that interest you, or open a dashboard’s action menu on the top right and Save As to create a copy in your own dashboard group.
On the new dashboard, create a new chart with the signal network.latency
change the extrapolation policy to Last value
change the unit of measurement to Time > Millisecond
in Chart Options, select Show on-chart legend with the value thousandeyes.source.agent.name. This will segment the chart by agent location from ThousandEyes.
Name and save the new chart, then copy it to create similar charts for network.jitter and network.loss by changing the metric in the copied charts signal, and adjusting the units of measure and visualization options as needed.
See the Dashboard Workshop for more in-depth guidance on creating custom dashboards and charts.
Tip
Think about other metrics in Observability Cloud that might be handy to view side-by-side with ThousandEyes test metrics.
For example, if you have API tests running in Synthetics, consider adding a heatmap of API test success by location.
Troubleshooting
15 minutes
This section covers common issues you may encounter when deploying and using the ThousandEyes Enterprise Agent in Kubernetes.
Test Failing with DNS Resolution Error
If your tests are failing with DNS resolution errors, verify DNS from within the ThousandEyes pod:
# Verify DNS resolution from within the podkubectl exec -n te-demo -it <pod-name> -- nslookup api-gateway.production.svc.cluster.local
# Check CoreDNS logskubectl logs -n kube-system -l k8s-app=kube-dns
Common causes:
Service doesn’t exist in the specified namespace
Typo in the service name or namespace
CoreDNS is not functioning properly
Connection Refused Errors
If you’re seeing connection refused errors, check the following:
# Verify service endpoints existkubectl get endpoints -n production api-gateway
# Check if pods are readykubectl get pods -n production -l app=api-gateway
# Test connectivity from agent podkubectl exec -n te-demo -it <pod-name> -- curl -v http://api-gateway.production.svc.cluster.local:8080/health
Common causes:
No pods backing the service (endpoints are empty)
Pods are not in Ready state
Wrong port specified in the test URL
Service selector doesn’t match pod labels
Network Policy Blocking Traffic
If network policies are blocking traffic from the ThousandEyes agent:
# List network policieskubectl get networkpolicies -n production
# Describe network policykubectl describe networkpolicy <policy-name> -n production
Solution:
Create a network policy to allow traffic from the te-demo namespace to your services:
If the ThousandEyes agent pod is not starting, check the pod status and events:
# Get pod statuskubectl get pods -n te-demo
# Describe pod to see eventskubectl describe pod -n te-demo <pod-name>
# Check logskubectl logs -n te-demo <pod-name>
Common causes:
Insufficient resources (memory/CPU)
Invalid or missing TEAGENT_ACCOUNT_TOKEN secret
Security context capabilities not allowed by Pod Security Policy
Image pull errors
Solutions:
Increase memory limits if OOMKilled
Verify secret is created correctly: kubectl get secret te-creds -n te-demo -o yaml
Check Pod Security Policy allows NET_ADMIN and SYS_ADMIN capabilities
Verify image pull: kubectl describe pod -n te-demo <pod-name>
Agent Not Appearing in ThousandEyes Dashboard
If the agent is running but not appearing in the ThousandEyes dashboard:
Check the OpenTelemetry integration is configured correctly in ThousandEyes
Verify the Splunk ingest endpoint URL is correct for your realm
Confirm the X-SF-Token header contains a valid Splunk access token
Ensure tests are assigned to the integration
Check test assignment:
# Use ThousandEyes API to verify integrationcurl -v https://api.thousandeyes.com/v7/stream \
-H "Authorization: Bearer $BEARER_TOKEN"
Common causes:
Wrong Splunk realm in endpoint URL
Invalid or expired Splunk access token
Tests not assigned to the OpenTelemetry integration
Integration not enabled or saved properly
Distributed Tracing Not Appearing in ThousandEyes
If your metric stream is working but the ThousandEyes Service Map is empty or no trace is found:
Verify the monitored endpoint:
It accepts HTTP headers
It is instrumented with OpenTelemetry
It propagates trace context downstream
It sends traces to Splunk APM
Common causes:
The endpoint is a page URL rather than an HTTP Server or API target
The service is not instrumented, so ThousandEyes can inject headers but no trace is emitted
The endpoint only returns a local health response and does not exercise downstream services
Recommended fixes:
Switch the ThousandEyes test to an instrumented backend API route
Confirm traces for that route already exist in Splunk APM
Re-run the test after enabling ThousandEyes distributed tracing
Missing ThousandEyes Link in Splunk APM
If the trace opens in Splunk APM but you do not see the ThousandEyes backlink or metadata:
Common cause:
The b3 propagator can override trace_state and clear the value that ThousandEyes expects to preserve for the reverse link.
Fix:
Set the propagators explicitly on the instrumented service:
OTEL_PROPAGATORS=baggage,b3,tracecontext
After changing the environment variable, restart the instrumented workload and generate new traffic.
Splunk APM Connector Authentication Errors
If the Generic Connector in ThousandEyes cannot query Splunk APM:
Check the following:
The connector target is https://api.<REALM>.signalfx.com
The token used in the connector has the API scope
The user creating the token has the required role in Splunk Observability Cloud
Token Reminder
The OpenTelemetry metrics stream uses a Splunk Ingest token. The ThousandEyes Generic Connector for APM uses a Splunk API token. Mixing them up is one of the most common causes of partial integration.
High Memory Usage
If the ThousandEyes agent pod is consuming excessive memory:
# Check current memory usagekubectl top pod -n te-demo
# Check for OOMKilled eventskubectl describe pod -n te-demo <pod-name> | grep -i oom
Solutions:
Increase memory limits in the deployment:
resources:limits:memory:4096Mi # Increase from 3584Mirequests:memory:2500Mi # Increase from 2000Mi
Reduce the number of concurrent tests assigned to the agent
Check if the agent is running unnecessary services
Permission Denied Errors
If you see permission denied errors in the agent logs:
Verify security context:
kubectl get pod -n te-demo <pod-name> -o jsonpath='{.spec.containers[0].securityContext}'
Solution:
Ensure the pod has the required capabilities:
Some Kubernetes clusters with strict Pod Security Policies may not allow these capabilities. You may need to work with your cluster administrators to create an appropriate policy exception.
Getting Help
If you encounter issues not covered in this guide:
