ThousandEyes Integration with Splunk Observability Cloud

90 minutes   Author Alec Chamberlain

This workshop demonstrates integrating ThousandEyes with Splunk Observability Cloud to provide unified visibility across your synthetic monitoring and observability data.

What You’ll Learn

By the end of this workshop, you will:

  • Deploy a ThousandEyes Enterprise Agent as a containerized workload in Kubernetes
  • Integrate ThousandEyes metrics with Splunk Observability Cloud using OpenTelemetry
  • Create synthetic tests for internal Kubernetes services and external dependencies
  • Monitor test results in Splunk Observability Cloud dashboards
  • Correlate synthetic test data with APM traces and infrastructure metrics

Sections

Tip

This integration enables you to correlate synthetic test results with real user monitoring (RUM), APM traces, and infrastructure metrics for comprehensive root cause analysis.

Prerequisites

  • A Kubernetes cluster (v1.16+)
  • RBAC permissions to deploy resources in your chosen namespace
  • A ThousandEyes account with access to Enterprise Agent tokens
  • A Splunk Observability Cloud account with ingest token access

Benefits of Integration

By connecting ThousandEyes to Splunk Observability Cloud, you gain:

  • 🔗 Unified visibility: Correlate synthetic test results with RUM, APM traces, and infrastructure metrics
  • 📊 Enhanced dashboards: Visualize ThousandEyes data alongside your existing Splunk observability metrics
  • 🚨 Centralized alerting: Configure alerts based on ThousandEyes test results within Splunk
  • 🔍 Root cause analysis: Quickly identify if issues are network-related (ThousandEyes) or application-related (APM)
  • 📈 Comprehensive analytics: Analyze synthetic monitoring trends with Splunk’s powerful analytics engine
  • ThousandEyes Agent Types Enterprise Agents Enterprise Agents are software-based monitoring agents that you deploy within your own infrastructure. They provide: Inside-out visibility: Monitor and test from your internal network to external services Customizable placement: Deploy where your users and applications are Full test capabilities: HTTP, network, DNS, voice, and other test types Persistent monitoring: Continuously running agents that execute scheduled tests In this workshop, we’re deploying an Enterprise Agent as a containerized workload inside a Kubernetes cluster.

  • This section guides you through deploying the ThousandEyes Enterprise Agent in your Kubernetes cluster. Components The deployment consists of two files:

    1. Secrets File (credentialsSecret.yaml) Contains your ThousandEyes agent token (base64 encoded). This secret is referenced by the deployment to authenticate the agent with ThousandEyes Cloud. apiVersion: v1 kind: Secret metadata: name: te-creds type: Opaque data: TEAGENT_ACCOUNT_TOKEN: <base64-encoded-token> 2. Deployment Manifest (thousandEyesDeploy.yaml) Defines the Enterprise Agent pod configuration with the following key settings:
  • About Splunk Observability Cloud Splunk Observability Cloud is a real-time observability platform purpose-built for monitoring metrics, traces, and logs at scale. It ingests OpenTelemetry data and provides advanced dashboards and analytics to help teams detect and resolve performance issues quickly. This section explains how to integrate ThousandEyes data with Splunk Observability Cloud using OpenTelemetry. Step 1: Create a Splunk Observability Cloud Access Token To send data to Splunk Observability Cloud, you need an access token. Follow these steps:
  • Replicating AppDynamics Test Recommendations AppDynamics offers a feature called “Test Recommendations” that automatically suggests synthetic tests for your application endpoints. With ThousandEyes deployed inside your Kubernetes cluster, you can replicate this capability by leveraging Kubernetes service discovery combined with Splunk Observability Cloud’s unified view. Since the ThousandEyes Enterprise Agent runs inside the cluster, it can directly test internal Kubernetes services using their service names as hostnames. This provides a powerful way to monitor backend services that may not be exposed externally.

  • This section covers common issues you may encounter when deploying and using the ThousandEyes Enterprise Agent in Kubernetes. Test Failing with DNS Resolution Error If your tests are failing with DNS resolution errors, verify DNS from within the ThousandEyes pod:

    Verify DNS resolution from within the pod kubectl exec -n te-demo -it <pod-name> – nslookup api-gateway.production.svc.cluster.local # Check CoreDNS logs kubectl logs -n kube-system -l k8s-app=kube-dns Common causes: