Always-On Profiling & DB Query Performance

Always-On Profiling & Metrics

3 min

When we installed the Splunk Distribution of the OpenTelemetry Collector using the Helm chart earlier, we configured it to enable AlwaysOn Profiling and Metrics. This means that OpenTelemetry Java will automatically generate CPU and Memory profiling for the application, sending them to Splunk Observability Cloud.

When you deploy the PetClinic application and set the annotation, the collector automatically detects the application and instruments it for traces and profiling. We can verify this by examining the startup logs of one of the Java containers we are instrumenting by running the following script:

The logs will show the flags that were picked up by the Java automatic discovery and configuration:

logs 
.  ~/workshop/petclinic/scripts/get_logs.sh
We are interested in the section written by the com.splunk.opentelemetry.profiler.ConfigurationLogger or the Profiling Configuration.

We can see the various settings you can control, such as the splunk.profiler.directory, which is the location where the agent writes the call stacks before sending them to Splunk. (This may be different depending on how you configure your containers.)

Another parameter you may want to change is splunk.profiler.call.stack.interval. This is how often the system captures a CPU Stack trace. You may want to reduce this interval setting if you have short spans like the ones we have in our Pet Clinic application. For the demo application, we did not change the default interval value, so Spans may not always have a CPU Call Stack related to them.

You can find how to set these parameters here . In the example below, we see how to set a higher collection rate for call stacks in the deployment.yaml, by setting this value in the JAVA_OPTIONS config section.

yaml 
env: 
- name: JAVA_OPTIONS
  value: "-Xdebug -Dsplunk.profiler.call.stack.interval=150"