Instrument REVIEWS for Tracing
1. Use Data Setup to instrument a Python application
Within the O11y Cloud UI:
Data Management -> Add Integration -> Monitor Applications -> Python (traces) -> Add Integration
Provide the following to the Configure Integration Wizard:
Service: review
Django: no
collector endpoint:
http://localhost:4317
Environment: rtapp-workshop-[YOURNAME]
Kubernetes: yes
Legacy Agent: no
We are instructed to:
- Install the instrumentation packages for your Python environment.
pip install splunk-opentelemetry[all]
splunk-py-trace-bootstrap
Configure the Downward API to expose environment variables to Kubernetes resources.
For example, update a Deployment to inject environment variables by adding
.spec.template.spec.containers.env
like:
apiVersion: apps/v1
kind: Deployment
spec:
selector:
matchLabels:
app: your-application
template:
spec:
containers:
- name: myapp
env:
- name: SPLUNK_OTEL_AGENT
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: "http://$(SPLUNK_OTEL_AGENT):4317"
- name: OTEL_SERVICE_NAME
value: "review"
- name: OTEL_RESOURCE_ATTRIBUTES
value: "deployment.environment=rtapp-workshop-stevel"
Enable the Splunk OTel Python agent by editing your Python service command.
splunk-py-trace python3 main.py --port=8000
The actions we must perform include:
- Update the Dockerfile to install the splunk-opentelemetry packages
- Update the deployment.yaml for each service to include these environment variables which will be used by the pod and container.
- Update our Dockerfile for REVIEW so that our program is bootstrapped with splunk-py-trace
We will accomplish this by:
- generating a new requirements.txt file
- generating a new container image with an updated Dockerfile for REVIEW and then
- update the review.deployment.yaml to capture all of these changes.
2. Update the REVIEW container
Generate a new container image
Update the Dockerfile for REVIEW (
/workshop/flask_apps_finish/review
)FROM python:3.10-slim WORKDIR /app COPY requirements.txt /app RUN pip install -r requirements.txt RUN pip install splunk-opentelemetry[all] RUN splk-py-trace-bootstrap COPY ./review.py /app EXPOSE 5000 ENTRYPOINT [ "splunk-py-trace" ] CMD [ "python", "review.py" ]
Note that the only lines, in bold, added to the Dockerfile
- Generate a new container image with docker build in the ‘finished’ directory
- Notice that I have changed the repository name from
localhost:8000/review:0.01
tolocalhost:8000/review-splkotel:0.01
Ensure you are in the correct directory.
pwd
./workshop/flask_apps_finish/review
docker build -f Dockerfile.review -t localhost:8000/review-splkotel:0.01 .
[+] Building 27.1s (12/12) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 364B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/python:3.10-slim 1.6s
=> [auth] library/python:pull token for registry-1.docker.io 0.0s
=> [1/6] FROM docker.io/library/python:3.10-slim@sha256:54956d6c929405ff651516d5ebbc204203a6415c9d2757aad 0.0s
=> [internal] load build context 0.3s
=> => transferring context: 1.91kB 0.3s
=> CACHED [2/6] WORKDIR /app 0.0s
=> [3/6] COPY requirements.txt /app 0.0s
=> [4/6] RUN pip install -r requirements.txt 15.3s
=> [5/6] RUN splk-py-trace-bootstrap 9.0s
=> [6/6] COPY ./review.py /app 0.0s
=> exporting to image 0.6s
=> => exporting layers 0.6s
=> => writing image sha256:164977dd860a17743b8d68bcc50c691082bd3bfb352d1025dc3a54b15d5f4c4d 0.0s
=> => naming to docker.io/localhost:8000/review-splkotel:0.01 0.0s
- Push the image to Docker Hub with docker push command
docker push localhost:8000/review-splkotel:0.01
The push refers to repository [docker.io/localhost:8000/review-splkotel]
682f0e550f2c: Pushed
dd7dfa312442: Pushed
917fd8334695: Pushed
e6782d51030d: Pushed
c6b19a64e528: Mounted from localhost:8000/review
8f52e3bfc0ab: Mounted from localhost:8000/review
f90b85785215: Mounted from localhost:8000/review
d5c0beb90ce6: Mounted from localhost:8000/review
3759be374189: Mounted from localhost:8000/review
fd95118eade9: Mounted from localhost:8000/review
0.01: digest: sha256:3b251059724dbb510ea81424fc25ed03554221e09e90ef965438da33af718a45 size: 2412
3. Update the REVIEW deployment in Kubernetes
review.deployment.yaml must be updated with the following changes:
- Load the new container image on Docker Hub
- Add environment variables so traces can be emitted to the OTEL collector
The deployment must be replaced using the updated review.deployment.yaml
Update review.deployment.yaml (updates highlighted in bold)
apiVersion: apps/v1 kind: Deployment metadata: name: review labels: app: review spec: replicas: 1 selector: matchLabels: app: review template: metadata: labels: app: review spec: imagePullSecrets: - name: regcred containers: - image: localhost:8000/review-splkotel:0.01 name: review volumeMounts: - mountPath: /var/appdata name: appdata env: - name: SPLUNK_OTEL_AGENT valueFrom: fieldRef: fieldPath: status.hostIP - name: OTEL_SERVICE_NAME value: 'review' - name: SPLUNK_METRICS_ENDPOINT value: "http://$(SPLUNK_OTEL_AGENT):9943" - name: OTEL_EXPORTER_OTLP_ENDPOINT value: "http://$(SPLUNK_OTEL_AGENT):4317" - name: OTEL_RESOURCE_ATTRIBUTES value: 'deployment.environment=rtapp-workshop-stevel' volumes: - name: appdata hostPath: path: /var/appdata
Apply review.deployment.yaml. Kubernetes will automatically pick up the changes to the deployment and redeploy new pods with these updates
- Notice that the review-* pod has been restarted
kubectl apply -f review.deployment.yaml
kubectl get pods
NAME READY STATUS RESTARTS AGE
kafka-client 0/1 Unknown 0 155d
curl 0/1 Unknown 0 155d
kafka-zookeeper-0 1/1 Running 0 26h
kafka-2 2/2 Running 0 26h
kafka-exporter-647bddcbfc-h9gp5 1/1 Running 2 26h
mongodb-6f6c78c76-kl4vv 2/2 Running 0 26h
kafka-1 2/2 Running 1 26h
kafka-0 2/2 Running 1 26h
splunk-otel-collector-1653114277-agent-n4dfn 2/2 Running 0 26h
splunk-otel-collector-1653114277-k8s-cluster-receiver-5f48v296j 1/1 Running 0 26h
splunk-otel-collector-1653114277-agent-jqxhh 2/2 Running 0 26h
review-6686859bd7-4pf5d 1/1 Running 0 11s
review-5dd8cfd77b-52jbd 0/1 Terminating 0 2d10h
kubectl get pods
NAME READY STATUS RESTARTS AGE
kafka-client 0/1 Unknown 0 155d
curl 0/1 Unknown 0 155d
kafka-zookeeper-0 1/1 Running 0 26h
kafka-2 2/2 Running 0 26h
kafka-exporter-647bddcbfc-h9gp5 1/1 Running 2 26h
mongodb-6f6c78c76-kl4vv 2/2 Running 0 26h
kafka-1 2/2 Running 1 26h
kafka-0 2/2 Running 1 26h
splunk-otel-collector-1653114277-agent-n4dfn 2/2 Running 0 26h
splunk-otel-collector-1653114277-k8s-cluster-receiver-5f48v296j 1/1 Running 0 26h
splunk-otel-collector-1653114277-agent-jqxhh 2/2 Running 0 26h
review-6686859bd7-4pf5d 1/1 Running 0 15s