Edge Processor integration guide (Experimental)¶
Intro¶
You can use the Edge Processor
to:
- Enrich log messages with extra data, such as adding a new field or overriding an index using
SPL2
. - Filter log messages using
SPL2
. - Send log messages to alternative destinations, for example,
AWS S3
orApache Kafka
.
How it works¶
stateDiagram
direction LR
SC4S: SC4S
EP: Edge Processor
Dest: Another destination
Device: Your device
S3: AWS S3
Instance: Instance
Pipeline: Pipeline with SPL2
Device --> SC4S: Syslog protocol
SC4S --> EP: HEC
state EP {
direction LR
Instance --> Pipeline
}
EP --> Splunk
EP --> S3
EP --> Dest
Set up the Edge Processor for SC4S¶
Set up on Docker / Podman¶
- On the
env_file
, configure the HEC URL as IP of managed instance, that you registered on Edge Processor. - Add your HEC token. You can find your token in the Edge Processor “global settings” page.
SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=http://x.x.x.x:8088
SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Kubernetes¶
-
Set up the Edge Processor on your
values.yaml
HEC URL using the IP of managed instance, that you registered on Edge Processor. -
Provide the hec_token. You can find this token on the Edge Processor’s “global settings” page.
splunk:
hec_url: "http://x.x.x.x:8088"
hec_token: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
mTLS encryption¶
Prepare your certificates¶
Before setup, generate mTLS certificates. Server mTLS certificates should be uploaded to Edge Processor
and client certifcates should be used with SC4S
.
Rename the certificate files. SC4S requires the following names:
key.pem
- client certificate keycert.pem
- client certificateca_cert.pem
- certificate authority
Docker / Podman¶
- Use HTTPS in HEC url:
SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=https://x.x.x.x:8088
. - Move your clients mTLS certificates to
/opt/sc4s/tls/hec
. - Mount
/opt/sc4s/tls/hec
to/etc/syslog-ng/tls/hec
using docker/podman volumes. - Define mounting mTLS point for HEC:
SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_MOUNT=/etc/syslog-ng/tls/hec
. - Start or restart SC4S.
Kubernetes¶
- Add the secret name of the mTLS certificates to the
values.yaml
file:
splunk:
hec_url: "https://x.x.x.x:8088"
hec_token: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
hec_tls: "hec-tls-secret"
- Add your mTLS certificates to the
charts/splunk-connect-for-syslog/secrets.yaml
file:
hec_tls:
secret: "hec-tls-secret"
value:
key: |
-----BEGIN PRIVATE KEY-----
Exmaple key
-----END PRIVATE KEY-----
cert: |
-----BEGIN CERTIFICATE-----
Exmaple cert
-----END CERTIFICATE-----
ca: |
-----BEGIN CERTIFICATE-----
Example ca
-----END CERTIFICATE-----
- Encrypt your
secrets.yaml
usingansible-vault encrypt charts/splunk-connect-for-syslog/secrets.yaml
. - Add the IP address for your cluster nodes to the inventory file
ansible/inventory/inventory_microk8s_ha.yaml
. - Deploy the Ansible playbook
ansible-playbook -i ansible/inventory/inventory_microk8s_ha.yaml ansible/playbooks/microk8s_ha.yml --ask-vault-pass