Edge Processor integration guide (Experimental)¶
Intro¶
You can use the Edge Processor
to:
- Enrich log messages with extra data, such as adding a new field or overriding an index using
SPL2
. - Filter log messages using
SPL2
. - Send log messages to alternative destinations, for example,
AWS S3
orApache Kafka
.
How it works¶
stateDiagram
direction LR
SC4S: SC4S
EP: Edge Processor
Dest: Another destination
Device: Your device
S3: AWS S3
Instance: Instance
Pipeline: Pipeline with SPL2
Device --> SC4S: Syslog protocol
SC4S --> EP: HEC
state EP {
direction LR
Instance --> Pipeline
}
EP --> Splunk
EP --> S3
EP --> Dest
Set up the Edge Processor for SC4S¶
SC4S using same protocol for communication with Splunk and Edge Processor. For that reason setup process will be very similar, but it have some differences.
- On the
env_file
, configure the HEC URL as IP of managed instance, that you registered on Edge Processor. - Add your HEC token. You can find your token in the Edge Processor “global settings” page.
SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=http://x.x.x.x:8088
SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY=no
- Set up the Edge Processor on your
values.yaml
HEC URL using the IP of managed instance, that you registered on Edge Processor. - Provide the hec_token. You can find this token on the Edge Processor’s “global settings” page.
splunk:
hec_url: "http://x.x.x.x:8088"
hec_token: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
hec_verify_tls: "no"
mTLS encryption¶
Before setup, generate mTLS certificates. Server mTLS certificates should be uploaded to Edge Processor
and client certifcates should be used with SC4S
.
Rename the certificate files. SC4S requires the following names:
key.pem
- client certificate keycert.pem
- client certificateca_cert.pem
- certificate authority
- Use HTTPS in HEC url:
SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=https://x.x.x.x:8088
. - Move your clients mTLS certificates (
key.pem
,cert.pem
,ca_cert.pem
) to/opt/sc4s/tls/hec
. - Mount
/opt/sc4s/tls/hec
to/etc/syslog-ng/tls/hec
using docker/podman volumes. - Define mounting mTLS point for HEC:
SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_MOUNT=/etc/syslog-ng/tls/hec
. - Start or restart SC4S.
- Add the secret name of the mTLS certificates to the
values.yaml
file:
splunk:
hec_url: "https://x.x.x.x:8088"
hec_token: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
hec_tls: "hec-tls-secret"
- Add your mTLS certificates to the
charts/splunk-connect-for-syslog/secrets.yaml
file:
hec_tls:
secret: "hec-tls-secret"
value:
key: |
-----BEGIN PRIVATE KEY-----
Exmaple key
-----END PRIVATE KEY-----
cert: |
-----BEGIN CERTIFICATE-----
Exmaple cert
-----END CERTIFICATE-----
ca: |
-----BEGIN CERTIFICATE-----
Example ca
-----END CERTIFICATE-----
- Encrypt your
secrets.yaml
:
ansible-vault encrypt charts/splunk-connect-for-syslog/secrets.yaml
-
Add the IP address for your cluster nodes to the inventory file
ansible/inventory/inventory_microk8s_ha.yaml
. -
Deploy the Ansible playbook:
ansible-playbook -i ansible/inventory/inventory_microk8s_ha.yaml ansible/playbooks/microk8s_ha.yml --ask-vault-pass
Scaling Edge Processor¶
To scale you can distribute traffic between Edge Processor managed instances. To set this up, update the HEC URL with a comma-separated list of URLs for your managed instances.
Update HEC URL in env_file
:
SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=http://x.x.x.x:8088,http://x.x.x.x:8088,http://x.x.x.x:8088
Update HEC URL in values.yaml
:
splunk:
hec_url: "http://x.x.x.x:8088,http://x.x.x.x:8088,http://x.x.x.x:8088"