Skip to content

Edge Processor integration guide (Experimental)

Intro

You can use the Edge Processor to:

  • Enrich log messages with extra data, such as adding a new field or overriding an index using SPL2.
  • Filter log messages using SPL2.
  • Send log messages to alternative destinations, for example, AWS S3 or Apache Kafka.

How it works

stateDiagram
    direction LR

    SC4S: SC4S
    EP: Edge Processor
    Dest: Another destination
    Device: Your device
    S3: AWS S3
    Instance: Instance
    Pipeline: Pipeline with SPL2

    Device --> SC4S: Syslog protocol
    SC4S --> EP: HEC
    state EP {
      direction LR
      Instance --> Pipeline
    }
    EP --> Splunk
    EP --> S3
    EP --> Dest

Set up the Edge Processor for SC4S

Set up on Docker / Podman

  1. On the env_file, configure the HEC URL as IP of managed instance, that you registered on Edge Processor.
  2. Add your HEC token. You can find your token in the Edge Processor “global settings” page. 
SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=http://x.x.x.x:8088
SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Kubernetes

  1. Set up the Edge Processor on your values.yaml HEC URL using the IP of managed instance, that you registered on Edge Processor.

  2. Provide the hec_token. You can find this token on the Edge Processor’s “global settings” page.

splunk:
  hec_url: "http://x.x.x.x:8088"
  hec_token: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

mTLS encryption

Prepare your certificates

Before setup, generate mTLS certificates. Server mTLS certificates should be uploaded to Edge Processor and client certifcates should be used with SC4S.

Rename the certificate files. SC4S requires the following names:

  • key.pem - client certificate key
  • cert.pem - client certificate
  • ca_cert.pem - certificate authority

Docker / Podman

  1. Use HTTPS in HEC url: SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=https://x.x.x.x:8088.
  2. Move your clients mTLS certificates to /opt/sc4s/tls/hec.
  3. Mount /opt/sc4s/tls/hec to /etc/syslog-ng/tls/hec using docker/podman volumes.
  4. Define mounting mTLS point for HEC: SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_MOUNT=/etc/syslog-ng/tls/hec.
  5. Start or restart SC4S.

Kubernetes

  1. Add the secret name of the mTLS certificates to the values.yaml file:
splunk:
  hec_url: "https://x.x.x.x:8088"
  hec_token: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  hec_tls: "hec-tls-secret"
  1. Add your mTLS certificates to the charts/splunk-connect-for-syslog/secrets.yaml file:
hec_tls:
  secret: "hec-tls-secret"
  value:
    key: |
      -----BEGIN PRIVATE KEY-----
      Exmaple key
      -----END PRIVATE KEY-----
    cert: |
      -----BEGIN CERTIFICATE-----
      Exmaple cert
      -----END CERTIFICATE-----
    ca: |
      -----BEGIN CERTIFICATE-----
      Example ca
      -----END CERTIFICATE-----
  1. Encrypt your secrets.yaml using ansible-vault encrypt charts/splunk-connect-for-syslog/secrets.yaml.
  2. Add the IP address for your cluster nodes to the inventory file ansible/inventory/inventory_microk8s_ha.yaml.
  3. Deploy the Ansible playbook ansible-playbook -i ansible/inventory/inventory_microk8s_ha.yaml ansible/playbooks/microk8s_ha.yml --ask-vault-pass