Before you start¶
Splunk Connect for Syslog is a containerized distribution of syslog-ng with a configuration framework designed to simplify getting syslog data into Splunk Enterprise and Splunk Cloud. Our approach is to provide a runtime-agnostic solution allowing customers to deploy using the container runtime environment of choice.
Syslog is an overloaded term that refers to multiple message formats AND optionally a wire protocol for transmission of events between computer systems over UDP, TCP, or TLS. The protocol is designed to minimize overhead on the sender favoring performance over reliability. This fundamental choice means any instability or resource constraint will cause data to be lost in transmission.
When practical and cost effective (considering the importance of completeness as a requirement), place the sc4s instance in the same VLAN as the source device.
Avoid crossing a Wireless network, WAN, Firewall, Load Balancer, or inline IDS.
- When High Availability of a single instance of SC4S is required, implement multi node clustering of the container environment.
- Avoid TCP except where the source is unable to contain the event to a single UDP packet.
- Avoid TLS except where the event may cross a untrusted network.
- Plan for appropriately sized hardware
SC4S is pre-configured to map each sourcetype to a typical index. For new installations, it is best practice to create them in Splunk when using the SC4S defaults. SC4S can be easily customized to use different indexes if desired.
- em_metrics (Optional opt-in for SC4S operational metrics; ensure this is created as a metrics index)
Install Related Splunk Apps¶
Install the following:
Configure the Splunk HTTP Event Collector¶
- Set up the Splunk HTTP Event Collector with the HEC endpoints behind a load balancer (VIP) configured for https round robin WITHOUT sticky session. Alternatively, a list of HEC endpoint URLs can be configured in SC4S (native syslog-ng load balancing) if no load balancer is in place. In most scenarios the recommendation is to use an external load balancer, as that makes longer term maintenance simpler by eliminating the need to manually keep the list of HEC URLs specified in sc4s current. However, if a LB is not available, native load balancing can be used with 10 or fewer Indexers where HEC is used exclusively for syslog.
In either case, it is strongly recommended that SC4S traffic be sent to HEC endpoints configured directly on the indexers rather than
an intermediate tier of HWFs.
- Create a HEC token that will be used by SC4S and ensure the token has access to place events in main, em_metrics, and all indexes used as event destinations.
NOTE: It is recommended that the “Selected Indexes” on the token configuration page be left blank so that the token has access to all indexes, including the
lastChanceIndex. If this list is populated, extreme care must be taken to keep it up to date, as an attempt to send data to an index not in this list will result in a
400error from the HEC endpoint. Furthermore, the
lastChanceIndexwill not be consulted in the event the index specified in the event is not configured on Splunk. Keep in mind just one bad message will “taint” the whole batch (by default 1000 events) and prevent the entire batch from being sent to Splunk.
Implement a Container Runtime and SC4S¶
- Linux host with Docker (CE 19.x or greater with Docker Swarm) or Podman enabled, depending on runtime choice (below).
- A network load balancer (NLB) configured for round robin. Note: Special consideration may be required when more advanced products are used. The optimal configuration of the load balancer will round robin each http POST request (not each connection).
- The host linux OS receive buffer size should be tuned to match the sc4s default to avoid dropping events (packets) at the network level.
The default receive buffer for sc4s is set to 16 MB for UDP traffic, which should be OK for most environments. To set the host OS kernel to
match this, edit
/etc/sysctl.confusing the following whole-byte values corresponding to 16 MB:
net.core.rmem_default = 17039360 net.core.rmem_max = 17039360
and apply to the kernel:
- Ensure the kernel is not dropping packets by periodically monitoring the buffer with the command
netstat -su | grep "receive errors".
- NOTE: Failure to account for high-volume traffic (especially UDP) by tuning the kernel will result in message loss, which can be very unpredictable and difficult to detect. See this helpful discusion in the syslog-ng Professional Edition documentation regarding tuning syslog-ng in particular (via the SC4S_SOURCE_UDP_SO_RCVBUFF environment variable in sc4s) as well as overall host kernel tuning. The default values for receive kernel buffers in most distros is 2 MB, which has proven inadequate for many.
In many distributions (e.g. CentOS provisioned in AWS), IPV4 forwarding is not enabled by default. This needs to be enabled for container networking to function properly. The following is an example to set this up; as usual this needs to be vetted with your enterprise security policy:
sudo sysctl net.ipv4.ip_forward=1
To ensure the change survives a reboot edit /etc/sysctl.conf, find (or add) the text below, and uncomment as shown:
# Uncomment the next line to enable packet forwarding for IPv4 net.ipv4.ip_forward=1
Select a Container Runtime and SC4S Configuration¶
|Container Runtime and Orchestration||Operating Systems|
|MicroK8s||Ubuntu with Microk8s|
|Podman 1.7 & 1.9 + systemd||RHEL 8.x or CentOS 8.x (best option), Debian or Ubuntu 18.04LTS|
|Docker CE 18 & 19 + systemd||RHEL or CentOS >7.7 (best option), Debian or Ubuntu 18.04LTS|
|Docker CE 18 & 19 + Swarm||CentOS >7.7 (best option), Debian or Ubuntu 18.04LTS|
|Docker CE 18 & 19 + Swarm||RHEL 7.7 Deprecated|
|Docker Desktop + Compose||MacOS|
|Bring your own Envionment||RHEL or CentOS 8.1 & 8.2 (best option)|
Offline Container Installation¶
Follow these instructions to “stage” SC4S by downloading the container so that it can be loaded “out of band” on a host machine, such as an airgapped system, without internet connectivity.
- Download container image “oci_container.tgz” from our Github Page. The following example downloads v1.12; replace the URL with the latest release or pre-release version as desired.
sudo wget https://github.com/splunk/splunk-connect-for-syslog/releases/download/v1.12.0/oci_container.tar.gz
- Distribute the container to the airgapped host machine using an appropriate file transfer utility.
- Execute the following command, using docker or podman as appropriate
<podman or docker> load < oci_container.tar.gz
- Note the container ID of the resultant load
Loaded image: docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci:90196f77f7525bc55b3b966b5fa1ce74861c0250
- Use the container ID to create a local label
<podman or docker> tag docker.pkg.github.com/splunk/splunk-connect-for-syslog/ci:90196f77f7525bc55b3b966b5fa1ce74861c0250 sc4slocal:latest
- Use this local label
sc4slocal:latestin the relevant unit or yaml file to launch SC4S (see the runtime options above) by setting the
SC4S_IMAGEenvironment variable in the unit file (example below), or the relevant
image:tag if using Docker Compose/Swarm. Using this label will cause the runtime to select the locally loaded image, and will not attempt to obtain the container image via the internet.
- Remove the entry
ExecStartPre=/usr/bin/docker pull $SC4S_IMAGE
from the relevant unit file when using systemd, as an external connection to pull the container is no longer needed (or available).