Skip to content

Performance and Sizing

Performance testing against our lab configuration produces the following results and limitations.

Tested Configurations

Splunk Cloud Noah

Environment

  • Loggen (syslog-ng 3.25.1) - m5zn.3xlarge
  • SC4S(2.30.0) + podman (4.0.2) - m5zn family
  • SC4S_DEST_SPLUNK_HEC_WORKERS=10 (default)
  • Splunk Cloud Noah 8.2.2203.2 - 3SH + 3IDX
/opt/syslog-ng/bin/loggen -i --rate=100000 --interval=1800 -P -F --sdata="[test name=\"stress17\"]" -s 800 --active-connections=10 <local_hostmane> <sc4s_external_tcp514_port>

Result

SC4S instance root networking slirp4netns networking
m5zn.large average rate = 21109.66 msg/sec, count=38023708, time=1801.25, (average) msg size=800, bandwidth=16491.92 kB/sec average rate = 20738.39 msg/sec, count=37344765, time=1800.75, (average) msg size=800, bandwidth=16201.87 kB/sec
m5zn.xlarge average rate = 34820.94 msg/sec, count=62687563, time=1800.28, (average) msg size=800, bandwidth=27203.86 kB/sec average rate = 35329.28 msg/sec, count=63619825, time=1800.77, (average) msg size=800, bandwidth=27601.00 kB/sec
m5zn.2xlarge average rate = 71929.91 msg/sec, count=129492418, time=1800.26, (average) msg size=800, bandwidth=56195.24 kB/sec average rate = 70894.84 msg/sec, count=127630166, time=1800.27, (average) msg size=800, bandwidth=55386.60 kB/sec
m5zn.2xlarge average rate = 85419.09 msg/sec, count=153778825, time=1800.29, (average) msg size=800, bandwidth=66733.66 kB/sec average rate = 84733.71 msg/sec, count=152542466, time=1800.26, (average) msg size=800, bandwidth=66198.21 kB/sec

Splunk Enterprise

Environment

  • Loggen (syslog-ng 3.25.1) - m5zn.large
  • SC4S(2.30.0) + podman (4.0.2) - m5zn family
  • SC4S_DEST_SPLUNK_HEC_WORKERS=10 (default)
  • Splunk Enterprise 9.0.0 Standalone
/opt/syslog-ng/bin/loggen -i --rate=100000 --interval=600 -P -F --sdata="[test name=\"stress17\"]" -s 800 --active-connections=10 <local_hostmane> <sc4s_external_tcp514_port>

Result

SC4S instance root networking slirp4netns networking
m5zn.large average rate = 21511.69 msg/sec, count=12930565, time=601.095, (average) msg size=800, bandwidth=16806.01 kB/sec
average rate = 21583.13 msg/sec, count=12973491, time=601.094, (average) msg size=800, bandwidth=16861.82 kB/sec
average rate = 20738.39 msg/sec, count=37344765, time=1800.75, (average) msg size=800, bandwidth=16201.87 kB/sec
m5zn.xlarge average rate = 37514.29 msg/sec, count=22530855, time=600.594, (average) msg size=800, bandwidth=29308.04 kB/sec
average rate = 37549.86 msg/sec, count=22552210, time=600.594, (average) msg size=800, bandwidth=29335.83 kB/sec
average rate = 35329.28 msg/sec, count=63619825, time=1800.77, (average) msg size=800, bandwidth=27601.00 kB/sec
m5zn.2xlarge average rate = 98580.10 msg/sec, count=59157495, time=600.096, (average) msg size=800, bandwidth=77015.70 kB/sec
average rate = 99463.10 msg/sec, count=59687310, time=600.095, (average) msg size=800, bandwidth=77705.55 kB/sec
average rate = 84733.71 msg/sec, count=152542466, time=1800.26, (average) msg size=800, bandwidth=66198.21 kB/sec

Guidance on sizing hardware

  • Though vCPU (hyper threading) was used, syslog processing is a CPU intensive task and oversubscription (sharing) of resources is not advised
  • The size of the instance must be larger than the absolute peek to prevent data loss; most sources can not buffer during times of congestion
  • CPU Speed is critical; slower or faster CPUs will impact throughput
  • Not all sources are equal in resource utilization. Well-formed “legacy BSD” syslog messages were used in this test, but many sources are not syslog compliant and will require additional resources to process.