AWS Network Load Balancer

If you choose an AWS Network Load Balancer (NLB) as a solution for SC4S on Amazon EKS, consider the following:

• Uneven flow-based traffic distribution: NLB load balancing is flow based, not message based. A high-volume TCP sender using one long-lived connection, or a UDP sender using the same source IP and source port, can be routed to one SC4S pod while other pods remain underused.

• Connection lifecycle behavior: Missing events can occur when a syslog sender breaks or closes TCP connections before SC4S and Splunk finish processing the traffic. This NLB Service avoids connection termination during deregistration by setting deregistration_delay.connection_termination.enabled=false.

• UDP limitations: UDP can be enabled for syslog ingestion, but it remains prone to data loss, and load balancers can introduce another point of data loss.

• Sticky sessions: This configuration keeps target group stickiness disabled. Source-IP stickiness can increase uneven distribution when many senders share the same source IP.

Note

Splunk only supports SC4S. If issues arise due to the load balancer, please reach out to the AWS support team.

Architecture

This EKS deployment path is:

flowchart LR
    subgraph Sources["Syslog Sources"]
        T["TCP syslog sender<br/>TCP 514"]
        U["UDP syslog sender<br/>UDP 514"]
    end

    subgraph AWS["AWS"]
        NLB["Internet-facing AWS NLB<br/>target type: ip<br/>TCP/UDP listeners<br/>stickiness: disabled"]
    end

    subgraph EKS["Amazon EKS Cluster"]
        SVC["Kubernetes Service<br/>type: LoadBalancer<br/>TCP/UDP 514 + healthcheck 8080"]

        subgraph SC4S["SC4S Pods"]
            P1["SC4S Pod 1<br/>syslog-ng"]
            P2["SC4S Pod 2<br/>syslog-ng"]
            P3["SC4S Pod 3<br/>syslog-ng"]
        end
    end

    SPLUNK["Splunk HEC"]

    T --> NLB
    U --> NLB
    NLB --> SVC
    SVC --> P1
    SVC --> P2
    SVC --> P3
    P1 --> SPLUNK
    P2 --> SPLUNK
    P3 --> SPLUNK

    NOTE1["Traffic allowed only from configured source CIDR"] -.-> NLB
    NOTE2["NLB distributes flows, not individual syslog messages"] -.-> SVC
    NOTE3["Long-lived TCP connections can skew ingestion distribution"] -.-> P1
    NOTE4["UDP has no retransmission; drops can appear as missing events"] -.-> P2

Set up EKS

Follow the AWS documentation to create an EKS cluster. The following example uses eksctl with an AWS CLI profile:

eksctl create cluster \
  --name <cluster-name> \
  --region <region> \
  --profile <aws-cli-profile>

Associate the IAM OIDC provider with the cluster:

eksctl utils associate-iam-oidc-provider \
  --region <region> \
  --cluster <cluster-name> \
  --profile <aws-cli-profile> \
  --approve

Create the IAM service account for the AWS Load Balancer Controller. This assumes the AWSLoadBalancerControllerIAMPolicy policy has already been created by following the AWS Load Balancer Controller installation guide.

eksctl create iamserviceaccount \
  --cluster <cluster-name> \
  --namespace kube-system \
  --name aws-load-balancer-controller \
  --attach-policy-arn arn:aws:iam::<account-id>:policy/AWSLoadBalancerControllerIAMPolicy \
  --override-existing-serviceaccounts \
  --region <region> \
  --approve \
  --profile <aws-cli-profile>

Install AWS Load Balancer Controller

This documentation assumes:

• You already have a working EKS cluster.
• kubectl and helm are configured for that cluster.
• SC4S pods are running in the sc4s namespace.
• The SC4S pods have labels matching the Service selector.

Note

The commands and manifests in this section provide an example configuration path. Update cluster names, regions, AWS profiles, IAM policy ARNs, tags, resource requests, labels, ports, source CIDR ranges, and Splunk HEC settings to match your environment and security requirements.

Refer to the AWS Load Balancer Controller installation guide for IAM and installation steps.

Install the controller:

helm repo add eks https://aws.github.io/eks-charts
helm repo update eks

helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=<cluster-name> \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

Verify the controller:

kubectl get deployment -n kube-system aws-load-balancer-controller
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller

Configure SC4S

Create the sc4s namespace:

kubectl create namespace sc4s

Create /opt/sc4s/env_file with the Splunk HEC settings:

SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=https://<splunk-hec-host>:8088
SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN=<hec-token>
SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY=yes

Create a ConfigMap from the environment file. The name must match the Deployment’s envFrom reference.

kubectl create configmap sc4s-env-config --from-env-file=/opt/sc4s/env_file -n sc4s

Deploy SC4S and the NodePort Service:

eks-sample-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sc4s-sample-linux-deployment
  namespace: sc4s
  labels:
    app: sc4s-sample-linux-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sc4s-sample-linux-app
  template:
    metadata:
      labels:
        app: sc4s-sample-linux-app
    spec:
      terminationGracePeriodSeconds: 900
      volumes:
        - name: config-volume
          configMap:
            name: sc4s-env-config
        # Uncomment only if local parser used
        # - name: local-filter-config
        #   configMap:
        #     name: sc4s-local-filter-config
      containers:
        - name: sc4s
          image: ghcr.io/splunk/splunk-connect-for-syslog/container3:latest
          imagePullPolicy: IfNotPresent
          resources:
            requests:
              cpu: "250m"
              memory: "512Mi"
            limits:
              cpu: "500m"
              memory: "1024Mi"
          env:
            - name: SC4S_RUNTIME_ENV
              value: "k8s"
          envFrom:
            - configMapRef:
                name: sc4s-env-config
---
apiVersion: v1
kind: Service
metadata:
  name: sc4s-nodeport-service
  namespace: sc4s
spec:
  selector:
    app: sc4s-sample-linux-app
  type: NodePort
  ports:
    - name: "udp514"
      port: 514
      targetPort: 514
      protocol: UDP
      nodePort: 30514
    - name: "tcp514"
      port: 514
      targetPort: 514
      protocol: TCP
      nodePort: 30514
    - name: "tcp601"
      port: 601
      targetPort: 601
      protocol: TCP
      nodePort: 30601
    - name: "tcp6514"
      port: 6514
      targetPort: 6514
      protocol: TCP
      nodePort: 30515
    - name: "healthcheck"
      port: 8080
      targetPort: 8080
      protocol: TCP
      nodePort: 30080

Apply the deployment:

kubectl apply -f eks-sample-deployment.yaml
kubectl get pods -n sc4s -o wide
kubectl get svc -n sc4s

Configure autoscaling

You can use a HorizontalPodAutoscaler for the SC4S Deployment:

hpa.yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: sc4s-autoscaler
  namespace: sc4s
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sc4s-sample-linux-deployment
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
  # Tune these values for your production scaling requirements.
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 30
      policies:
        - type: Percent
          value: 30
          periodSeconds: 15
    scaleUp:
      stabilizationWindowSeconds: 10
      policies:
        - type: Percent
          value: 80
          periodSeconds: 15
        - type: Pods
          value: 2
          periodSeconds: 15
      selectPolicy: Max

Apply the HPA:

kubectl apply -f hpa.yaml
kubectl get hpa -n sc4s

You can also install the Kubernetes Cluster Autoscaler:

helm repo add autoscaler https://kubernetes.github.io/autoscaler
helm repo update

helm upgrade --install cluster-autoscaler autoscaler/cluster-autoscaler \
  --namespace kube-system \
  --set autoDiscovery.clusterName=<cluster-name> \
  --set awsRegion=<region> \
  --set rbac.create=true \
  --set extraArgs.balance-similar-node-groups=true \
  --set extraArgs.skip-nodes-with-local-storage=false \
  --set extraArgs.expander=least-waste \
  --set serviceAccount.create=true \
  --set serviceAccount.name=cluster-autoscaler

Fine-tune NLB

Load balancer support and fine-tuning is outside the scope of the SC4S team’s responsibility. Review the AWS documentation for NLB target groups, health checks, and target group attributes before using an NLB in production.

This configuration uses the following NLB settings:

• Target type: ip
• Scheme: internet-facing
• Stickiness: disabled
• Deregistration delay: 300 seconds
• Connection termination after deregistration delay: disabled
• Access control: loadBalancerSourceRanges

The deregistration settings help avoid connection termination when a target is deregistered:

stickiness.enabled=false,deregistration_delay.timeout_seconds=300,deregistration_delay.connection_termination.enabled=false

Use internet-facing instead of internal only when syslog sources must reach SC4S over public networking and the security model allows it.

Preserving source IP

As a best practice, preserve or verify the original source IP of the sending device. Otherwise, logs that do not specify a hostname in the message may appear with the load balancer, node, or proxy IP. See the Kubernetes source IP behavior documentation for more information.

This configuration uses:

• NLB target type ip
• Source allowlisting with loadBalancerSourceRanges
• No PROXY protocol configuration
• No source-IP sticky session configuration

Verify source IP behavior in Splunk for the specific EKS, NLB, and Service configuration before relying on source-IP based parsing, host enrichment, or compliance reporting.

Configuration

Use the following Service manifest:

Set loadBalancerSourceRanges to the IP ranges that should be allowed to send validation or syslog traffic to the NLB.

Note

Treat this Service manifest as a starting point. Adjust the namespace, selector labels, allowed source ranges, tags, exposed ports, and NLB annotations according to your deployment model. Validate any change to target type, protocol mix, stickiness, or deregistration behavior before using it for production ingestion.

load-balancer-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: load-balancer-service
  namespace: sc4s
  labels:
    app: sc4s-load-balancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: "stickiness.enabled=false,deregistration_delay.timeout_seconds=300,deregistration_delay.connection_termination.enabled=false"
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "splunkit_data_classification=private,splunkit_environment_type=non-prd"
spec:
  type: LoadBalancer
  selector:
    app: sc4s-sample-linux-app
  ports:
    - name: healthcheck
      protocol: TCP
      port: 8080
      targetPort: 8080
    - name: tcp-port
      protocol: TCP
      port: 514
      targetPort: 514
    - name: udp-port
      protocol: UDP
      port: 514
      targetPort: 514
  loadBalancerSourceRanges:
    - x.x.x.x/32

Apply the Service:

kubectl apply -f load-balancer-service.yaml
kubectl get svc -n sc4s load-balancer-service -o wide
kubectl describe svc -n sc4s load-balancer-service

Validate your configuration

Get the NLB hostname:

export NLB_HOST=$(kubectl get svc -n sc4s load-balancer-service -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo "$NLB_HOST"

Send several validation TCP messages:

for i in {1..5}; do echo "nlb tcp validation $i" | nc -w2 "$NLB_HOST" 514; done

Send several validation UDP messages:

for i in {1..5}; do echo "nlb udp validation $i" | nc -u -w2 "$NLB_HOST" 514; done

Verify the results in Splunk:

• Events reached Splunk.
• The event count matches the expected validation count.
• The source host or source IP matches your deployment expectations.
• Events are routed to the expected index and sourcetype.

Operational considerations

With this AWS NLB configuration:

• Broken or closed TCP connections from a syslog sender can result in missing events.
• The Service keeps stickiness disabled and explicitly sets deregistration_delay.timeout_seconds=300 and deregistration_delay.connection_termination.enabled=false.
• NLB distributes flows, not individual syslog messages. A small number of high-volume TCP connections or consistent UDP flows can still create uneven pod utilization.
• UDP ingestion can be enabled, but missing events can occur because UDP does not provide retransmission, delivery acknowledgement, or connection-level backpressure. Validate UDP capacity and loss tolerance before production use.
• Source IP behavior must be verified in Splunk for the specific EKS, NLB, and Service configuration before relying on source-IP based enrichment.

Note

Load balancer support and fine-tuning is beyond the scope of the SC4S team’s responsibility. For assistance with AWS NLB behavior, target group attributes, health checks, or AWS Load Balancer Controller behavior, contact AWS support.