Scaling

Scaling SOC4Kafka¶

To handle higher throughput, you can deploy multiple instances of the SOC4Kafka. Kafka natively supports partition-based scaling, allowing multiple consumers within the same consumer group to share the workload.

Steps to scale horizontally¶

Configure Kafka partitions¶

1. Ensure the Kafka topics you are consuming from are partitioned appropriately.
2. The number of partitions should match or exceed the number of collector instances to ensure even distribution.

Use the same consumer group¶

Configure all SOC4Kafka collectors to use the same group_id. Kafka ensures that each partition is consumed by only one collector instance within a consumer group.

receivers:
  kafka:
    brokers: ["localhost:9092"]
    logs:
      topics:
        - "example-topic"
      encoding: "text"
    group_id: <GROUP ID>

exporters:
  splunk_hec:
    token: "your-splunk-hec-token"
    endpoint: "https://splunk-hec-endpoint:8088/services/collector"
    source: my-kafka
    sourcetype: kafka-otel
    index: kafka_otel
    splunk_app_name: "soc4kafka"
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 10000
      block_on_overflow: true
      sizer: items
      batch:
        min_size: 1000

service:
  pipelines:
    logs:
      receivers: [kafka]
      exporters: [splunk_hec]