The requirements are as follows.
- To capture 10Gbps of 128B packets into Apache Kafka
- Implement basic filtering using IP addresses
- Save traffic into one Kafka topic
- Dell PowerEdge R730
- Intel Xeon CPU E5-2660 v4 @ 2.00GHz
- Intel 82599ES 10G NIC
- Hardware RAID controller PERC H730 Mini for 8 disks
- 6 SCSI disks in RAID-0 configuration
- DPDK 18.11
- Apache Kafka 2.12-2.1.0
- librdkafka v0.11.1
The following key design ideas helped to achieve 5Gbps capture speed.
- Use XFS filesystem
- Combine small packets into the big Kafka messages, 500KB each
- Run 4 Kafka brokers on one physical server simultaneously
- Allocate 20 partitions per topic
The decision was made to use two servers to be able to capture all 10Gbps traffic.
- Processing trillions of events per day with Apache Kafka on Azure
- A tale of performance debugging: from 1.3X slower to 48X faster than Apache Kafka
- A Practical Introduction to Kafka Storage Internals
- Apache Kafka Best Practices
- Benchmarking Apache Kafka: 2 Million Writes Per Second (On Three Cheap Machines)
- Tuning Kafka for Optimal Performance
- Tuning Your Apache Kafka Cluster
- Apache Kafka – linger.ms and batch.size settings
- Tuning Kafka (librdkafka client).md
- ZOOKEEPER & KAFKA INSTALL : SINGLE NODE AND MULTIPLE BROKERS
- Don’t fear the filesystem!
- Squeezing the firehose: getting the most from Kafka compression