Learning DPDK : Capture to Kafka

kafka_logo

Requirements

The requirements are as follows.

  • To capture 10Gbps of 128B packets into Apache Kafka
  • Implement basic filtering using IP addresses
  • Save traffic into one Kafka topic

Hardware

Software

Solution

The following key design ideas helped to achieve 5Gbps capture speed.

  • Use XFS filesystem
  • Combine small packets into the big Kafka messages, 500KB each
  • Run 4 Kafka brokers on one physical server simultaneously
  • Allocate 20 partitions per topic

Conclusion

The decision was made to use two servers to be able to capture all 10Gbps traffic.

References