Apache Kafka
An open-source distributed event streaming platform that stores streams of records in a replicated, partitioned log so many applications can read and replay them.
Photo: Stephen Dawson / Unsplash
Apache Kafka is a distributed log maintained by the Apache Software Foundation. Producers append records to topics, each topic is split into partitions spread across a cluster of brokers, and consumers read at their own pace by tracking an offset. Unlike a classic message queue, Kafka does not delete a record once it is read: records stay for a retention period set per topic, so a new application can start from the beginning and a fixed bug can be reprocessed from history. That makes it a common backbone for stream processing and for feeding databases and data lakes.
In IoT, Kafka usually sits behind the device layer rather than in front of it. Its clients expect stable, well-connected servers, so devices talk MQTT to a broker and a bridge or connector moves the data into Kafka, where analytics, alerting and storage each consume the same stream. Gateways on reliable networks sometimes produce to Kafka directly. The trade-off is operational weight: a cluster to run, size and secure. See The IoT data pipeline.
Part of the IoT Glossary. Spotted an error or a missing term? Tell us.