Skip to content
Data

Data pipeline

The chain of stages that carries device data from ingestion through storage and processing to the applications and people who act on it.

Every IoT system has a data pipeline, whether or not anyone drew it. Devices publish, an ingestion layer receives and validates, a queue buffers, stream processing handles real-time logic, a time-series database and a lake store, batch jobs aggregate, and a dashboard or an API delivers the result. Each stage has a different scaling profile and failure mode.

Designing the pipeline explicitly pays off because the choices interact: the device’s reporting interval sets ingestion load, the retention policy sets storage cost, and the required reaction time decides whether streaming is needed at all. The IoT data pipeline walks through the stages and the decisions at each one.

Part of the IoT Glossary. Spotted an error or a missing term? Tell us.