Skip to content
Data

Batch processing

Processing accumulated data in scheduled runs, such as nightly aggregation of a day's readings, rather than continuously as events arrive.

Batch processing is the older and simpler of the two pipeline styles. Data collects in a data lake or database, and a job runs on a schedule to clean it, join it with other sources, compute daily or monthly figures and load the results into reporting tables. ETL tooling and SQL engines are built for this.

For IoT the question is latency tolerance. An energy report for a building can wait until the morning; a leak alarm cannot. Batch jobs are easier to write, test and rerun, and they handle corrections well since a job can simply be repeated over the fixed period. The usual architecture pairs batch for the historical view with stream processing for anything that needs a reaction within seconds.

Part of the IoT Glossary. Spotted an error or a missing term? Tell us.