Inference is the deployment phase of machine learning: the model is fixed, and the device feeds it fresh inputs to get outputs. Unlike model training, it needs no labels and far less compute, which is what makes it feasible on gateways and microcontrollers. Where inference runs decides much of an IoT system’s cost and behavior.
Cloud inference is easy to update and can use large models, but each call costs bandwidth and adds latency. On-device inference answers within milliseconds and works offline, at the price of model size limits and the need for an update mechanism. A smart doorbell that recognizes a face locally rather than uploading each frame is a common example. See Edge AI explained for the placement trade-offs.
Part of the IoT Glossary. Spotted an error or a missing term? Tell us.