TinyML on the ESP32: frameworks, the S3's vector instructions and the memory budget

You run TinyML on an ESP32 with one of four toolchains: Espressif’s port of LiteRT for Microcontrollers (the former TensorFlow Lite Micro), Espressif’s own ESP-DL framework, the ESP-SR speech library for wake words and voice commands, or Edge Impulse as a hosted workflow that exports an ESP-IDF or Arduino library. The chip matters as much as the framework. The ESP32-S3 has vector instructions that Espressif’s kernels use, which makes it the usual choice; the ESP32-P4 is the step up for vision; the classic ESP32 and the RISC-V C-series run small models, more slowly.

TinyML: machine learning on microcontrollers covers the general workflow, from data collection to an int8 model in firmware. This guide is the ESP32-specific part: which chip, which framework, how much memory is really available, and what kinds of tasks work.

Which ESP32 is best for TinyML?

The deciding feature is SIMD. The ESP32-S3 datasheet describes processor instruction extensions with 128-bit registers and 128-bit vector operations, added to “improve the operation efficiency of specific AI and DSP” algorithms. ESP-NN, Espressif’s library of optimized neural network kernels, has assembly versions for the S3 that use those instructions, SIMD-optimized versions for the ESP32-P4, and only generic optimizations for the classic ESP32 and the ESP32-C3. It covers the operations small networks spend their time in: convolution, depthwise convolution, fully connected layers, pooling and softmax.

Espressif publishes the effect in the esp-tflite-micro repository, using the person detection example. With ESP-NN, one inference takes 54 ms on the ESP32-S3 at 240 MHz, 73 ms on the ESP32-P4 at 360 MHz (where the repository notes optimization is still in progress), 380 ms on the classic ESP32 at 240 MHz and 426 ms on the ESP32-C3 at 160 MHz. Without ESP-NN the same inference takes more than a second on every chip. These are Espressif’s figures for one model; your own model will differ, but the ranking is a fair guide.

So the S3 is the default for audio, vibration and low-resolution vision models. The P4, with its 400 MHz dual core, camera and display interfaces and no radio, suits vision products that pair it with a wireless companion chip, as described in Choosing an ESP32. A C3 or C6 is fine for small models that run rarely, such as an anomaly score computed once a minute.

Which framework should you use?

LiteRT for Microcontrollers (esp-tflite-micro). Google’s LiteRT for Microcontrollers needs no operating system support and no heap, and its core runtime fits in 16 KB on an Arm Cortex-M3; the page lists ESP32 boards among the supported targets. Espressif’s esp-tflite-micro component adds the ESP-NN kernels and ships the hello_world, micro_speech and person_detection examples. Choose it when your model comes from TensorFlow or Keras, or when you want code that also runs on non-Espressif microcontrollers. It supports only a subset of TensorFlow operations and no on-device training.

ESP-DL. ESP-DL is Espressif’s own inference framework for its chips. Models are quantized with the ESP-PPQ tool from ONNX, so PyTorch and TensorFlow models go through ONNX first, to 8-bit, 16-bit or mixed precision, and stored in a .espdl format based on FlatBuffers. Its model zoo includes object detection with YOLO11n, MobileNetV2 classification and, on the P4, text recognition. Choose it for vision on the S3 or P4, the chips its benchmarks and model zoo target.

ESP-SR. For speech, ESP-SR bundles an audio front end (echo cancellation, noise suppression, voice activity detection), the WakeNet wake word engine and the MultiNet command recognizer, which handles up to 300 Chinese or English commands. WakeNet runs on most of the family, including the C3, C5 and C6 through variants for chips without PSRAM or SIMD; MultiNet needs the S3, P4 or S31.

Edge Impulse. Edge Impulse covers data collection, feature design, training and export. For ESP32 it produces a C++ library for ESP-IDF, an Arduino library or ready firmware for the ESP-EYE, and it uses Espressif’s ESP-DSP library to speed up feature extraction such as MFCC for audio. It suits teams that want one tool from raw data to a deployable library.

How much memory is there for the model?

Less than the datasheet suggests. The ESP32-S3 has 512 KB of on-chip SRAM, but the Wi-Fi and TLS stacks, FreeRTOS, network buffers and your application all live there too. The model’s activations need a contiguous working buffer, the tensor arena in LiteRT, and that is what usually runs out first.

PSRAM is the relief valve. The S3 is sold with up to 16 MB of PSRAM in the package, and modules add more options. Large arenas, camera frames and audio buffers can go there, at the cost of slower access than internal SRAM, so buffers on the critical path are better kept internal. Without PSRAM, keep the model small. The micro_speech model in the tflite-micro examples is less than 20 kB.

Weights live in flash memory, next to the firmware. If the device receives OTA updates, the partition table needs at least two application slots, so the model counts twice against the flash budget unless it sits in a separate data partition.

What tasks work well on an ESP32?

Keyword spotting. The micro_speech example recognizes “yes” and “no” with a model under 20 kB and is the right place to learn the pipeline. For a product wake word, ESP-SR’s WakeNet and audio front end deal with echo and noise that a demo model ignores.

Anomaly detection on vibration or current. An accelerometer on I2C or SPI, a window of samples, an FFT from ESP-DSP, then a small autoencoder or classifier that outputs a score. The device publishes the result, not the raw signal, which turns a stream of samples into one small MQTT message:

{"ts": 1767225600, "rms": 0.42, "anomaly": 0.07, "state": "normal", "model": "pump-v3"}

Including the model version in the payload makes it possible to tell later which firmware produced which scores. The predictive maintenance guide covers the signal processing in more depth.

Simple vision on the S3 or P4. Person or object presence detection at low resolution, counting, or a reading from a dial. Anything that needs fine detail or many classes belongs on a larger processor; see On-device computer vision.

How does inference fit a battery budget?

Inference is a wake phase like any other: the energy is the active current times the inference time, so the per-inference figures above translate directly into battery life. Two techniques keep the cost down. Let the ULP coprocessor or LP core watch a cheap signal, such as a vibration threshold, and wake the main cores only when there is something to classify. And run the model before the radio, so that a “normal” result can be stored and sent later in a batch. ESP32 on batteries covers the power modes and the estimation method.

Where to go next

TinyML: machine learning on microcontrollers explains the general workflow and constraints.
Choosing an ESP32 compares the S3, P4 and the rest of the family.
ESP32 on batteries shows how to budget the energy of each wake.
Predictive maintenance with IoT and AI puts on-device anomaly scores into a maintenance process.

Topics: TinyML, ESP32, ESP32-S3, TensorFlow Lite, ESP-DL, Edge Impulse