Running a local LLM on a Raspberry Pi: memory, quantization, llama.cpp, Ollama and the AI HATs

Yes, within limits. A Raspberry Pi 5 can run small large language models entirely on its CPU with llama.cpp or Ollama, provided the model is quantized to fit in its RAM, which the Raspberry Pi 5 offers in sizes from 1 to 16 GB. Models of a few billion parameters at 4-bit precision fit on an 8 or 16 GB board; generation is slow enough that the Pi suits short, non-urgent tasks rather than a chat that feels instant. The vision-oriented AI HAT+ does not help with language models at all; the newer AI HAT+ 2 does.

This guide explains how to tell what fits, what quantization trades away, how the two common runtimes differ, what Raspberry Pi’s AI accelerator boards actually accelerate, and where a local model earns its place in an IoT system. It does not quote speed figures: they depend on the model, the quantization, the context length and the software version, and the only reliable number is the one you measure on your own board.

What fits in a Raspberry Pi’s memory?

Start with arithmetic. A model’s weights take roughly its parameter count times the bits per weight, divided by eight, in bytes. A model with 1 billion parameters needs about 2 GB at 16-bit precision, about 1 GB at 8 bits and about half a gigabyte at 4 bits, plus some overhead for the scaling factors that quantization stores and for the layers often kept at higher precision.

Weights are not the whole bill. During inference the runtime also holds a key-value cache that grows with the context length, so a long prompt, such as a day of log lines, needs memory on top of the weights. And the Pi is probably doing other jobs: an operating system, an MQTT broker, perhaps Home Assistant. Size the model to what is left after those, with some headroom, not to the total on the box.

Memory also sets the speed. To produce each token, a CPU runtime reads essentially all of the weights from RAM, so generation speed is limited by memory bandwidth more than by the number of cores, and a larger model is proportionally slower. That is why the realistic choice on a Pi is a small model, quantized, with a short context.

What does quantization cost?

Quantization stores each weight in fewer bits. The llama.cpp project supports integer quantization at 1.5, 2, 3, 4, 5, 6 and 8 bits, and stores models in GGUF, a single-file format that carries the weights and everything needed to load them.

Fewer bits means a smaller file, less memory and faster generation, at the cost of accuracy. The loss grows as the bit width shrinks, and small models, with less redundancy to absorb it, tend to suffer more than large ones. For an IoT task the test is practical: run your own prompts through two or three quantization levels and compare the answers, rather than relying on a generic benchmark.

llama.cpp or Ollama?

Both run quantized models on the Pi’s 64-bit Arm CPU. They differ in how much they decide for you.

llama.cpp is, in the words of its README, a plain C/C++ implementation without dependencies. It gives the most control: you pick the exact GGUF file, the number of threads and the context size. Its llama-server exposes an OpenAI-compatible HTTP API, so existing client code can point at the Pi.

Ollama packages model download, storage and serving behind one command, with an ARM64 Linux build. Its HTTP API listens on port 11434 of the local machine by default, as its FAQ explains, and can be opened to the network with a setting. Home Assistant’s Ollama integration uses that API as a conversation agent; it can control devices exposed to Assist only with models that support tools, and its documentation suggests exposing fewer than 25 entities.

Choose llama.cpp when you want to tune every parameter or embed inference in your own program; choose Ollama when you want a service that other software on the network can call.

What do the AI HAT+ and AI Kit accelerate?

Raspberry Pi sells add-on boards with Hailo neural processing units, and they are often assumed to speed up language models. Most of them do not.

According to the AI HAT+ documentation, the AI HAT+ comes in a 13 TOPS version built on the Hailo-8L and a 26 TOPS version built on the Hailo-8. Both use the Pi 5’s own memory and target vision networks such as object detection; LLM and vision-language model support is listed as not supported. The older AI Kit, a Hailo-8L on the M.2 HAT+, is the same class of accelerator and, per the Raspberry Pi AI documentation, is no longer in production.

The AI HAT+ 2 is different. It carries a Hailo-10H rated at 40 TOPS (INT4) and has 8 GB of memory of its own, and the documentation says it runs LLMs and vision-language models of up to about 6 billion parameters. Models are served through hailo-ollama, a REST service from Hailo’s generative AI model zoo. Because the model lives in the HAT’s memory, the Pi’s RAM stays free for the gateway’s other work. All of these boards are for the Pi 5 only.

TOPS figures describe peak arithmetic for a given precision, not language model speed, which still depends on memory and on how well a model has been compiled for the accelerator. Check that the model you need is available for the HAT before buying it.

Where does a local LLM help in IoT?

A private voice assistant. Home Assistant’s local voice pipeline runs speech recognition and speech synthesis on your own hardware. Its documentation reports that Speech-to-Phrase, which recognizes a fixed set of commands, transcribes in under a second on a Raspberry Pi 4, while the open-ended Whisper model takes around 8 seconds there. A local LLM fits behind that pipeline as a fallback for requests the fixed commands do not cover, not as the first stage.

Summaries of logs and events. A nightly job can feed the day’s alarms, door events or error messages from the broker to a small model and publish a short summary. Speed matters little when the answer is needed in the morning, and the data never leaves the site.

Turning free text into structure. Maintenance notes, ticket descriptions or operator comments can be classified or tagged locally before they enter a database.

It is the wrong tool for control loops, safety decisions and anything that needs a guaranteed correct answer. A language model can produce confident errors, so keep it advisory, with deterministic automations doing the switching. For the agent patterns that do act on devices, see AI agents for the smart home; for automations that run without any cloud service, see Local smart home automation.

When should the model run somewhere else?

If answers must arrive in a second or two, or the task needs a model larger than a few billion parameters, put the model on a machine with more memory bandwidth on the same network, such as a small PC or a GPU box running Ollama, and let the Pi call it. The data still stays on site, and the Pi can keep doing what it does well as a Raspberry Pi gateway.

Where to go next

Edge AI explained places local LLMs among the other tiers of on-device AI.
AI agents for the smart home covers what happens when a model is allowed to act.
On-device computer vision covers the workloads the vision-only AI HAT+ is built for.

Topics: LLM, Raspberry Pi, edge AI, quantization, llama.cpp, Ollama