ESP32 with MQTT: clients, TLS, topics, reconnection and Home Assistant discovery

An ESP32 joins Wi-Fi, opens a TCP connection to an MQTT broker, usually wrapped in TLS, and then publishes readings to topics and subscribes to command topics. In ESP-IDF the client is Espressif’s ESP-MQTT component; in the Arduino world it is one of several MQTT libraries running over a network client; in ESPHome it is a few lines of YAML. The protocol is the same in all three. What differs is how much control you get over TLS, buffering and reconnection.

This guide assumes you know the basics of publish-subscribe from MQTT explained and have picked a chip (see Choosing an ESP32). It covers the client options, certificates, topic design, what happens when the connection drops, and how to make the device appear in Home Assistant without manual configuration.

Which MQTT client should you use?

ESP-MQTT (ESP-IDF). Espressif’s own client is the most complete option. Its documentation lists MQTT 5.0 support alongside 3.1.1, transports over plain TCP (mqtt://), TLS (mqtts://), WebSocket (ws://) and secure WebSocket (wss://), and an event-driven API: your code receives events such as connected, disconnected, data and published rather than polling. Since ESP-IDF v6.0 the client lives in its own component, added with idf.py add-dependency espressif/mqtt, as the ESP-IDF MQTT page explains.

Arduino libraries. The long-standing choice, PubSubClient, works over any Arduino network client, but its README sets clear limits: it publishes only at QoS 0, subscribes at QoS 0 or 1, caps a message at 256 bytes including header by default, and carries a notice that the library is not maintained. That is fine for a small sensor; for anything you will maintain for years, prefer an actively maintained library or move to ESP-IDF.

ESPHome. If the device exists to feed Home Assistant, ESPHome generates the firmware from a configuration file. Its MQTT component supports TLS on ESP32 and Home Assistant discovery, enabled by default. The same page notes that for Home Assistant you may prefer ESPHome’s native API, in which case MQTT is not needed at all.

How do you secure the connection with TLS?

Espressif’s examples connect to TLS brokers on port 8883, the usual port for MQTT over TLS. TLS on an ESP32 has two jobs: encrypt the traffic and prove that the device is talking to the right broker. The second job is the one that gets skipped.

In ESP-MQTT, the broker is verified in one of three ways: a PEM certificate for the broker’s CA in broker.verification.certificate, the ESP-IDF certificate bundle attached through broker.verification.crt_bundle_attach, or a global CA store. The certificate bundle is built from Mozilla’s NSS root store, offered as the full list or a smaller set of common roots, and can be updated with an OTA update. Use your own CA certificate for a private broker and the bundle for a public cloud endpoint. The documentation warns that skip_cert_common_name_check weakens TLS and opens the door to man-in-the-middle attacks; leave it off.

For the device to prove its own identity, ESP-MQTT accepts a client certificate and key in credentials.authentication, the mutual TLS setup that cloud IoT services typically require. It can also keep the key out of readable flash by using a secure element or the chip’s digital signature peripheral. On Arduino, the NetworkClientSecure class offers the same building blocks: setCACert for the broker’s CA, setCertificate and setPrivateKey for mutual TLS, or a pre-shared key.

Plan for certificates to change. A broker certificate that rotates to a new CA, or a client certificate that expires, will lock out every device that cannot be updated. Device identity, certificates and PKI covers how to provision and rotate them.

How should you design the topics?

A topic tree is an API. Once a few hundred devices publish to it, renaming is expensive, so decide the structure before the first deployment. A pattern that works for most fleets puts the site or tenant first, then the device, then the kind of message:

home/greenhouse-01/state        retained JSON with all readings
home/greenhouse-01/availability retained "online" or "offline"
home/greenhouse-01/cmd/pump     commands to the device
home/greenhouse-01/config       retained settings the device reads at boot

Three habits pay off. Publish state as one small JSON object per device rather than one topic per number, which cuts the number of messages the radio has to send. Mark state and configuration messages as retained, so a dashboard that subscribes later sees the last value immediately. Keep commands on separate topics that only the device subscribes to, so a broker access control list can allow the device to read commands and write state, and nothing else.

What happens when the connection drops?

On a real network, Wi-Fi drops, the broker restarts and routers reboot at night. The client has to recover without a person pressing reset.

ESP-MQTT reconnects automatically unless network.disable_auto_reconnect is set, waiting network.reconnect_timeout_ms between attempts, 10 seconds by default according to the ESP-MQTT documentation. Its keepalive defaults to 120 seconds. Messages sent at QoS 1 or 2 wait in an outbox in RAM until the broker acknowledges them, which survives a dropped connection but not a reboot or a power cut. If readings must not be lost, store them in flash and resend them after reconnecting.

Register a Last Will on the availability topic with the payload offline and the retain flag, then publish a retained online right after each connection. The broker publishes the will only if the device disappears without a clean disconnect. A device that wakes from deep sleep, publishes and disconnects cleanly never triggers its will, so do not use availability for it. Use an expiry on the consumer side instead: Home Assistant’s MQTT sensor has an expire_after option that marks a value stale when no update arrives in time.

Subscriptions also need care. With a clean session, the broker forgets them when the connection ends, so subscribe again in the connected event handler, not once at startup.

How does Home Assistant MQTT discovery work?

Home Assistant (see Home Assistant explained) can create entities from MQTT messages without any YAML. The device publishes a configuration message to a topic of the form <discovery_prefix>/<component>/[<node_id>/]<object_id>/config, where the prefix defaults to homeassistant, as described in the MQTT integration documentation. For a temperature reading from the topic above:

{
  "name": "Greenhouse temperature",
  "unique_id": "greenhouse-01-temp",
  "state_topic": "home/greenhouse-01/state",
  "value_template": "{{ value_json.temp }}",
  "device_class": "temperature",
  "unit_of_measurement": "°C",
  "availability_topic": "home/greenhouse-01/availability",
  "device": { "identifiers": ["greenhouse-01"], "name": "Greenhouse node" }
}

published to homeassistant/sensor/greenhouse-01-temp/config. The unique_id lets the entity be renamed in the interface, and the device block groups several entities under one device. Newer releases also accept a single device-level message that declares all of a device’s entities at once, which the documentation recommends for devices with several sensors.

Home Assistant publishes online to homeassistant/status when its MQTT integration starts. The recommended pattern is for the device to subscribe to that topic and resend its discovery messages when it sees the birth message, ideally after a short random delay so a fleet does not flood the broker. Retaining discovery messages also works, but the documentation warns that retained configurations can leave “ghost” entities behind; an empty retained payload on the configuration topic removes the entity.

Where to go next

MQTT explained covers QoS, retained messages and sessions in depth.
MQTT vs HTTP for IoT explains when a persistent MQTT connection beats plain HTTP requests.
Device identity, certificates and PKI explains how to provision and rotate the certificates used here.
Choosing an ESP32 compares the chips that run these clients.

Topics: ESP32, MQTT, TLS, Home Assistant, hardware