A Complete Postmortem of an MQTT Auto-Discovery Failure

Q: Devices suddenly vanished from HA — how do I confirm it is an MQTT auto-discovery problem?

Look for MQTT discovery errors in the HA log ('Invalid discovery', 'Payload is not JSON', topic conflicts). Then run mosquitto_sub -t '#' -v to capture actual topics and check whether the device is still publishing. If it publishes but HA does not recognize it, the fault is in the discovery topic or payload format, not the device going offline.

Q: How do I quickly tell 'topic drift' from 'device offline'?

Run mosquitto_sub -t '#' -v for 30 seconds and see whether the device's original topic (for example zigbee2mqtt/<device>) still carries messages. If the prefix changed (for example zigbee2mqtt/<device>/ became zigbee2mqtt/<device>/state, or a firmware update changed the base topic), it is drift. If that topic is silent, the device is offline or the broker never got the message.

Q: Why do retained messages cause duplicate names or ghost entities?

MQTT retained messages are stored by the broker and replayed to every new subscriber. If the old config or state retained messages are never cleared, HA sees the old entity and the new one side by side after a reconnect, or the same name twice. mosquitto_sub -t 'homeassistant/#' -v shows the replayed retained config; clearing it removes the ghost.

Q: Why did devices vanish after a firmware update?

Many device or gateway firmware upgrades change the MQTT base topic or the discovery format. Common cases: zigbee2mqtt changes its discovery prefix or the device's topic structure after an update, so the old topic HA subscribed to stops carrying messages. Check the firmware changelog or capture '#' to see the new topic, then adapt the HA config.

Q: Besides topic conventions, what should I monitor on the MQTT broker itself?

Subscription count and message rate. A drop in subscriptions (mass device offline) or a spike (reconnect storm) surfaces broker or gateway trouble early; a sudden message-rate drop means devices went silent. Watch $SYS/broker/clients/connected and $SYS/broker/publish/messages/received and alert on thresholds.

This reconstructs one representative class of "devices in Home Assistant suddenly vanished or duplicated, and the discovery topic drifted." Your devices might use a different protocol or broker, but the diagnostic ladder and the traps are universal: capture the topics to see whether the device is still publishing, compare the discovery topic with the device's real topic, and do not let retained leftovers fool you. The single most useful habit in this class of failure is to reach for mosquitto_sub before you reach for the gateway — it converts a guess into a fact in under a minute.

Summary

In the morning, a batch of Zigbee devices suddenly vanished from the HA panel, while a few others showed duplicate entities. The devices were powered and the gateway was online, but the entities were either gone or duplicated. A single mosquitto_sub -t '#' -v settled it — the devices were still publishing, but their topic prefix had changed, so the old discovery topic HA had subscribed to stopped carrying messages and HA decided the devices were offline.

Impact

The vanished devices could not be controlled (switches and states stopped updating), and every automation depending on those entities silently failed.
The duplicate entities caused HA to generate conflicting entity IDs, so some control actions targeted the wrong device.
Roughly 1.5 hours from discovery to restoring the discovery subscription.
Because the devices "looked alive" and the gateway LED was normal, it was easy to misread the whole thing as a gateway fault.
A second-order effect: the automations built on those entities (like lights turning off on leave, or temperature-linked scenes) all went quiet, so the user-facing symptom was "automation seems flaky" rather than an obvious hardware error.

Timeline

Time	Event
T+0	Users report a batch of devices missing from the HA panel
T+15	Devices confirmed powered and gateway online; hardware offline ruled out
T+30	`mosquitto_sub -t '#' -v` captures topics; devices are still publishing
T+45	Discovery topic compared with real device topic; a prefix drift is confirmed
T+60	Traced to a gateway firmware update that changed the base topic; adaptation begins
T+80	HA config updated, stale retained cleared, discovery subscription restarted
T+90	Device entities restored and duplicates removed; postmortem closes

Root cause

The real trigger was a gateway firmware update that changed the MQTT base topic: after the update, the device's publish topic drifted from zigbee2mqtt/<device> to zigbee2mqtt/<device>/state and the discovery prefix changed too, so the auto-discovery that subscribed to the old topic never received another config/state message. Two amplifiers turned it into a visible failure:

Retained leftovers: the old config (homeassistant/.../config) retained messages were never cleared, so a HA reconnect replayed the old entities, producing duplicates and ghosts alongside the new ones.
Discovery subscription conflict: the auto-discovery topic and the manually configured fixed topic overlapped, and after the firmware change they no longer aligned, so HA fed some entities the wrong payload.

The decisive clue (and the watershed of the whole postmortem): mosquitto_sub -t '#' -v showed the devices publishing, but HA did not recognize them — the fault was in the discovery/subscription chain, not the devices.

Diagnostic traps that actually cost time

What slowed recovery was not "couldn't find it" — it was blaming hardware or the gateway. Real lessons:

Capture topics before touching the gateway. The first reflex when devices vanish is "the gateway is broken." Run mosquitto_sub -t '#' -v for 30 seconds first; if the device is publishing, it is not hardware.
Do not trust the HA panel. "Entity vanished" only means HA stopped receiving or recognizing it, not that the device is offline. Distinguish "device not publishing" from "HA not recognizing" — capturing topics is the only way.
Retained replays old config. homeassistant/# retained messages replay on every subscriber reconnect, resurrecting old entities as ghosts. Clear retained thoroughly, or they keep coming back.
Firmware updates are a common trigger. Device or gateway upgrades often change the topic structure or discovery format. Record the current topics before upgrading and diff them after — drift jumps out instantly.
Duplicates are not always a bug. Same-name entities can also come from both a manual config and auto-discovery creating one device, or retained and live messages coexisting. Check the payload source before deleting entities.

Fix

Climb the ladder, capture topics first:

# 1) Capture every topic to see whether the device is still publishing (-v shows topic and payload)
mosquitto_sub -t '#' -v -C 30
# device topic has messages -> it is publishing, not hardware offline

# 2) Capture the auto-discovery topic and compare config/state
mosquitto_sub -t 'homeassistant/#' -v -C 30
# does the discovery topic line up with the device's real topic?

# 3) Check for retained leftovers (replayed on reconnect)
mosquitto_sub -t 'homeassistant/#' -v -C 30 --retained-only 2>/dev/null || \
mosquitto_sub -t 'homeassistant/#' -v -C 30   # if no retained-only flag, watch the replay

For immediate containment, clear stale retained + restart the discovery subscription so HA rediscovers:

# Clear old discovery retained with an empty payload
mosquitto_pub -t 'homeassistant/sensor/<old_device>/config' -n -r
# Confirm it is gone
mosquitto_sub -t 'homeassistant/#' -v -C 10
# Trigger rediscovery: restart HA's MQTT integration or HA itself

For the permanent fix, unify the topic prefix + adapt the config + version control. Check the firmware changelog for the new base topic, update the HA config to match, and establish a topic-naming convention:

# zigbee2mqtt example: pin an explicit prefix and discovery config
# in HA configuration.yaml, hardcode the base topic to avoid silent drift
mqtt:
  discovery: true
  discovery_prefix: homeassistant

Prevention

Three things this incident exposed, each with a guard:

Topic naming convention: pin the base topic and discovery prefix; record current topics before a firmware upgrade and diff after; add "topic change" to the upgrade checklist.
Retained cleanup policy: build a "stale cleanup" for discovery/state topics and periodically clear retained messages no longer in use, so ghosts cannot accumulate.
MQTT broker monitoring: monitor subscription count ($SYS/broker/clients/connected) and message rate ($SYS/broker/publish/messages/received), alerting on a mass drop (devices offline) or a reconnect-storm spike.
Configuration versioning: put HA's MQTT/zigbee2mqtt config under version control (git) so you can diff before and after an upgrade and roll back instantly.
Failure drill: periodically simulate a topic drift and verify the capture-and-clear-and-restart flow completes within the expected window.

Lesson

When devices vanish, capture the topics first — mosquitto_sub -t '#' -v tells you whether the device is still publishing; if it is, it is not hardware. The diagnostic order is always: capture every topic → compare the discovery topic → check for retained leftovers → adapt to the new prefix. The thing that is easiest to miss is retained leftovers: they replay old config on reconnect and manufacture duplicates and ghosts.

The most valuable takeaway is simple: if you monitor the wrong object, you always react half a beat late. We used to "watch the HA panel," which only signaled once entities vanished. Now we "monitor MQTT subscription count and message rate," which surfaces a topic drift the moment it begins. Moving the monitor one layer up shortens MTTR more reliably than writing another alert rule — that, more than any single command, is what this incident taught me.

It also reinforced that the cheap diagnostic is usually the right one: one mosquitto_sub -t '#' -v told us more about where the fault lived than any dashboard. When devices vanish, capture the topics first, adapt the prefix second, and only then declare the incident resolved. The whole recovery hinged on a single command that costs nothing and takes seconds — which is exactly why the habit of "capture before you blame" is worth more than any monitoring rewrite.

Related reading: [HA MQTT Auto-Discovery../../homeassistant/home-assistant-mqtt-auto-discovery.md) · [MQTT Setup../../homeassistant/mqtt-setup.md) · [Zigbee2MQTT Gateway Config../../homeassistant/zigbee2mqtt-gateway-config.md)