MQTT Got Your Data Off the Device. Now What?

What happens to IoT data after it leaves the broker — and how to think about the messaging layer between the edge and everything else.

MQTT won the device edge for good reasons. It is lightweight, it tolerates flaky networks, its publish/subscribe model decouples the thing producing data from the things consuming it, and it runs happily on hardware with almost nothing to spare. Two decades after Andy Stanford-Clark and Arlen Nipper designed it for oil-and-gas telemetry over satellite links, it is the default way sensors, gateways, and controllers talk to the rest of the world.

But getting data off the device is only the first hop. The harder architectural questions start after the broker: How does telemetry from a thousand sites reach the analytics platform, the historian, the ML pipeline, and the dashboards — reliably, in order, and without every consumer needing a direct connection to every broker? How do you buffer data when the link to the core goes down for four hours? How do you keep a stream durable so a consumer that was offline can catch up from where it left off?

These are not MQTT's job, and they were never meant to be. Understanding where the protocol's responsibility ends is the key to designing an IoT data architecture that holds together as it scales.

What MQTT is built to do — and what it isn't

MQTT is a transport protocol. Its contract is narrow and well-defined: move small messages from publishers to subscribers through a broker, with three quality-of-service levels to trade off between speed and delivery guarantees, retained messages so new subscribers get the last known value, and last-will notifications when a client drops. For connecting constrained devices over unreliable networks, this is close to ideal.

What MQTT deliberately does not provide is long-term storage, replay of historical messages, cross-region data movement, or stream processing. A broker is designed to keep data flowing, not to hold it. As the Unified Namespace community is fond of pointing out, using an MQTT broker as your system of record makes it slow and cumbersome — that is not what it is for. The moment your requirements include "let a consumer replay yesterday's events" or "guarantee this ordered stream survives a broker restart," you have crossed out of MQTT's problem space and into the territory of streaming and durable messaging.

This is not a criticism of MQTT. It is a boundary. Good architectures respect it.

The layer above the broker

Between the device edge and the enterprise core sits a layer whose job is data movement and durability, not device connectivity. Several distinct patterns live here, and most real deployments use more than one:

Bridging. Getting data from the MQTT broker into whatever consumes it — a historian, a time-series database, a cloud IoT service, a data lake. This is often handled by broker extensions, edge gateways, or connectors that subscribe to topics and forward messages onward.

Streaming and durability. When you need ordered, replayable, persistent streams — so a consumer can rewind, so an outage doesn't lose data, so multiple independent pipelines can read the same events at their own pace — you need a log-oriented system. Apache Kafka is the best-known example in the enterprise; there are others with different operational profiles.

Store-and-forward across the WAN. Remote sites lose connectivity. A well-designed edge buffers locally and forwards to the core when the link returns — and, critically, controls the rate of catch-up so that four hours of backed-up messages don't arrive as a single thundering herd that overwhelms downstream consumers the moment the network heals.

Fan-out to many consumers. In a mature system, the same telemetry feeds monitoring, analytics, ML, and archival simultaneously. Point-to-point integration between every producer and every consumer becomes the "integration spaghetti" that Unified Namespace advocates warn about. A shared pub/sub or streaming backbone decouples them.

Where the options diverge

There is no single right answer for this layer, and the honest tradeoffs matter more than any vendor's pitch.

MQTT-native ecosystems (brokers plus their streaming extensions and Sparkplug tooling) keep everything in one protocol family, which is operationally simple if your world is mostly devices and industrial gateways. The tradeoff is that heavy durability and stream processing are bolt-ons rather than the core design.

Log-based streaming platforms like Kafka excel at high-throughput, replayable, ordered streams and have an enormous ecosystem. The tradeoff is operational weight: they are cluster-heavy, not designed to run on a constrained edge gateway, and typically require a bridge from MQTT rather than speaking to devices directly.

Unified pub/sub-and-streaming systems such as NATS aim to cover request/reply, pub/sub, and durable streaming in one lighter-weight layer that can run from the far edge to the cloud, and some support MQTT natively so existing devices connect without a rewrite. The tradeoff is a smaller ecosystem than Kafka's and a different operational model to learn.

The right combination depends on your constraints: how many sites, how bad the connectivity, how much history you need to replay, whether the edge hardware can run anything heavier than a broker, and how many independent teams need to consume the same data.

A useful mental model

Think of it in three tiers. The device tier is where MQTT (or CoAP, or a fieldbus protocol behind a gateway) lives — optimized for constrained hardware and lossy links. The movement tier is where data is bridged, buffered, made durable, and fanned out — optimized for reliability and scale. The consumption tier is where historians, analytics, ML, and applications live — optimized for query and compute.

Most IoT architecture pain comes from asking one tier to do another tier's job: expecting a broker to be a database, expecting a cloud streaming cluster to run on a gateway, or wiring consumers directly to brokers and calling it an architecture. MQTT is excellent at the device tier. Choosing the movement tier deliberately — rather than by default — is what separates a pilot that works from a fleet that scales.

The takeaway

MQTT solved the hardest part of IoT's first mile, and it deserves its dominance at the device edge. But the protocol's job ends at the broker. The reliability, replayability, and reach of your system are decided in the layer above it — a layer worth designing on purpose. Start by writing down what you actually need after the broker: how much history, how much distribution, how much tolerance for disconnection. The answer to those three questions will tell you far more about the right messaging backbone than any protocol comparison chart.