Patterns from the prototyping era

On a constrained device, every synchronous call is an opportunity for the device to hang. The radio dropped out, the upstream service is slow, the battery sagged — any of them can stall a thread that has nowhere else to go. The pattern that survives: events flow asynchronously through queues; the device produces messages and moves on; the consumer side handles ordering, retries, and idempotency. Synchronous calls exist only where the device cannot proceed without an immediate answer, and even those are wrapped in tight timeouts and fallback paths.

The same discipline applies in production systems at scale. Tight RPC coupling between domains scales until it doesn’t; events flowing through a shared bus give every consumer room to evolve and fail independently. The intuition came from devices; it generalized to platforms.

Networks fail mid-request. The device sent the message; you do not know whether the upstream received it; the device retries. If the upstream is not idempotent, the action happened twice, and now your inventory shows two units when the user only sold one. On constrained hardware where reconnects are routine, idempotency is not optional — it is the only way to keep the system correct under realistic conditions.

The same is true at platform scale. Every state-changing operation has an idempotency key; consumers can retry safely; replays of the same message do not accumulate side effects. The pattern is invisible when it works, which is exactly the point.

On a low-power device with a slow link, "as fast as possible" is meaningless. The question is: how much time can this operation take before the user notices, the battery drops below a threshold, or the next event arrives? You declare the budget — say, 800ms end-to-end for a sensor reading round-trip — and then every component in the chain has a slice of that budget. If a component exceeds its slice, it is the component that has to be fixed; the budget is the constraint, not a wish.

Production systems benefit from the same discipline. SLOs are latency budgets stated formally. Without one, every component believes its own performance is acceptable and the user-facing experience drifts unmeasured. With one, the slow component is the one that gets attention, instead of the loudest one.

A device that loses network and goes dark is a broken device. A device that loses network and continues to do its job — buffering events, falling back to cached state, reconnecting when the link returns — is a working device. Designing for the failure modes is not a polish task; it is a primary design choice that informs every other decision. The same code path is exercised every day.

Cloud systems get to pretend that infrastructure is always available, until the day a region degrades or a dependency rolls out a bad release. The systems that handle that day well are the ones that designed for degradation from the start: cached responses when the source is unreachable, queued writes when the database is slow, partial-result UIs when one upstream is down. The discipline is the same as the embedded one; the surface is different.