Skip to content
Nasir Nasir-Ameen
Notes

eventOS

eventOS: what should happen when a system you depend on goes quiet

A fast internal lookup layer that verification leaned on went down. The interesting design question wasn't how to prevent that. It was deciding, in advance, what "safe but degraded" actually means.
4 min read

Some parts of a platform lean on a fast internal layer to avoid doing expensive work on every request. A shortcut that's usually there, and usually fast, right up until the moment it isn't.

Verification was one of those parts. When that shortcut went quiet, the interesting question wasn't "how do we stop this from happening". Dependencies go down, that's not a solvable problem, only a schedulable one. The real question was: what is this part of the system allowed to do when its shortcut isn't there to ask?

Two bad defaults

Left undecided, a dependency going quiet tends to resolve itself into one of two bad defaults, neither of which anyone actually chose on purpose:

Fail closed everywhere, by accident. The lookup doesn't answer, so the code that was relying on it throws, and now something that was never about that dependency. An unrelated verification path, a login, a routine check. Breaks because it happened to touch code that assumed the shortcut would always be there.

Fail open everywhere, by accident. The opposite mistake: something catches the error, shrugs, and lets the request through as if verification passed, because nobody explicitly decided what "I couldn't check" should mean, and "let it through" was simply what happened when nothing else did.

Both are defaults nobody would choose if you asked them directly. Both happen anyway when "what do we do if this is unavailable" isn't answered before the dependency actually goes down.

Deciding the answer on purpose

So we made it a deliberate decision instead of an accidental one: which specific checks are allowed to degrade to a safer, slower fallback path when the shortcut isn't available, and which ones are not allowed to proceed at all without it. Because for those, an unclear answer is worse than a slow one.

That distinction is the actual design work. It's not "add a try/catch." It's sitting down and being honest about which decisions in the system are cheap to get slightly conservative about, and which decisions are the kind you'd rather stall on than guess at.

Why this is worth writing down

The dependency coming back is not the interesting part of this story. Dependencies always come back. The interesting part is that "safe but degraded" turned out to need its own definition, per decision, instead of one blanket rule for the whole system. Some things should get slower when their shortcut disappears. A few things should simply refuse to guess.

Deciding that in advance, instead of discovering it live while a dependency is down, is the entire difference between a graceful degradation and an accidental one.