A change went out that was supposed to be low-risk: a small adjustment to how a certain kind of access was determined. It reviewed cleanly. It made sense on paper. And shortly after release the service stopped behaving normally.
The instinct to fix forward
The natural pull, mid-incident, is to find the specific thing that's wrong and patch it in place. Forward fixes feel more professional than reverts. A revert can look like giving up on understanding the problem.
That instinct is wrong more often than it feels like it should be, and this was one of those times.
Why revert won
Two things made the revert the obviously correct call, once we stopped and thought about it instead of reacting to it:
The whole service was affected, not one feature. There's a real difference between "this feature is behaving oddly" and "everything behind it is affected." When the problem is that wide, every minute spent understanding the specific mechanism is a minute the problem continues. A revert doesn't require understanding the bug. It only requires trusting that the previous state worked, which we already knew, because it had been working minutes earlier.
Diagnosing under pressure produces worse diagnoses. Reading a diff carefully while a service is unhealthy is a bad environment for careful reading. Reverting first bought the one resource that was actually scarce: time to think clearly, with the system already healthy again.
So we reverted, confirmed the service was healthy, and only then went looking for what had actually gone wrong.
Reverting is not the same as giving up on the idea
The part that matters more than the revert itself is what happened after. The underlying thing the change was trying to do was still worth doing. It just wasn't safe to do the way it first shipped. So the revert wasn't treated as a rejection of the idea, it was treated as a rejection of that specific implementation, under that specific amount of scrutiny.
The real version came back afterward, smaller in scope, with a clearer picture of the exact condition that had caused the problem, and with a better way to confirm the service's health.
The actual lesson
The mistake worth avoiding isn't shipping a change that breaks something. That will happen occasionally no matter how careful anyone is. The mistake worth avoiding is treating "revert immediately" as a last resort instead of a first instinct when the blast radius is the whole system.
Fast, boring, and reversible beats clever, slow, and uncertain whenever the whole service is affected.