Debugging Race Conditions the Hard Way

We had a test suite that failed about once every twenty runs, always on CI, never locally. That kind of flakiness usually means a race condition, and this one was hiding in a background job that updated a counter without any locking.

Two workers could read the same value of counter, both increment it, and both write back the same result, silently dropping an update. Adding a proper SELECT ... FOR UPDATE around the read-modify-write sequence fixed it completely, and the flaky failures disappeared overnight.

What made this bug hard wasn't the fix, it was reproducing it. I ended up writing a small script that spawned dozens of workers hammering the same row at once, which turned a once-in-twenty-runs bug into a guaranteed failure I could actually debug.