Calm on-call: three habits that saved my weekends
Small rituals, big return.
A couple of years ago I had a rotation where I got paged every Saturday morning for the same alert. By the time I'd finished typing my password, the service had always recovered on its own. I was furious and very tired, which is a bad combination for making good decisions about anything, including on-call.
Three things I changed after that. None of them are clever. All of them stuck.
1. Write the runbook before the fire
Every alert I own has a runbook with three sections: what fired, first thing to check, how to fix it. I don't try to make it comprehensive. If I can get a sleepy version of myself to the right dashboard in two clicks, I've won.
The real trick isn't the runbook, it's the link. The runbook has to be attached to the alert itself. Not "in Confluence somewhere", not "ask Dave", not "it's pinned in #ops-alerts". On the page. PagerDuty has a field for it. Use it.
2. Kill flappy alerts the morning after
Anything that pages me and then resolves before I've done something goes on a list. The next morning I either raise the threshold, widen the window, or delete the thing and replace it with an alert on the real symptom. I used to leave them. "It's mostly fine." They're never mostly fine; they just train you to ignore pages, which is the worst outcome.
The test I use: would I be upset if this alert was silent for a week? If the answer is no, it shouldn't be paging me at all.
3. Tiny postmortems, even for nothing
Not the full theatrical version with a doc template and a Zoom call and action items nobody does. Five bullets in a Slack thread:
- what happened
- why we didn't catch it sooner
- smallest change that would've prevented it
- who's doing it, by when
- anything worth adding to the runbook
That's the whole thing. Sometimes it's four bullets and a shrug. It still works, because the alternative is forgetting by Thursday and tripping over the same problem six weeks later.
Six months of doing these three things, Saturday mornings got boring again, which is exactly what I wanted. On-call didn't quiet down on its own. I made it quieter, one small cleanup at a time.