How to Answer "How do you debug a production issue?"

Backend engineer

"Real example: last year our payment webhook processing started silently dropping events at 3% — no errors, just missing revenue reconciliation. First move wasn't diagnosis, it was mitigation: I confirmed charges were succeeding at the processor, so users were fine, and I replayed the dropped events from processor logs before touching anything — impact stopped in forty minutes. Then isolate: the drop rate correlated with deploys, and deploys restart consumers. Cheapest hypothesis first: our shutdown handler was killing consumers before in-flight events acknowledged. I reproduced it exactly in staging with load plus a restart. The fix: graceful shutdown with a drain timeout, plus an at-least-once guard so a repeat of ANY cause self-heals. Verified with a chaos test that restarts consumers under load weekly. The postmortem added an alert on webhook lag — the real failure was that reconciliation, not our monitoring, caught it first. Detection gaps are the actual bug in every incident."

Recent graduate on an internship

"During my internship our internal dashboard started timing out every morning around 9 a.m., and my mentor let me own it — with training wheels. My instinct was to optimize the slow query everyone blamed, but he made me do the boring part first: reproduce and isolate. I pulled three days of APM traces and found the timeouts only hit users in one office — which killed the query theory, since queries don't care about geography. Cheapest hypothesis next: something on that office's network path. It turned out a scheduled security scan ran at 8:55 and saturated that site's VPN concentrator. My actual fix was small — rescheduling plus a timeout bump — but the lesson was the big one: my first hypothesis was confident, popular, and wrong, and only isolate-before-optimize discipline caught it. I still write my hypothesis list down before touching code."