When the pager goes off, you should already know what to do.

The first fifteen minutes of an incident decide how the next five hours go. Most of the damage in a bad incident isn't the outage itself. It's the confusion: nobody sure who's driving, context scattered across three Slack threads, and the same question ('is this actually customer-facing?') asked four times by four different people.

Minute zero to two: declare it. Anyone can declare a Sev1. The cost of a false alarm is minutes, the cost of a delayed declaration is hours. Post in the incident channel with what you observed, not what you assume caused it. Minute two to five: name an incident commander. Their only job is coordinating, not fixing. If the most senior engineer is both commanding and debugging, decisions get missed.

Minute five to ten: open a dedicated war room (call or channel), pull in the on-call for every system in the request path, and start a running timeline, timestamped, in one place, updated as facts come in, not opinions. Minute ten to fifteen: post a first customer-facing status update, even if it just says 'we're investigating.' Silence past fifteen minutes is what turns an outage into a trust problem.

The template we hand clients has a fill-in-the-blanks Slack message for each of these checkpoints, plus a post-incident doc skeleton that starts capturing context while it's still fresh, instead of trying to reconstruct it two days later at the retro.

A leaked credential (an API key in a public repo, a token in a log line, a shared secret in a Slack message) is a race against whoever finds it first. The instinct is to panic-rotate everything at once, which is exactly how a security incident turns into a self-inflicted outage.

Step one is scoping, not rotating: what can this credential actually touch? Check IAM policy or scope grants before you do anything else, because that tells you the blast radius and whether this is a five-minute fix or an all-hands event. Step two is revoking access at the identity layer first (disable the key so it stops working immediately) while the replacement is prepared in parallel, not sequentially.

For zero-downtime rotation, the pattern is always: provision the new credential, deploy it alongside the old one via your secrets manager, confirm every consumer has picked it up (check logs or metrics for usage of the new key ID), then revoke the old one. Never revoke before you've confirmed the new credential is live everywhere. That's what turns a security fix into a production outage.

Close it out with an audit: pull access logs for the exposed credential's lifetime and check for any usage that wasn't yours. Most leaked-credential incidents end with 'nothing happened,' but you only know that if you actually looked.

RDS primary failures fall into two categories: the kind Multi-AZ handles for you automatically, and the kind where the instance is technically up but serving corrupted or wrong data, the far more dangerous case, because automatic failover won't save you from bad data replicated to the standby.

For an automatic failover, the runbook is mostly about verification, not action: confirm the failover completed (check the RDS event log, not just that connections recovered), validate application health against the new primary, and check replica lag on any read replicas before trusting them for reads again. Don't assume 'connections work again' means 'data is consistent.'

For point-in-time restore (the scenario that needs a human decision, not automation), the critical first move is deciding your recovery point before touching anything. Restoring creates a new instance; it doesn't overwrite the broken one, so you have a safety net by default. We always restore to a new instance, validate data integrity there, and only cut the application over once validation passes, keeping the original instance untouched in case the recovery point was wrong.

This procedure gets tested quarterly against a non-production clone, because a failover runbook nobody has actually executed is a hypothesis, not a plan. The teams we work with treat that quarterly drill as non-negotiable: it's the only way to know the restore actually completes within your RTO before you need it to.

Not every bad deploy needs the same fix, and picking the wrong one wastes the minutes that matter most. The decision tree starts with one question: is the previous version known-good and still compatible with current data/schema state? If yes, revert. If the deploy included a migration that's already run, a straight revert can break things worse than the original bug.

Revert is fastest when your pipeline supports it: re-deploy the last known-good artifact, which sidesteps 'why did the new build fail' entirely and buys time to debug without customer impact. Redeploy (rebuilding and shipping a fix) is the right call when the bad deploy is caused by something external (a bad config, a dependency version) rather than the code itself, and the fix is small and obvious.

Feature-flag flip is the move when the bad behavior is isolated to a specific code path guarded by a flag: turn it off, no deploy needed, and you've bought yourself time to fix properly without the pressure of an active incident. This is why we push clients toward flag-gating risky changes before they ship, not after the first incident makes the case for it.

Whichever path you take, the runbook doesn't end at recovery. The post-incident steps (root cause, the gap in testing or review that let it through, and one concrete change to the pipeline or process) are what actually prevent the same outage twice. Recovery without a post-incident fix just schedules the repeat.

A wave of 5xx errors behind a load balancer has four usual suspects: the LB itself, target health, the application, or a downstream dependency. Guessing wrong burns the minutes you don't have. The fastest way to triangulate is to check target health first: if targets are failing health checks, the LB is doing its job correctly and the problem is downstream of it.

If targets are healthy but requests are still failing, pull application logs filtered to the error spike's exact time window (not the last hour, the exact window) and look for a pattern: is it every request, or requests to one specific endpoint or downstream call? A single endpoint pointing to one dependency narrows the search immediately.

Downstream dependency failures are the trickiest because the application logs will show a timeout or connection error without saying why. Check the dependency's own health/status, and check for connection pool exhaustion on your side. A slow downstream call can back up your own connection pool and start failing unrelated requests, which is what makes these incidents look worse than they are.

Once you've found the layer, the fix is usually one of: scale the target group, add a circuit breaker around the failing dependency, or roll back a recent change to the affected endpoint. The guide includes a decision flowchart so an on-call engineer under pressure at 3am doesn't have to reconstruct this reasoning from scratch.

A cloud bill that doubles overnight is almost never one giant mistake. It's usually a small change with a compounding effect: a misconfigured autoscaling policy, a data-transfer path that changed, a runaway job that didn't stop, or a new feature that got far more usage than load-tested for. The first move is always Cost Explorer's day-over-day breakdown by service, not a guess.

Once you've found the service driving the spike, the second question is usage vs. price: did the workload actually grow, or is the same workload now costing more per unit? Those need completely different fixes: a genuine usage spike might be a good problem (product-market fit), while a price/configuration change is almost always a bug to fix immediately.

The immediate containment options, roughly in order of speed: cap autoscaling group maximums to stop the bleeding without taking the service down, kill any job or process that's clearly runaway, and (for data transfer surprises) check for a misconfigured route sending traffic cross-region or cross-AZ instead of within a zone.

The controls that stop it from happening the same way twice: budget alerts with anomaly detection (not just a monthly threshold, which fires too late), autoscaling maximums set deliberately rather than left at defaults, and a tagging policy that lets you attribute spend to a team or feature before the bill arrives, not after.