The post-incident checklist that stops retry storms from becoming outages

Your on-call page says an endpoint is slow; the next page is from a client retrying it. In the GitHub.com outage on 17 August 2026, the service stayed degraded for 7 hours and 47 minutes. In Central US, an Istio sidecar pod reached its concurrency limit while GitHub's autoscaler measured the host service rather than the sidecar. At peak, web and API errors ran about 20 percent, and archive and raw repository download failures ran about 50 percent.

Scale the component that fills up

An autoscaler follows a metric. If it follows the wrong metric, capacity grows while the saturated component stays full. A host service can look healthy while the sidecar in front of it is already refusing work. The post-incident note should name the saturated component, not the symptom. If the autoscaler does not watch that component, the fix is a metric, not more capacity.

The autoscaling blind spot is the difference between the component that saturates and the metric the scaler watches. A dashboard that only shows request volume can hide a queue that is already full. The on-call lead should be able to point to the metric that would have paged earlier. The postmortem should record the missing metric, the metric that should have been watched, and the owner who will add it.

Cap client-created traffic

A retry storm adds load. During the same incident, GitHub's Copilot Token Service rose from a normal 7,000 to 9,000 requests per second to 70,000 to 100,000 requests per second, a roughly tenfold increase. GitHub identified a possible VS Code retry flaw, set off by sluggish replies from a single internal endpoint, as a contributor to higher load and slower recovery. Pausing all four saturated HAProxy load-balancer nodes at once produced immediate broad service recovery.

A retry budget caps the extra work a client may create. Without a cap, one slow endpoint can pull the whole system into extra work. The budget should be small enough to protect the service and large enough to preserve useful retries. Backoff should spread retries over time, and a circuit breaker should stop the client from continuing to call a failing path. Name the client that amplified the traffic, even if the client is outside the company's codebase.

Track shipped clients as production dependencies

A client that leaves the company's codebase does not stop being part of the system. It can retry, cache, batch, or fail in ways the service team did not design. The ops lead should keep a client inventory that records the client name, the version in use, the retry behavior, and the team that can change it.

That inventory belongs next to the service dependency map, not in a spreadsheet that only appears during an incident. When a client is old, say so in the postmortem. When a client cannot be patched quickly, the service should assume it will retry and design for that. A missing client record is a gap in the dependency map, not a paperwork problem.

Run the checklist in the postmortem

The checklist belongs in the incident postmortem, not in a chat thread. It should be short enough to run in the same meeting where the team decides what changed, and each item should be testable quickly.

• Name the saturated component. Write the exact service, pod, queue, or endpoint that filled up.
• Check the autoscaler metric. Confirm the scaling policy reads the saturated component, not a healthy parent service.
• Set a retry budget. Give every client a maximum retry count, backoff, and circuit breaker.
• List shipped clients as production dependencies. Record version, retry behavior, and who can patch it.

A shipped client failure is a production failure. The team cannot hotfix a client it does not control, so the postmortem should treat that client like a service with an owner, a version, and a known failure mode. If the client retries without a budget, the service must be designed to absorb that load.