Solo Ops Runbook for a Spiking Customer
When a customer's traffic swamps a shared stack, a solo operator should block the source, drain the backlog, and defer that customer's expensive jobs until others recover.
You are the only person on call when one customer's launch turns a shared queue into a multi-customer outage. Vemetric accepted 1,016,237 events tied to Outbid between 18:00 and 22:00 UTC.
Outbid generated $207,000 in revenue and attracted over one million visitors within 131 hours of launch. Vemetric's ingestion service and Redis shared one host, so the two consumed the same CPU and memory resources. That load arrives as ordinary requests, then becomes a queue that affects every customer on the same infrastructure.
- Block the project at the edge before more work enters ingestion.
- Drain the cheap event backlog until the dashboard can be trusted.
- Defer that project's expensive device and session jobs until other customers are healthy.
Stop the source before it creates more work
Before you block, confirm the identifier that separates the spiking customer from the rest. In a multi-tenant stack, that identifier is usually a project token, API key, or customer-specific route. If you cannot identify it quickly, the block will either miss the source or hurt other customers.
Block the spiking customer's requests at the edge before they enter ingestion. The block is working when new traffic for that customer is rejected or dropped at the boundary, while other customers' requests continue to arrive. Vemetric used a Cloudflare rule to stop ingestion requests that carried Outbid's project token. The edge rule is the cheapest control you can deploy, because it prevents work from being created in the first place.
Write the decision in the incident note: what you blocked, when, and which customers remain affected. If the block is temporary, record the condition that will remove it. A solo operator needs a written trigger, not a dashboard check.
If the spike is a bad payload rather than a volume problem, the edge rule should drop the malformed request before it reaches the worker.
Clear the cheap backlog before the dashboard is trusted
Drain the event backlog before you try to explain the incident to customers. You can trust the dashboard once the queue depth falls to normal, the delayed events appear in the product, and missing data stops. Vemetric's event backlog was cleared after about six hours, and the delayed events then showed up in the dashboard. Until that happens, every number you show is a partial number.
While the backlog drains, keep the monitoring view on queue depth, ingestion rate, and error rate. You do not need a new alert for every metric. You need a screen that tells you whether the system is moving toward normal or toward another failure.
Do not promise customers a precise recovery time until the queue depth is falling. A vague update is better than a confident time that you cannot defend. If the queue stops falling, say so and move to the next control.
Defer the expensive jobs until other customers recover
Hold the spiking customer's device and session jobs until the rest of the system is healthy. The step is done once other customers' enrichment, segmentation, or reporting jobs finish on time, while the spiking customer's heavy work sits in a known, bounded queue. The device and session queues for Outbid still had 600,000 to 800,000 jobs left, and their existing throughput would have required days to clear them. To bring other customers back first, Vemetric applied a temporary exclusion that kept device and session workers from processing Outbid's jobs.
The mistake is to treat the spiking customer's queue as the highest-priority work because it is the most visible. That queue is already blocking everyone else. The expensive jobs can wait; the other customers' service cannot.
Set an expiry on the exclusion. The exclusion is ready only with a dated note that says when the spiking customer's jobs will resume and what must be true before you remove it. If the system is healthy, re-enable the jobs in a controlled batch rather than releasing the full queue at once.
Notify the spiking customer with the same facts you would use for any customer: what was blocked, why, what is delayed, and what the next check is. Give the customer a clear state and a next step, not a blame narrative. Keep the exclusion visible in the runbook until the queue is below the normal threshold.