How much error budget is left?
A client-facing service has a target that 99.9% of requests succeed over a 30-day month. It receives a steady 2,000 requests a minute. It is now day 20 of the month, and there have been three incidents:
- A full outage lasting 12 minutes.
- 90 minutes during which 20% of requests failed.
- A bad deploy that made 1% of requests fail for 6 hours before it was rolled back.
The team wants to run a database migration this week. If it goes wrong, they expect 30 minutes of 50% errors before they could roll back. Should they go ahead?
Hint
Count the budget in failed requests, not minutes. Then compare what is left with the worst case of the migration.
Worked solution
The budget. A 30-day month has 43,200 minutes, so 43,200 × 2,000 = 86,400,000 requests. 0.1% of that is 86,400 failed requests.
Used so far.
- Outage: 12 × 2,000 = 24,000.
- Partial errors: 90 × 2,000 × 0.2 = 36,000.
- Bad deploy: 360 minutes × 2,000 × 0.01 = 7,200.
Total 67,200, which is about 77.8% of the budget, with 19,200 failed requests left. Two-thirds of the way through the month, the service has used more than three-quarters of its budget.
The migration. The worst case is 30 × 2,000 × 0.5 = 30,000 failed requests, more than the 19,200 left. If it goes wrong, the service misses its target.
So: not as planned. Either wait until the new month, or reduce the blast radius. For example, move 10% of traffic first: the same failure would then cost 30 × 200 × 0.5 = 3,000 failed requests, well within what is left. Faster rollback would help too. A strong answer also notes that the error budget is a tool for a conversation with the client and the product team, not just a number.