The double refund
Patients pay a $25 deposit when they book, and get it back when they cancel. A patient on a train tapped Cancel once, yet was refunded twice. Here is what the server logged:
08:14:02.110 POST /appointments/7781/cancel request A received 08:14:02.115 appointment 7781 status = booked 08:14:07.120 request A: refund of $25 sent to payment provider 08:14:07.300 client timeout after 5 s, app retries 08:14:07.305 POST /appointments/7781/cancel request B received 08:14:07.309 appointment 7781 status = booked 08:14:08.010 request A: status set to cancelled, 200 OK (client already gone) 08:14:12.400 request B: refund of $25 sent to payment provider 08:14:12.900 request B: status set to cancelled, 200 OK
Explain exactly why the patient was refunded twice, and how you would change the endpoint so it can't happen again, even if the app retries.
Hint
Look at when each request reads the appointment's status and when it writes it. Then think about what the app could send that would let the server recognise a retry.
Worked solution
Why it happened. The endpoint checks the status, calls the payment provider, and only then marks the appointment as cancelled. The refund call took five seconds, so the app timed out and retried. Request B read the status at 08:14:07.309, while it still said "booked", because request A didn't write "cancelled" until 08:14:08.010. Both requests passed the check, so both refunded. This is a check-then-act race made worse by doing slow work before saving the state.
A fix in layers:
- Idempotency key. The app generates a unique key once per tap and sends it with every retry. The server stores the key with the result of the first request. A request with a key it has seen returns the stored result instead of doing the work again.
- Change state atomically. Move the appointment from "booked" to "cancelling" in a single conditional update (for example, update where status = 'booked'). Only the request that changes the row goes on; the other sees nothing was updated and stops.
- Make the refund itself idempotent. Payment providers accept their own idempotency key. Use the appointment and the cancellation as that key, so even a bug in our code can't refund twice.
- Don't hold the request open. Record the refund as pending and process it in the background, so the client gets a quick answer and has no reason to retry.
A good answer also adds a test that sends two identical requests at the same time and checks there is exactly one refund.