Six alerts at 3 a.m.
It's 03:10 in New York and you are on shift in Manila. Six alerts are waiting. Put them in the order you would handle them, and say what you would do first for each.
- A reporting server at a credit union is 85% full and filling by about 1% a day.
- On a hospital billing workstation, a word processor has started a command shell, which is now connecting to an unfamiliar domain.
- A health system's prior authorization service has returned errors for 30% of requests for the last 6 minutes.
- The security certificate for a county's public records portal expires in 9 days.
- One broker-dealer user account has had 400 failed sign-ins in 5 minutes, from many different addresses.
- Last night's backup failed for one non-production test environment.
Hint
Ask two questions of each alert: is harm happening right now, and how quickly does it get worse if nobody acts? Some alerts need one quick action before a longer investigation.
Worked solution
A sensible order: 2, 3, 5, 1, 4, 6.
- 2 first, but briefly. A document starting a shell that reaches out to an unknown domain is a classic sign of a malicious macro, and ransomware spreads in minutes. Isolate the workstation from the network (one action in the endpoint tool), then open an incident. Containment takes seconds; the investigation can continue in parallel.
- 3 straight after, as a live incident. Clinicians are failing to get approvals now, which delays care. Open a major incident, page the service's on-call engineer, check for recent changes, and update the client. In a real team, 2 and 3 would be split between two people at once; saying so is a good answer.
- 5 next. Many failed sign-ins from many addresses against one account is an attack on that account. Check whether any attempt succeeded, make sure multi-factor authentication is enforced, and block or lock as the client's runbook allows. Most likely the defences are holding, but a success would move this to the top.
- 1, then 4, then 6. At 1% a day the disk has about two weeks; raise a ticket for the client's team. The certificate has 9 days; schedule the renewal and confirm an owner. A failed backup of a test environment matters, but nothing is at risk tonight; rerun it and note it for the day shift.
Whatever order you choose, write down why. The handover note is how the next shift knows what was done and what wasn't.