Book Appointment

A Hospital Records Office Tries DisasterRecoveryOps: A Fictional Log

Healthcare IT

This is an invented story, assembled from common patterns and not describing any real hospital or person. Imagine a mid-sized hospital group whose administrative technology team has just been told to take DisasterRecoveryOps seriously. Nothing clinical is involved: the systems at stake are the appointment calendar, the document-request portal and the finance back office. What follows is a dated log of how that team learned, with the numbers read analytically so you can compare them with your own.

Week one: the binder and the surprise

The team started with a thick recovery binder written years earlier. It listed servers, phone numbers and a vague promise that everything would be restored “promptly”. Their new lead asked one pointed question: how many minutes is promptly? Nobody could answer. The group therefore borrowed vocabulary from xOps School’s Ops Atlas, which frames recovery objectives as a time limit plus a point in time to return to.

For the first time they wrote two figures next to each system: how long it could stay unavailable, and how much recent activity it could afford to lose. The calendar got an hour and five minutes. The document portal got eight hours and one hour. The finance archive got three days and a day. That tiering mirrors an example in the Ops Atlas entry covering DisasterRecoveryOps, though the team chose its own numbers after talking with department heads.

Week three: the first rehearsal, and what the stopwatch said

On a quiet evening the team simulated the loss of the primary site hosting the calendar. They expected the technical failover to take most of the hour. It did not. The machines moved in about half an hour. The remainder vanished into a conference call where three managers waited for each other to speak.

An analytical reading of that evening is useful. The Atlas insists that the recovery clock includes the decision, and its worked example splits a drill into people time and systems time. Our fictional team found the same shape: roughly half the elapsed minutes were human hesitation. They had measured the wrong half for years.

The deadline they wrote on the whiteboard

Using the Atlas method, the team subtracted its slowest rehearsal time and a small buffer from the one-hour target, and ended up with a deadline for the go/no-go call. They assigned the call to the duty manager, with a named deputy, and added a rule: if the deadline arrives and the cause is still unclear, the plan is invoked anyway, because switching early costs less than missing the objective.

Week six: the lesson about copies

The second rehearsal went better on timing and worse on surprises. During the fencing step, which stops the old system from accepting new work, the team discovered that nightly export jobs and a reminder service ignored the traffic switch entirely. They kept writing to the old location. The Atlas warns about precisely this: switching routing off redirects only traffic that passes through the routing layer, so scheduled jobs, queue workers and any client holding a remembered address may carry on using the old route.

The team added an explicit fencing checklist with one line per writer, and for each line an owner who could stop it. Their administrators began to talk about who writes where, which they had never discussed before.

That discovery sits close to another Atlas idea: effective data loss is set by the slowest copy of any store the service needs, so a calendar that depends on three data stores is only as current as the laggiest of them.

Week eight: remembering that fires happen

To keep the effort from feeling theoretical, the lead read the team a real episode that the Atlas summarises. The March 2021 blaze at the French provider OVHcloud’s Strasbourg campus consumed one data centre building, and a few clients discovered that their only copies had sat on that very site next to the systems they guarded. A French court later ordered damages, and the provider is appealing. The message for a hospital back office is plain: copies kept beside the original share its fate.

Back in the fictional group’s own world, the team relocated its archive copies to a different site and then wrote a test proving they could be read from there. Accreditation reviewers and procurement officers, who often ask where records live, were given a one-page description. If your own organisation is trying to understand what independent reviewers look for, a post on why accreditation matters when comparing hospitals abroad shows how outsiders think about proof.

Week ten: scripts instead of clicks

The third rehearsal exposed a different weakness. Two engineers followed the written plan; a third, trying to be helpful, recovered a component by hand in the console. The result was a configuration nobody else had seen. The Atlas calls manual console recovery “click ops” and cites a cloud provider’s own guidance listing it as an anti-pattern.

The team decided that every step must be repeatable by script or a checklist. A practical primer on automating operations tasks with a configuration tool gave them a starting point for turning repeated steps into code.

Week eleven: the locked front door

A smaller scare came when the team rehearsed with the corporate sign-in service switched off. Nobody could open the recovery tooling, because every administrator account depended on the very directory that was supposedly down. The Atlas cites an October 2025 regional cloud outage in which sign-in itself failed, and suggests a sealed set of emergency logins stored somewhere the outage cannot reach. The team created a pair of sealed emergency accounts, wrote down who may open them and when, and checked them every month by logging in from a clean laptop.

Week twelve: the report

When the quarter closed, the team produced a one-page report with a table of tiers, drill dates, people minutes and system minutes, and a list of open gaps. The analytical lesson was not that they had become perfect. It was that they now had comparable numbers across rehearsals, so trends were visible. Decision time had halved. Fencing was complete for the calendar but still missing for the finance archive. A return-trip rehearsal, the process of moving back to the main site once it recovers, had not yet been attempted.

The Atlas treats that return trip as a second migration and recommends planning it early. The group put a date on it for the next quarter.

What an evidence-led reader should take away

  • Separate people minutes from machine minutes in every drill report.
  • Give each service its own tier instead of a single blanket promise.
  • List every process that writes data, not only the ones behind the switch.
  • Store copies in a different place than the originals, and prove they can be read.
  • Replace hand steps with scripts, then compare the plan with what ran.
  • Schedule the return trip before you need it.

The story above is illustrative only. It is not a statement about any actual healthcare provider, and nothing in it is medical, legal or investment guidance. It is a reminder that recovery discipline is built in small, measured rehearsals long before a bad night arrives.