A team lead I will call Marco inherited his hospital network’s automation the way most people inherit anything operational: without a map. It is a textbook AutomationOps case, minus the label. The presenting complaint was narrow — a nightly archival job had been failing on Tuesdays for two months, and nobody could say why. By the time we finished looking, the Tuesday failure turned out to be the least of it. The estate held thirty-one scheduled jobs, eleven with no identifiable owner, one service account with administrator rights on every machine in the building, and at least four scripts doing the same task slightly differently. Diagnosis before treatment is the whole game in medicine, and it is the whole game here too. The Ops Atlas’s AutomationOps walkthrough makes the same argument from the engineering side: automated actions are patients, and most of what goes wrong is detectable long before the emergency. What follows is the case worked the way a clinical audit would run it — complaint, causes, prescription, follow-up.
The Presenting Complaint
Estates like Marco’s rarely arrive with one dramatic failure. They arrive with a syndrome, and the signs repeat across organisations so consistently they deserve a laminated poster:
- Jobs that fail quietly and are discovered by accident, months later.
- Nobody able to say who owns a given script, or what happens if it is switched off.
- A shared login used by many jobs, known to many people, rotated never.
- Duplicate tools for one task, each slightly wrong in its own way.
- A team afraid to modify anything, because nothing is testable.
Each item looks minor on its own. Together they describe an estate where nobody is in charge, which is the actual disease. The Tuesday archive failure was a symptom; the missing ownership was the condition.
Differential Diagnosis
Work the causes back and the pattern sharpens. Scripts living in personal folders belong to people, not to the organisation — when the person leaves, the knowledge leaves, and the job keeps running on inherited permissions. That is governance debt accruing interest. The all-powerful shared account exists because separating identities felt like overhead at the time; the bill arrives later, with compound interest, in the form of actions nobody can attribute or contain. Jobs written as blind gestures rather than state-claims misbehave on their second run, which is why they are always “fixed” by hand after every failure. And the absence of any rehearsal path means every change is a live experiment on a working system. None of this is a technology failure. It is an ownership failure wearing a technology costume.
The AutomationOps Prescription, in Five Lines
The corrective plan is short enough to fit on one card, and the Atlas spells out each line in detail. Every automated job in the estate should carry five properties before it is trusted. It can show its intended work without executing, though a careful operator verifies that the preview actually exercises the dangerous lines, because some check modes skip raw commands. It refuses to proceed when its target list arrives empty — an empty set must abort the run, because on at least one famous occasion an unpopulated list was interpreted as permission to continue across everything. It works in small groups of targets with a health check between groups, never the entire fleet in one motion. It stops itself once its errors cross a small threshold, on the principle that a job failing repeatedly is gathering information, not making progress. And it can be halted instantly, per job and across the whole estate, by any engineer on shift. Prescriptions only work when taken; each of these five needs a test that proves it, not a belief that it holds.
Triage Before Treatment
Not every patient needs the intensive care unit, and not every job needs every control. Sensible programmes grade their automation by consequence. Read-only checks — a status probe, a report — sit in the observation ward: safe, no special precautions. Reversible actions against a single machine get basic precautions and a named owner. Reversible actions across many machines graduate to the full battery: preview mode, staged groups, health gates. The fourth grade is the one that concentrates the mind — actions that destroy data or cannot be undone. Those demand a second qualified person who signs off before the run fires, a soft-delete pathway wherever the storage layer offers one, and a staged introduction even under time pressure. Security auditors will recognise the shape at once: two-person sign-off on destructive operations is a formal control in the major compliance catalogues, catalogue numbers included. Automation governance did not invent the idea; it finally applied it to cron jobs.
The Malpractice Case on the Syllabus
Every discipline has the case study it teaches first. Automation’s is from 2012: a trading firm in the United States pushed new code into production alongside old code that was supposed to be dormant, with no human having reviewed what would actually execute. The leftover logic bought stock at volumes nothing intended, and in about three-quarters of an hour the firm had bled away some 460 million dollars. There was no attacker. There was no exotic bug. There was an automated action, in production, with nobody holding the leash — which is the same sentence you could write about the archival job deleting the wrong year of scans. The lesson is not “avoid automation”. The lesson is that reach without review is the hazard, in a hospital and on a trading floor alike.
AutomationOps Numbers on the Follow-Up Chart
Treatment that ends at discharge is not treatment, and automation work that ends at “it works now” is the field’s most common relapse. A mature programme keeps a handful of numbers on the chart. What fraction of runs complete without a human rescue? How often does an automated action itself cause an incident, adapted from delivery research’s change-failure measure? How frequently do the guardrails fire — a number that should be small but never zero, because a brake that never engages is probably disconnected? And what proportion of jobs have a live human being attached as their owner? That last figure is the one Marco’s team watches now; it started near a third and sits past ninety percent, which is the single most predictive number on the page. For teams that prefer a sequence to a scorecard, the Atlas prescribes a simple ninety-day arc: count and register the twenty riskiest jobs first, concentrate on one runner with proper identities, drill the stop lever, then expand with evidence in hand.
When the Right Answer Is Not to Automate
A diagnosis service also includes knowing when to send the patient home with advice rather than a procedure. The Atlas sketches a sequence of gates worth stealing outright. Is the task genuinely repeated, and is it understood well enough to be written down? If neither, the honest answer is documentation, not code. Would the automated version be easy to remove if it misbehaved? If removal is unthinkable, fix the thing at its source instead of papering over it. Does the task need human judgement mid-flight — a whiff of something off, a value that looks wrong? Then build the human a steering position, not a spectator seat. Finally, does the repetition actually repay the building and upkeep, or is consistency the only real prize? Both are legitimate reasons, but they justify different levels of investment, and confusing them is how bloated, half-loved tooling happens. Walking the gates costs ten minutes. Skipping them costs quarters.
What Good Looks Like at Discharge
Cancer care teaches the same discipline this whole subject is circling. Good centres build individualised treatment plans because protocol without context is how errors compound, and an automation estate is individualised the same way — graded by consequence, not governed by blanket rules. Patients are coached to arrive with the right questions after a diagnosis; automation owners should ask identical ones about every job in the fleet. Who owns this. What does it touch. How do we stop it. What does it cost to maintain. What is the plan when it fails. And because no estate is an island, the whole discipline slots into the broader family of operations practices catalogued in this overview of modern IT operations. Marco’s Tuesday job? Fixed in an afternoon, once somebody finally owned it. The eleven orphans were harder, and mattered more. Diagnose the estate, not the symptom. Then write the plan down, and put a name on every line.