Book Appointment

Insufficient Capacity? A CloudOps Diagnosis in Three Checks

Healthcare IT

The message on the screen was polite and misleading: “insufficient capacity.” The platform lead, who ran the analytics and scheduling platform for a regional hospital group’s back office, had seen it three times that Monday morning. Nothing about patients was involved, only a nightly reporting job that refused to start. But the message set off a familiar chain of guesses, and by lunchtime half the team believed the cloud provider had simply run out of machines. This is a made-up story, and it follows a diagnosis pattern that CloudOps teams know well: a problem that looks like one thing and turns out to be another.

The symptom: jobs that would not start

The reporting job had run quietly for a year. Each night it launched a set of temporary compute machines, crunched the day’s appointment and billing records into dashboards, and shut itself down. Then, after the group added a second analytics project, the launches began failing intermittently. Sometimes the job started at the third retry, sometimes not at all.

She wrote the problem on a whiteboard the way a clinician’s office manager might chart a recurring complaint: what happens, when, how often and what changed. Under “what changed” there were two entries. The new project had arrived the previous week, and the finance team had asked for the team’s reserved capacity to be extended for the month-end close.

First diagnosis: blame the provider

The easy theory was that the provider was short of machines in that Region. It was easy because the error text said so. But a reference called the Ops Atlas, which she had bookmarked from a conference talk, insisted on separating three failures that share a look: a genuine capacity shortfall on the provider’s side, hitting your own account’s quota, and throttling when requests come too fast. The CloudOps entry in that atlas treats the ability to tell them apart as a core skill, because each has a different cure.

She also remembered a larger event the Atlas describes, in which launch failures reported as capacity problems during a big regional outage turned out to mask a failure in the control plane itself. The lesson was not that the provider is never short of machines. It was that an error label is a clue, not a verdict.

The test: three questions, one afternoon

Instead of debating, the team ran three short checks.

  1. Does the same request succeed in another Region or machine family? It did, which weakened the provider-shortage theory but did not kill it.
  2. What does the account’s quota page say? The limit for that machine family had been nearly reached for days.
  3. Who is holding the allowance? Here the surprise appeared.

The month-end reservation that finance requested was sitting mostly unused. Reserved but idle capacity still counts against the same on-demand allowance, so the reservation was quietly eating the room that the nightly job needed. The Atlas lists this exact trap among common quota surprises. The second analytics project had added more demand on top, and the two together pushed the account over its cap on many nights.

There was a human side to the afternoon as well. Two engineers wanted to open a support ticket with the provider straight away, and one wanted to move the whole job to another Region as a precaution. The lead asked for ten minutes of evidence first, and the delay turned out to be the cheapest decision of the day. A ticket would have returned a polite answer about quotas, and a move would have carried the same cap and the same reservation along with it.

The cure: a cap raised, a reservation trimmed, a forecast started

The fix was three modest moves rather than a heroic one.

  • The lead filed a request to raise the family’s cap, with a short justification tied to the new project.
  • The team asked finance to shrink the reservation to what the month-end close genuinely used, freeing the allowance.
  • They started a monthly “runway” calculation: how many days until usage reaches the warning line, given recent growth. The Atlas shows one worked calculation of that kind, and the lead adapted its shape to her own figures.

The failures stopped that week. Nothing about the hospital group’s systems had broken; the team had mislabelled the problem, then labelled it correctly.

A second problem hiding behind the first

It is tempting to stop once the red lights go green. The lead had learned to ask a follow-up question after every fix: why did the system let this happen quietly for a week? The answer, in this case, was that nobody had been watching the headroom. The quota page existed, but no alarm was attached to it, so the first alert anyone received was a failed job.

While tidying the quota page, the team noticed that the analytics project and the scheduling platform shared one cloud account. That was why a new project could crowd out an old one. The Atlas calls a single large account an anti-pattern because quotas, the bill and the blast radius are all shared. The group did not rebuild everything. They created a separate account for analytics, copying a standard baseline of access rules and budget alerts, and left the scheduling account alone.

The team also discovered that some resources carried no owner tag. The Atlas warns that on at least one platform, unlabelled resources do not show up as violations in compliance results, so a spotless report can be hiding the real gaps. She added a monthly sweep for unlabelled items to her routine, and a short note to the finance team explaining why tags help bills make sense.

Where they looked for advice, and where they stopped

When the team wanted to understand how other organisations check outside parties, a colleague shared a piece titled evaluating overseas facilities and the weight of accreditation. The lead borrowed the spirit rather than the content: in both worlds, a claim should be backed by evidence you can look at. For the wider engineering picture, another colleague pointed to an overview of modern software delivery with DevOps, reliability engineering and cloud-native practice. None of this replaced the Atlas as a checklist, and none of it touched clinical matters, which stayed with the clinical teams.

What went into the team’s notebook

At the end of the week she summarised the lessons in the plain way she summarised everything.

  • Read an error message as a symptom. Find out which of the three failure kinds it really is before escalating to anyone.
  • Check what is already consuming an allowance before asking for a bigger one.
  • Keep an eye on the days of headroom, not just the current percentage.
  • Give each major workload its own account so one team’s growth cannot starve another’s.
  • Label resources, and sweep for missing labels on a schedule.

She also added a longer-term idea from the Atlas’s maturity ladder: aim first for foundations, then for a governed state where most accounts come from a template, and only later for measured and resilient stages. The Atlas presents that ladder as its own proposal, and the lead treated it as a roadmap to argue with, not a rulebook.

Why the story matters beyond one Monday

Hospital back offices rarely have large engineering departments. They rely on small teams who must keep scheduling, reporting and communications running on rented infrastructure. In that setting the most valuable CloudOps habit is a calm method: describe the problem precisely, separate look-alike causes, test cheaply, make the smallest correct change, then measure whether it worked. The vocabulary can come later.

The Atlas notes that this field is young and its framing is partly editorial, which is reassuring in its own way. Nobody has a final rulebook, so a team that documents its own findings is contributing to the field rather than falling behind it. The next project on her list was a rehearsal: pretend the provider’s control plane is unreachable for an afternoon, and check what the group’s most important systems can still do without it. Her prediction was that the answer would hold more surprises than the quota page did.