Operational Resilience: The 20-Minute Fix That Took 2 Hours and 40 Minutes

TL;DR

• A 20-minute Kubernetes fix turned into a 2-hour-40-minute outage: MTTD was 1 minute, MTTA was 38 minutes, and MTTR took the rest.
• UK regulators require mapping 5 dependency types: people, process, technology, facilities, and information, and testing "severe but plausible" scenarios, not just tabletop drills.
• July 13, 2026: HM Treasury designated AWS, Google Cloud, Microsoft, and Oracle as the UK's first Critical Third Parties, placing cloud concentration risk under direct regulatory oversight.
• MTTD, MTTA, and MTTR fail for 3 different reasons. Fixing the wrong phase wastes effort on a problem that was never there.
• Firefly closes the Technology gap with a live Inventory and Relationships graph, and now codifies PagerDuty on-call schedules and escalation policies too.

Three of the most-cited recent outages weren't obscure infrastructure failures. AWS, Azure, and Cloudflare have all had public incidents that took down large parts of the internet, and 2025 also saw major cyberattacks hit household names like Jaguar Land Rover, M&S, and the Co-op. The UK's Financial Conduct Authority now cites these specific incidents in its own guidance as exactly the kind of "severe but plausible" scenario firms are expected to plan for, not because they're rare, but because they keep happening to organizations that assumed they had a plan.

Operational resilience is the discipline built around a hard truth: technology fails; that part is unavoidable, so the real differentiator is whether an organization keeps delivering what its customers actually depend on while the technology underneath it is on fire. This piece covers what that requires in practice: breaking recovery time into MTTD, MTTA, and MTTR instead of treating it as one number, the five dependency types regulators require mapped, how manual mapping differs from automated discovery, building an incident command structure with defined roles before an incident forces the question, testing severe-but-plausible scenarios through chaos engineering rather than just tabletop discussion, why concentration risk is now a directly supervised regulatory concern rather than a theoretical one, and where infrastructure visibility fits into making any of it real instead of a document nobody checks during an actual incident.

What is Operational Resilience?

Operational resilience is an organization's ability to continue delivering its critical business services when disruptions affect the people, processes, technology, or third parties that support them, not just its ability to bring a specific server or database back online.

Disaster recovery asks: can we restore this system? Operational resilience asks a broader question: can we keep serving customers while parts of the underlying system are broken, degraded, or missing? A platform team can fail over a region cleanly in Terraform; compute, database replicas, and load balancers all come up healthy, terraform plan shows zero drift, and the service is still down. The cause never touched Terraform: an external partner's API allowlist still points at the old region's egress IPs, a Route 53 health check still targets the old ALB, and a TLS certificate provisioned in the old region's ACM never replicated. The infrastructure recovered. The dependency chain around it didn't, because "recover the infrastructure" and "recover the service" aren't automatically the same checklist.

UK financial regulators formalized this distinction into specific, enforceable requirements. The FCA's PS21/3 and the PRA's SS1/21, in force since March 2022 with a transition deadline of March 2025, require regulated firms to identify their "important business services," set a maximum tolerable disruption for each, and map every person, process, technology, facility, and information dependency the service actually relies on. That framing turns "be resilient" from a vague aspiration into a checkable set of practices and applies well beyond financial services.

The UK isn't unique here; it's just the most specific about it. Australia's CPS 230 requires the same kind of critical-operations mapping and tolerance testing, which has been mandatory since July 2025. The EU's DORA covers similar ground for financial entities directly. US banking regulators recommend nearly identical practices in joint interagency guidance, without the same binding enforcement structure. The vocabulary shifts by jurisdiction: "important business services" in the UK, "critical operations" in Australia, but the underlying requirement, know what you can't afford to lose and prove you can survive losing it anyway, shows up everywhere regulators have looked at this seriously.

Setting an impact tolerance only means something if there's an honest way to measure whether an organization actually meets it, and total incident duration turns out to be a worse measurement than it looks.

Breaking Down Recovery Time Into MTTD, MTTA, and MTTR

Total incident duration isn't one number; it's three distinct phases stacked together, and each one breaks down for a completely different reason. Treating "recovery time" as a single metric hides which phase is actually broken and, as a result, which fix would actually help.

Apply this to the earlier scenario: the Kubernetes node pool failing at 2 AM. Detection was fast; the monitoring caught it within a minute, so MTTD wasn't the problem. Acknowledgment was where the first real damage occurred: the on-call engineer was paged but didn't own the service, so MTTA stretched to nearly 40 minutes while they found the right person across three Slack channels. Resolution then split in two: the actual Kubernetes fix took twenty minutes once the right engineer was engaged, but the runbook's stale load balancer reference added another hour of confusion on top of it, inflating MTTR far past what the technical fix alone required.

This is why "we improved our recovery time" is a meaningless claim without saying which phase improved and why. A team that adds more monitoring improves MTTD. A team that fixes on-call routing and ownership clarity improves MTTA. A team that keeps runbooks current and dependency maps accurate improves MTTR. Most organizations have a bottleneck concentrated in one of these three phases, not spread evenly across all three, and fixing the wrong one wastes effort on a phase that was never actually the problem.

The Five Things Regulators Require You to Map

FCA and PRA guidance is specific about what "mapping" actually covers, and the specificity is useful even for organizations that aren't directly regulated by it. Every important business service needs its dependencies documented across five categories.

Most operational resilience failures trace back to a gap in one of these five categories that nobody had documented, not a failure of the underlying technology itself. The next two sections go deeper into the two categories that cause the most trouble in practice: Technology, because it's the hardest to keep accurate by hand, and Process. After all, it's the category most likely to only exist as tribal knowledge.

The Technical Mechanics of Dependency Mapping

Mapping "technology" dependencies sounds like a documentation exercise, but done properly, it's closer to a discovery problem, and the two approaches to solving it produce very different results.

Manual mapping starts with an architecture diagram or a spreadsheet, usually built during an initial audit or compliance push. Someone interviews engineering teams, asks what each service depends on, and documents the answers. This captures what people believe is true at the time of the interview, which is meaningfully different from what's actually running. Manual maps miss dependencies nobody thought to mention, resources created after the last update cycle, and drift that happened between the mapping exercise and today.

Automated discovery takes a different starting point: instead of asking people what they think is connected, it observes what's actually connected. This appears in several technical forms, depending on the layer being mapped. Distributed tracing tools (OpenTelemetry being the most common open standard) capture actual request paths across services, revealing runtime dependencies that a static diagram would never show, such as a service quietly calling another service's internal API that nobody documented. Service mesh tooling (Istio, Linkerd) provides a live topology view of which services actually talk to which, enforced at the network layer rather than assumed from documentation. At the infrastructure layer, a continuously scanned cloud asset inventory reveals dependency chains between resources, an EC2 instance's actual VPC, security group, IAM role, and attached storage, regardless of whether anyone remembered to draw that relationship on a diagram.

The gap between these two approaches is exactly where the earlier 2 AM scenario originated: a manual map said the checkout service depended on Database A. The actual, current dependency, discovered only during the incident, was a second database that got added during a migration and never made it back into the documentation. Automated discovery would have caught that the moment it happened. A manual map catches it only at the next audit cycle, if at all.

Building an Incident Command Structure Before You Need One

The Process category in the mapping table above is where most of the coordination failure in the earlier scenario occurred, and it's worth breaking down what "process" means in practice during an active incident rather than leaving it abstract.

A functioning incident command structure assigns specific roles before an incident starts, borrowed loosely from the incident command systems used in emergency response: an Incident Commander who owns the overall response and has explicit authority to make calls without escalating every decision, a Communications Lead who owns external and internal status updates so they don't scatter across multiple channels, and Subject Matter Experts pulled in for the specific systems involved, rather than whoever happens to answer a page first. Without these roles predefined, every incident spends its first several minutes figuring out who's in charge, which is exactly the MTTA problem from earlier, compounding under pressure.

The other half of process mapping is the decision rights question: who has the authority to declare an incident customer-facing, who can approve an emergency change outside the normal review process, and who can authorize communicating with customers or regulators about an ongoing disruption. These sound like governance questions, but they have direct technical consequences. An engineer who has the technical fix ready but lacks the authority to deploy it outside a change window introduces a delay unrelated to technical capability. Defining these decision rights in advance and ensuring the right people have the access and permissions required by the role removes an entire category of delays that otherwise surface mid-incident.

Testing Severe But Plausible Scenarios, Not Just Convenient Ones

"Severe but plausible" is the specific standard UK regulators use for scenario testing, and the phrase is doing real work. A scenario doesn't need to be apocalyptic to qualify; it needs to be one that could genuinely happen and would seriously strain the organization's ability to remain within its impact tolerances.

Testing methodology splits into two broad categories, and mature programs use both rather than picking one. Tabletop exercises are discussion-based: a facilitator walks a team through a hypothetical scenario, "the primary AWS region is unreachable, walk me through what happens next." The team talks through their response without touching real systems. These are cheap to run and good for testing processes and decision rights, but they can't catch a stale runbook or an undocumented dependency, since nothing actually breaks.

Chaos engineering and game days go further by actually breaking something, in a controlled way, and watching what happens. Tools like AWS Fault Injection Simulator or Gremlin can terminate an instance, inject network latency, or block traffic to a specific dependency, on a schedule, with a defined blast radius and a rollback plan if things go worse than expected. A minimal AWS FIS experiment template for the identity provider scenario below looks like this:

{
  "description": "Block egress to identity provider, observe downstream cascade",
  "targets": {
    "checkout-service-sg": {
      "resourceType": "aws:ec2:security-group",
      "resourceTags": { "Service": "checkout" },
      "selectionMode": "ALL"
    }
  },
  "actions": {
    "block-idp-egress": {
      "actionId": "aws:network:disrupt-connectivity",
      "parameters": {
        "duration": "PT10M",
        "scope": "availability-zone"
      },
      "targets": { "Subnets": "checkout-service-sg" }
    }
  },
  "stopConditions": [
    { "source": "aws:cloudwatch:alarm", "value": "arn:aws:cloudwatch:us-east-1:account:alarm:checkout-error-rate" }
  ]
}

The stopConditions block is the part worth paying attention to: it automatically halts the experiment if a real alarm fires past a defined threshold, which is what keeps a deliberate 10-minute blast radius from turning into an actual incident. A well-run game day might simulate exactly the kind of concentration risk scenario regulators now care about: block egress to the organization's identity provider and watch how many downstream services fail in ways nobody predicted, since IdP outages tend to cascade in ways a tabletop discussion rarely anticipates. The output of a good game day isn't a passing grade; it's a list of dependencies and assumptions that turned out to be wrong, which then feeds back into updating the dependency map from the previous sections.

A cloud region outage is severe but plausible; AWS, Azure, and Cloudflare have all demonstrated it in the last two years. A ransomware attack that hits both production and backup systems simultaneously is severe but plausible, exactly what happened to several major retailers in 2025. The test isn't whether an organization can survive a scenario it already knows how to handle. It's whether it can survive one that combines two failures at once, since that's usually how the worst incidents actually unfold.

Concentration Risk: When Everyone Depends on the Same Third Party

On July 13, 2026, HM Treasury designated Amazon Web Services, Google Cloud, Microsoft, and Oracle as the UK's first Critical Third Parties, placing all four under direct resilience supervision by the Bank of England, the PRA, and the FCA. A huge share of the UK financial sector runs on a small number of cloud providers, and a serious disruption at any one of them could ripple across banks, insurers, and financial market infrastructure simultaneously, a single point of failure sitting underneath organizations that believed they'd diversified.

The UK isn't acting alone on this. DORA gives EU supervisory authorities the same power under a different name, Article 31 lets them directly designate and oversee "Critical ICT Third-Party Providers," and the EU's own list is expected to converge on the same handful of hyperscale cloud providers the UK just named. Two of the largest regulatory blocs in the world independently arrived at the same conclusion: a small number of cloud providers now sit beneath enough of the financial system that the providers themselves need direct oversight, not just the firms that use them.

Designation doesn't remove a regulated firm's own responsibility. Firms still have to do their own third-party risk assessment, maintain contingency plans, and demonstrate they can operate within impact tolerances even if a critical provider goes down, the same obligations that existed before the CTP regime, now paired with direct oversight of the providers themselves. For a technical team, the practical takeaway is the same whether or not the organization is directly in scope of UK or EU financial regulation: know exactly which parts of the business depend on a single vendor with no fallback, and treat that dependency with the same seriousness as an internal single point of failure, because from an operational resilience standpoint, that's exactly what it is.

This is also where multi-cloud architecture and operational resilience become the same conversation from two different angles. An organization that runs entirely on a single cloud provider for every critical service faces a concentration risk that no amount of internal redundancy can fix, since that redundancy is often built on the same underlying provider. Diversifying isn't free, and it isn't always the right call for every workload. Still, the decision should be made deliberately, with a clear view of which business services would be affected if that one provider had a bad day, rather than discovered for the first time during an actual outage.

A concentration limit can be enforced directly as a policy rather than left as a design principle that nobody checks. A Guardrail Policy Rule restricting where a critical service's compute can land looks like this:

package firefly.guardrails.concentration

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_instance"
  resource.change.after.tags.Service == "checkout"
  count([r | r := input.resource_changes[_]
             r.type == "aws_instance"
             r.change.after.tags.Service == "checkout"
             r.change.after.availability_zone == resource.change.after.availability_zone]) > 3
  msg := sprintf("More than 3 checkout instances would land in %v, exceeding the single-AZ concentration limit", [resource.change.after.availability_zone])
}

That rule blocks a plan the moment it would put more than three instances of a supposedly redundant service in the same availability zone, catching the exact failure mode that a chaos engineering game day is designed to surface before the plan ever applies.

Why Technology Mapping Breaks Down Without Live Visibility

Of the five dependency categories, Technology is the one that changes fastest and gets documented the worst. A spreadsheet mapping which services depend on which infrastructure is accurate on the day someone builds it and increasingly wrong after that, since infrastructure keeps changing through deployments, console changes, and decommissions that nobody circles back to update in the mapping document.

This is where operational resilience and infrastructure governance turn out to be the same underlying problem wearing different names. You cannot accurately map a technology dependency you don't know exists, and most organizations have some percentage of their cloud footprint that was never properly tracked in the first place, created during an incident, spun up for a one-off migration, or provisioned before anyone standardized on infrastructure as code. A mapping exercise built on top of that gap inherits the gap, no matter how carefully the exercise itself is run or how good the chaos engineering program layered on top of it is.

How Firefly Supports Operational Resilience in Practice

Firefly's Cloud Asset Inventory gives the Technology mapping requirement something actually to stand on: a continuously updated record of every resource across AWS, Azure, GCP, and Kubernetes, including the ones nobody remembers mapping. Each resource includes an Owner field and an IaC status, so the questions "which team is responsible for this?" and "is this even tracked anywhere?" are both answered from the same view, instead of requiring a separate spreadsheet maintained by hand. Firefly was named in Gartner's 2026 Hype Cycle for Site Reliability Engineering under a newly defined "Continuous Resilience Automation" category, a recognition built specifically around this kind of continuous, always-current mapping rather than periodic audits.

The dependency mapping problem from earlier in this piece gets solved the same way automated discovery solves it elsewhere: clicking into any resource opens a Relationships view showing the actual dependency graph Firefly traced, not just the resources someone remembered to tag. An EC2 instance connects to its VPC, subnet, security groups, IAM role, and attached storage, the same kind of automated topology discovery covered earlier, but applied to infrastructure rather than request traces.

Ownership attribution extends into incident response directly through Event Center, which logs every mutation event, whether it came through the console, CLI, or IaC, along with who or what triggered it. When the 2 AM incident from earlier happens again, the on-call engineer won't have to page three Slack channels to find the right owner, which is precisely the MTTA bottleneck identified earlier. They're looking at a resource record that already names one, backed by a change history showing exactly what happened to it recently and who touched it.

AI SRE compresses that investigation further. A question like "which team owns the resources behind the checkout service, and has anything changed on them in the last 24 hours?" pulls from Inventory, Event Center, and Governance in a single query, instead of an engineer manually cross-referencing three separate systems while a customer-facing incident is still open. That's the specific gap between a twenty-minute technical fix and a two-hour operational incident: not the fix itself, but the MTTA and early-MTTR time spent figuring out who owns what and what changed.

Codifying the Process Layer Too: Operations-as-Code for PagerDuty

Everything above addresses the Technology category. Firefly's newer Operations-as-Code capability, launched through a direct integration with PagerDuty, extends the same codification and drift-detection model to parts of the Process and People layers as well, treating on-call schedules, escalation policies, and user access controls as assets to codify and monitor, not just as alert destinations.

The mechanics mirror cloud infrastructure exactly: Firefly maps existing PagerDuty configurations, identifies which ones are unmanaged, and codifies them into version-controlled definitions. An escalation policy managed this way looks like a standard Terraform resource:

resource "pagerduty_escalation_policy" "checkout_service" {
  name      = "checkout-service-escalation"
  num_loops = 2

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.checkout_primary.id
    }
  }

  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.checkout_secondary.id
    }
  }
}

Once that policy is codified, a manual change made directly in the PagerDuty console, someone adding an unapproved escalation step at 2 AM during an incident and never reverting it, shows up as drift the same way an unauthorized Security Group change would. That closes a specific, easy-to-miss version of the earlier scenario: the on-call engineer wasn't paging the wrong Slack channel by any bad luck; it was because the escalation policy itself had silently drifted from what the team thought it was configured to do. Firefly forwards the resulting alerts directly to PagerDuty, where they can be correlated with observability signals from tools like Datadog or New Relic for faster root cause analysis, closing the loop from detection to a codified fix within the same incident response workflow the team already uses.

Concentration risk gets a practical technical lever through Guardrails, as shown above. A team that discovers, through this kind of policy enforcement, that a supposedly redundant service is quietly concentrated in a single availability zone has found exactly the kind of gap a chaos engineering game day is designed to surface, before an actual regional event does it for them.

None of this replaces the People and Facilities work; operational resilience also requires that mapping still depends on organizational knowledge, which no infrastructure tool can generate. What it does is make sure the Technology category, and now a meaningful slice of the Process category through Operations-as-Code, stay accurate without anyone having to remember to update them, and give the testing work from earlier sections something reliable to test against.

Conclusion: Where Should You Start With Operational Resilience?

Start with the dependency category most likely to be wrong already: Technology. People and process mapping require organizational knowledge that must be gathered deliberately. Still, technology mapping can be verified against reality immediately by checking whether the live inventory actually matches what anyone assumes is running. If that check turns up unmanaged resources, undocumented ownership, or a concentration risk nobody had named, the rest of the operational resilience exercise, the impact tolerances, the incident command structure, the chaos engineering program, is being built on a foundation that won't hold up during the next severe but plausible scenario.

Start with the Technology gap since it's checkable today: connect Firefly's Cloud Asset Inventory and see exactly what's unmanaged, undocumented, or concentrated in a single AZ before the next severe but plausible scenario finds it for you.

FAQs

What is operational resilience?

Operational resilience is an organization's ability to continue delivering its critical business services through disruption, covering the people, processes, technology, facilities, and information those services depend on, not just the ability to restore a specific system or server.

What is the difference between operational resilience and disaster recovery?

Disaster recovery focuses on restoring specific systems after a failure. Operational resilience is broader: it covers whether a business service continues to function for customers even when parts of the underlying system are disrupted, including the people and processes needed to respond, not just the technical restoration.

What is the difference between MTTD, MTTA, and MTTR?

MTTD (mean time to detect) measures how long it takes for an issue to be noticed. MTTA (mean time to acknowledge) measures how long it takes for someone to start working on it after detection. MTTR (mean time to resolve) measures how long the actual fix takes once acknowledged. Each breaks down for different reasons, and improving overall recovery time requires knowing which phase is actually the bottleneck.

What is an impact tolerance?

Impact tolerance is the maximum level of disruption a specific business service can absorb before the harm becomes unacceptable, typically expressed as a time window and sometimes combined with other metrics. Setting it in advance turns incident-response decisions into pre-agreed actions rather than improvised judgment calls made under pressure.

What does "severe but plausible" mean in resilience testing?

It's the standard UK regulators use for scenario testing: a scenario doesn't need to be extreme or unlikely; it needs to be something that could realistically happen and would seriously test whether the organization stays within its impact tolerances. Cloud region outages, ransomware attacks that also hit backup systems, and third-party provider failures are commonly cited examples.

What is concentration risk in operational resilience?

Concentration risk is the risk that many organizations rely on the same small set of third-party providers, so a single provider's disruption can cascade across an entire sector at once. UK and EU regulators have both formalized supervision of this risk, the UK through its Critical Third Parties regime and the EU through DORA's Article 31, converging on the same small set of hyperscale cloud providers.

How does Firefly support operational resilience?

Firefly maintains a continuously updated inventory of cloud infrastructure with ownership attribution and dependency graphs, tracks every change through Event Center with actor attribution, lets teams query that data directly through AI SRE during an incident, and now extends the same codification model to on-call schedules and escalation policies through its Operations-as-Code integration with PagerDuty.

Does operational resilience only apply to financial services?

No. Specific regulatory requirements (FCA PS21/3, PRA SS1/21, the UK's CTP regime, DORA, Australia's CPS 230) apply to regulated financial firms in their respective jurisdictions. Still, the underlying practices, mapping dependencies, setting impact tolerances, testing severe but plausible scenarios, and managing concentration risk, apply to any organization whose customers are affected when a critical service goes down.

‍