Skip to content
All field notes

Operations9 min read

Rehearsing the outage.

Most operations have bought the security measures. Far fewer have ever watched what happens when one of the things they run on simply stops answering. The gap between owning a measure and staying open is not closed by buying more. It is closed by rehearsal, and the rehearsal fits inside one quiet hour.

Editorial illustration: a glass arch broken at its crown, the two halves standing apart.

An operation rarely stops because someone attacked it. It stops because something it runs on becomes unavailable, and no one has rehearsed what happens in the hour after. Owning a measure and staying open are different states. The second one is the only one customers experience.

The mechanism, and the gap it exposes.

In 2024, 92.76% of EU enterprises used at least one ICT security measure. In 2023, 21.54% of enterprises in the EU suffered various consequences due to ICT related security incidents. These are EU-wide figures covering enterprises of every size, and the two readings are taken from different reference years, so together they describe a standing condition rather than a trend.

The measure was purchased. The measure was documented, perhaps audited. Then it was filed. What was never purchased, documented, or audited is what happens when the thing the measure was meant to protect becomes unavailable anyway. That gap is not a purchasing failure. It is a rehearsal failure.

Buying a measure closes the procurement loop. Only a rehearsal closes the operational one.

What the list actually looks like.

A measure is bought once and verified rarely. The list of things the operation actually runs on is held in habit, not on paper. The booking system, the card terminal, the shared drive, the phone line, the supplier portal, the calendar that tells staff when to arrive. Each one has an owner in theory. Few have a fallback that anyone has actually performed.

The weak point is almost never the measure itself. The weak point is that no one knows the order in which work stops. No one has mapped which single unavailable service takes the rest down with it. That is not a security observation. It is an operating one, and it only surfaces when something stops.

The dependency list is not a risk register. It is not a matrix or a diagram. It is the answer to a plain question: if this one thing stops answering, what is the next task that cannot be completed? Write that down before the drill, not during it.

Field rule

Name the dependency before the drill begins: a fallback nobody has written down is not a fallback.

The rehearsal, as a scheduled hour.

Name one dependency. Tell the team it is a drill and name the condition that ends it, not a time limit, a named condition: the first task that cannot be completed, or the first workaround that has been actually performed. Pick the quietest hour of the operational week. Then switch that one dependency off on purpose.

Then watch. Write down only what actually happened.

  1. Time to first failure. Note how long until the first task could not be completed. Not an estimate. The actual elapsed time.
  2. The workaround. Write down what the workaround turned out to be, and who already knew it. If someone already knew it, that knowledge now has a name attached.
  3. The hard stop. Note what simply could not be done at all. That is the tolerance reading for this dependency.
  4. The condition met. Stop on the named condition. Not when it gets uncomfortable, not when the mood shifts. The named condition is the discipline.

Do it again next month with a different dependency. The rehearsal is not a project. It is a cadence, one dependency, one quiet hour, repeated until the list is covered.

Worked example

Start with whatever stands between a customer and the ability to buy or book, because that is the failure with the shortest tolerance, then work toward the quieter dependencies nobody thinks about until they stop.

What the rehearsal leaves behind.

The drill is not judged by whether the operation coped. It is judged by what it produced. Three artefacts come out of a well-run drill, and all three are prose, not a checklist.

First: a named owner for each dependency. Not a team, not a role, a person who was present during the drill and can be reached when the real failure happens. Second: a written fallback that a real person has now actually performed once. The word "performed" is load-bearing. A fallback described in a document is a hypothesis. A fallback someone has walked through, in real conditions, with real friction, is a procedure. Third: a stop condition that everyone recognises. The drill ends when the condition is met, and everyone who was present knows what that condition was.

Those three artefacts are what turn a purchased measure into something the operation can rely on. The measures were bought once and filed. The rehearsal is the only thing that reports back on whether they hold.

Common questions.

Is this not just a disaster recovery plan?
A plan is a document. The rehearsal is the act of finding out whether the document is true. Most operations discover during the drill that the written fallback depends on a person, an account, or a device that is no longer available. The artefact that matters is a fallback someone has actually performed, not one that has been described.

Is this worth an hour for a small operation?
Small operations are the ones where a single unavailable service stops everything, because the same person and the same device often sit behind several dependencies at once. The drill is cheaper the smaller the operation: one dependency, one quiet hour, one page of notes. The alternative is discovering the same information during a week nobody chose.

Which dependency should be switched off first?
Start with whatever stands between a customer and the ability to buy or book, because that is the failure with the shortest tolerance. After that, work toward the quieter dependencies that nobody thinks about until they stop, such as the shared drive, the supplier portal, or the system that tells staff when to arrive.

How detailed does the written fallback need to be?
Detailed enough that a different person, reading it cold, could perform the workaround without asking anyone. If it requires tribal knowledge to interpret, it is still a hypothesis. The test is whether someone who was not present during the drill can execute it.

Does the drill need to be kept secret from the team to be realistic?
No. A surprise test is a different exercise, and a more disruptive one. The rehearsal described here is announced, scheduled, and stopped on a named condition. The goal is not to catch the team unprepared. The goal is to find out what actually stops, in a controlled hour, with everyone able to think clearly.

If you want to map the dependencies in your operation and structure the first drill, start a conversation or see how we approach this kind of operational audit in our process.

Written by

Edward Salvatierra

Founder & CEO · Vision, intelligent systems and engineering

Edward holds the product vision and builds the systems on every engagement: the AI workflows, the data layer and the engineering that keeps them running after launch.

With Alexander Koch and Kevin Silva.

The Signal Brief

The Morvion Signal Brief

Field notes from live CRM, AI and dashboard work. One brief a week, most Mondays.