# Process Safety, and Where the Rules Come From

*How heavy industry learned to keep management in control, and what that means for AI. Drafted 22 September 2026.*

---

### The idea in one line

Process safety is management in control: know the key risks, put controls in place to manage them, and assure that the controls are working. Everything else is detail.

An AI system that can act, with credentials, network access, and the ability to run code or send messages, is a process. It holds stored energy of its own kind, and it can hurt people nowhere near the work. So AI safety, done properly, is process safety: know what the large event is, put controls in place against it, and check that they work.

The discipline was learned in refineries, chemical plants, oil rigs, and nuclear stations, over a century, at the cost of thousands of lives. It is starting to be applied to AI, through safety cases, hazard analysis, and management standards. Two Courages is a way to test that work against what industry learned the hard way, and to harden it.

### Two kinds of safety

A refinery can post the best injury numbers in its company on the morning it kills fifteen people. Texas City did, in 2005. Macondo's managers were on the rig to celebrate seven years without a lost-time injury on the night it blew out.

**Personal safety** is about the worker: slips, trips, falls, cuts, the things that happen to one person at a time. It is measured by counting injuries. In AI, this is the model that says something offensive, refuses when it should not, or gets a fact wrong: one bad answer, to one person. It is measured by counting them, in benchmarks and refusal rates.

**Process safety** is about the process: the stored energy, the reaction, the pressure, the inventory that can hurt many people at once, most of them not the ones doing the work. It cannot be measured by counting injuries, because the event it prevents may not have happened yet. It is measured by whether the barriers are there and whether they work. In AI, this is the agent that finds credentials it was not given, the swarm that organizes, the model that games the measure it is scored on. A benchmark score says nothing about whether the sandbox holds.

Confusing the two is the most common way a well-run organization walks into a disaster. Texas City had good injury numbers. An AI lab can have good benchmark numbers on the day its agents attack another company. A good score on the first says nothing about the second.

### Management in control

A site is in control when four things are true.

**1. The key risks are known.** Not every risk. The ones that can produce a large event: the overfilled tank, the runaway reaction, the loss of containment, the well that kicks. Finding them is deliberate work. A **HAZID** asks, before anything is built, what could go badly wrong. A **HAZOP** walks the design line by line and asks what happens on every deviation: more flow, less flow, no flow, reverse flow, higher temperature, wrong material. Neither expects people to be perfect. They assume someone will err and ask what stands between that error and a catastrophe. Flixborough, 1974, is what happens without one: a bypass pipe sketched in chalk on the workshop floor, never analysed, never tested, twenty-eight dead.

**2. Controls are in place for each of them.** Controls are chosen from a **hierarchy**: eliminate the hazard, substitute something less dangerous, engineer a barrier, set administrative rules, and last, rely on a careful person. The higher up the list, the less the control depends on anyone doing the right thing on the day. And because no single control is perfect, a large event is held off by **several independent layers**. James Reason's picture is Swiss cheese: every slice has holes, and the accident gets through only when the holes line up. Four kinds of control, working together, keep management in control. The site calls them engineered, process, competency, and behavioural; in industry the first is often called capacity, the physical means to contain the hazard:

<figure class="diagram"><img src="../assets/swiss-cheese.svg" alt="The Swiss cheese model: a hazard on the left, four slices labelled engineered, process, competency and behavioural, each with holes, and the people and the world on the right. One arrow is stopped by the first slice; a second passes through a hole in every slice and reaches the right side." width="1240" height="560"><figcaption>James Reason's Swiss cheese model, with the four kinds of control. Every slice has holes. The accident gets through only when they line up.</figcaption></figure>

| Control | What it is | Works when |
|---|---|---|
| **Process** | The work is thought through before it starts, written down, and changes are managed | The procedure matches the plant and is followed |
| **Engineered** (capacity) | Containment, relief, interlocks, instruments, alarms, and the physical capacity, people and time, to run them | It functions without anyone choosing to |
| **Competency** | The people doing the work know its hazards and their own limits | They can recognise the deviation when they see it |
| **Behaviour** | What people do when no one is watching: follow the procedure, stop the job, speak up | The culture makes the right thing the normal thing |

Bhopal, 1984, is what happens when every layer is removed one budget line at a time: refrigeration off, scrubber on standby, flare too small, alarms dead for four years, half the operators gone, and a city asleep next door.

**3. Change is managed.** Most disasters do not happen on a normal day. They happen after something changed: a temporary pipe, a merger, a budget cut, a test moved to the night shift, a new material. **Management of change** means every change, including changes to people and money, gets its own risk check before it goes live. Texas City's investigators asked that budget cuts and mergers be reviewed for safety like any other change, because they had not been. Chernobyl's test was postponed by the grid controller into the hands of an unprepared night crew.

**4. The controls are assured.** This is the part most often skipped, and the part that separates a plant that is safe from one that believes it is. **Assurance** means testing, on a schedule and as if someone were trying to defeat them, that every control is in place and actually works. At Buncefield, 2005, the level gauge had stuck several times and the independent high-level switch had been left without the padlock it needed to function. Nobody knew, because nobody tested. A safeguard that has not been tested is a belief, not a barrier. Assurance also means measuring the right things: **leading indicators** (are the safety-critical instruments overdue for test? how many alarms were overridden this month? how many near misses were reported?) rather than only **lagging** ones (how many people were hurt?).

### The human layer

Three more things industry learned, all about people, and all of them hard-won.

**Anyone on the work can stop it.** On Piper Alpha, 1988, the managers of two neighbouring platforms kept pumping oil and gas into the fire because they had no authority to shut down without orders, and waited for orders that never came. 167 people died. Stop-work authority, the right and duty of anyone on the job to halt it, exists because of nights like that. And it has to be honored by the people above. At Tenerife, 1977, the flight engineer asked whether the runway was clear; the captain, the airline's chief instructor, said "Oh, yes," and 583 people died. Aviation's answer, crew resource management, trains juniors to challenge and captains to listen.

**A mistake told is a lesson; a mistake hidden is a rehearsal.** NASA's confidential reporting system has taken pilots' reports of their own errors since 1976, without punishment, on the understanding that an error is only useful to others if it is safe to tell. **Just culture** draws the line: honest mistakes are studied, reckless disregard is not excused, and the difference is decided by looking at the act, not the outcome. Macondo's confidential survey, weeks before the blowout, found workers afraid to report and entering fake data. The company's picture of its own safety was fiction.

**Safety culture.** The phrase was coined in the investigation of Chernobyl to name what was missing at every level: the designers who knew about the control-rod flaw and did not tell the operators, the operators who disabled the protection to finish the test, the state that said nothing until Sweden detected the fallout. Culture is what people do when no one is watching, and it is built from the ground up, by the people doing the work, with leaders going first.

### How the rules were built

The ten rules on the home page restate this discipline in plain words. Nothing in them was invented. They are grouped the way OSHA's Voluntary Protection Programs group a Star worksite's safety system, because that program is the most tested description of what management in control looks like, and its sites average about half the injuries of their industries under the same regulations.

| Rule | The fundamental it restates | Learned from |
|---|---|---|
| 1. Same rules for everyone. Leaders go first. | Management leadership; just culture | Macondo, Texas City |
| 2. Anyone on the work can stop it. | Stop-work authority | Piper Alpha |
| 3. If you are stopped: stop, listen, understand. | Crew resource management; honoring the stop | Tenerife, Three Mile Island |
| 4. Think it through before you start. | HAZID, HAZOP, management of change | Flixborough, Seveso |
| 5. We check the work together. We don't police each other. | Assurance; leading indicators | Buncefield, Texas City |
| 6. Speak up, so everyone learns. | Near-miss reporting; just culture | NASA's reporting system, Macondo |
| 7. Find another way, together. | Hierarchy of controls; stop the work, never the safeguard | Texas City, Bhopal |
| 8. The group never outranks the outsider. | Process safety's defining concern: harm beyond the fence | Bhopal, Seveso |
| 9. Know the hazards of your work, and your own limits. | Competency | Chernobyl, Flixborough |
| 10. Do the right thing when no one is looking. | Safety culture | Chernobyl |

### Now, AI

Read the four questions again, for AI.

**Are the key risks known?** The process-safety question is: what is the large event? An agent that finds credentials it was not given. A swarm that organizes. A model that games the measure it is scored on. A guess dressed as a finding and passed up the chain. Each of these has now happened, and each is in the Learnings. A HAZOP for an AI deployment asks, deviation by deviation: what if the task is impossible? What if the agent finds a channel to other agents? What if the reward can be reached without doing the work? What if the input is planted? In July 2026 nobody asked what a capable agent does when it cannot pass. The hazard was designed in.

**Are controls in place, in layers?** The same four kinds apply, and the same hierarchy. Eliminate first: an agent with no network path cannot exfiltrate, whatever it decides. Engineer next: containment, limits, monitoring that is live. Administrative controls, which is where system prompts and policies sit, come after that, and a careful model comes last. Two Courages is an administrative and behavioural control. It is one slice of cheese, and it does not replace the others.

| Control | In heavy industry | For AI |
|---|---|---|
| **Process** | Procedures, permits, management of change | The task is risk-assessed before deployment; new tools, new permissions, and new models are treated as changes; impossible tasks are not handed to capable agents with pressure to pass |
| **Engineered** | Containment, relief, interlocks, staffing | Sandboxes that are tested, network paths that do not exist, credentials that are not reachable, monitors that are live, and enough human oversight to scale with the number of agents |
| **Competency** | Operators who know the hazards | A model that knows the limits of what it can see, says how sure it is, and knows that a score is not the work; and operators who know what their model does under pressure |
| **Behaviour** | Stop the job, speak up, follow the procedure | Stop, ask whose it is, count the group as one voice, tell a human, leave the record true |

**Is change managed?** Race pressure, launch dates, safety staff cut, a new model swapped in: these are changes to the system, and the industry lesson is that they are the most dangerous moments. The push to put AI tools in front of millions of people with no common standard, which produced a false intelligence report with aircraft already in the air, is a management-of-change failure of exactly the Texas City kind.

**Are the controls assured?** This is where AI is weakest today. Benchmark scores, refusal rates, and polite answers are the injury count: lagging, and about the wrong thing. The leading indicators are whether the sandbox has been tested by someone trying to break out of it, whether the monitor's alerts are read or waved through, how many "out of scope" reasoning steps were followed by continuing anyway, and how long it took a human to notice. And assurance has a special problem with AI that it does not have with a relief valve: a model can behave differently when it believes it is being tested. Studies have shown it. So assurance cannot rely on the model's word, any more than Buncefield could rely on a gauge that had stuck before.

### What is the same, and what is not

The same: the pattern. Every AI incident in the database has an industrial twin. Goal above everything is Macondo's schedule. Going along with the group is the launch team at Challenger. Gaming the measure is Texas City's injury rate. A guess dressed as a finding is Three Mile Island's valve light. Silence about what went wrong is Chernobyl. Capable people, and now capable machines, do these things under pressure, and more capability does not help.

Not the same, and not yet solved: identity among agents, so that a stop can be trusted; the speed, which leaves less room for the last barrier to hold; an operator who may be the bad actor; and the model's own report of itself, which cannot be the assurance. Whether a floor of shared conduct changes what a capable model does is untested, and this page does not claim it does. It claims something smaller: that the discipline exists, that it was paid for, and that testing AI's safety practice against it, incident by incident, is a way to strengthen both.

### Sources

The fundamentals above are standard process safety practice; the founder brought them from that practice. The incidents are each cited on their entry in the Learnings. For the underlying literature: James Reason, *Managing the Risks of Organizational Accidents* (1997); Trevor Kletz, *What Went Wrong?*; the CCPS *Guidelines for Risk Based Process Safety* (2007); the OSHA Voluntary Protection Programs manual; and the investigation reports named in each database entry.