Your RL Agent Doesn’t Care About Safety (Unless You Make It)

Imagine this. You've trained an RL agent to control a chemical reactor in simulation. It's fast, it's efficient, and it beats any traditional controller, like PID or MPC, that has been used to operate the reactor for the last couple of decades. With all of these accolades, you expect great things from it, so you deploy it on the real thing.

Then the real reactor behaves a little bit differently from the simulator, as real systems always do, and you just sit there and watch your "incredible" controller completely destroy the reactor in a matter of a couple of steps. This isn't a hypothetical worry. A trained policy is, at the end of the day, just a function that maximises reward. If cutting a safety corner earns more reward, nothing in the basic setup stops it from doing exactly that. So, safety has to come from somewhere else, either from how we train the agent or from something that limits what it is allowed to do.

During my Master's dissertation, I worked on applying RL to process control, where this gap between simulation and reality is a constant concern. So I spent a lot of time researching and learning about how you can make an RL agent actually safe.

Broadly, there are two answers. You can teach the agent to be careful, or you can stop trusting it to have the final say. They sound similar, but only one of them can actually guarantee safety.

Option 1: Teach the agent to be careful

The first approach changes what the agent is trying to optimise. Normally, an RL agent maximises its expected return, and that's it. With a Constrained Markov Decision Process (CMDP), we add a rule: maximise return, but keep the expected cumulative cost of constraint violations below a certain threshold. For our use case, a "cost" could be every time the temperature goes above a safe limit, for example.

The most common way to solve this is with a Lagrangian method, where each constraint is turned into a penalty in the objective with a certain weight, called a multiplier, that adapts during training. If the agent keeps violating a constraint, the multiplier grows and the penalty becomes heavier, until the agent learns that breaking the limit costs more than it gains.

This works reasonably well, and it has been applied in many areas, such as autonomous driving, healthcare, and robotics. But there's a catch, and it's hiding in the word expected:

• The constraints only hold on average, so a single episode can still violate them badly.

• They only hold in the states the agent encountered during training.

Let's rewind a little to the opening statement of this blog post. In our imaginary reactor, the situation that destroyed it was exactly the kind the agent never encountered during training. In other words, a CMDP encourages safety, but it can't guarantee it. And if you're operating a physical system, "safe on average, in situations we've seen before" usually isn't good enough.

Option 2: Don't let the agent have the final say

The second approach takes the opposite philosophy. Instead of changing the agent, we leave it exactly as it is, but we take away its direct authority over the system. A separate mechanism, usually called a safety layer, sits between the agent and the reactor and evaluates every action the agent proposes. If an action cannot be certified as safe, it gets replaced before it ever reaches the system.

The key point is that this mechanism doesn't care how well the agent learned. A brilliant agent, a mediocre one, or even a completely random policy all get the same treatment and, consequently, the same protection, because the safety layer has the final say either way.

The Simplex architecture

The best-known version of this idea is the Simplex architecture. It was originally developed in the 90s to allow control software to be upgraded safely while a system was running, long before anyone was putting RL agents on reactors, or on anything else, really. In its commonly used simplified form, it has three parts:

• An advanced controller that is efficient and high-performing, but unverified. In our case, this is the RL agent.

• A safe controller that should be simple, conservative, and verified to keep the system within its limits.

• A decision module that monitors the system and switches control to the safe controller whenever the advanced one threatens to violate the constraints.

Here's a simplified representation of that idea, using a toy tank that must never overflow:

def is_recoverable(model, x, action, safe_ctrl, x_max, horizon=10):
    """Apply the RL action once, then check whether the safe
    controller can keep the level below x_max afterwards."""
    x = model(x, action)
    for _ in range(horizon):
        if x > x_max:
            return False
        x = model(x, safe_ctrl(x))
    return x <= x_max


def simplex_step(x, rl_policy, safe_ctrl, model, x_max):
    action = rl_policy(x)
    if is_recoverable(model, x, action, safe_ctrl, x_max):
        return action          # the RL agent stays in control
    return safe_ctrl(x)        # the safe controller takes over

At every step, the decision module asks one question. "If I let the agent do this, can my safe controller still keep the situation under control afterwards?" If the answer is yes, the agent keeps control and its action goes through. If not, the safe controller takes over.

Simplex isn't the only way to build a safety layer. Shields, control barrier functions, and predictive safety filters all take different approaches, but they share the same principle. Every action the agent proposes is checked before it reaches the system.

So which one should you use?

In practice, often both. The safety layer is what gives you the guarantee, since it protects the system regardless of how good or bad the agent is. But if it has to intervene all the time, then your conservative backup controller is effectively running the plant, and you lose all the benefits that made RL attractive in the first place. Training the agent with constraints keeps those interventions rare, so the RL agent stays in control most of the time and the safety layer only steps in when it really matters.

In short, CMDPs make the agent careful, and safety layers make sure that being careful isn't the only thing standing between you and a destroyed reactor.