Survival depends not only on maximising rewards but on minimising harm. While Reinforcement Learning (RL) provides a normative framework for adaptive decision-making, standard algorithms often fail to account for the catastrophic costs of errors during learning, in the same way as animals do. In this review, we synthesise recent advances connecting the engineering of safe RL with behavioural and systems neuroscience.
We adopt a principled taxonomy distinguishing between two strategies: (1) modifying the optimisation criterion to embed safety into the learning objective, and (2) modifying the exploration process to prevent dangerous actions. We show that engineering solutions, such as worst-case optimisation, risk-sensitive objectives (e.g., conditional value at risk, CVaR), and constrained Markov decision processes (MDPs), offer precise computational accounts for anxiety, risk-aversion, and defensive reflexes. Conversely, biological mechanisms, including innate priors, interoception, and meta-control based on uncertainty, reliability and competing needs, offer robust solutions to the problem of safe exploration in artificial agents.
By bridging these domains, we highlight a converging computational landscape where safety is achieved through a dynamic interplay between reactive instincts and deliberative planning. The notion of ‘learning safely’ may appear paradoxical at first glance. Learning from experience necessarily involves making mistakes, yet some mistakes, such as severe injury or death, can be so catastrophic that they preclude further learning altogether.
This raises a fundamental question for any intelligent agent, biological or artificial: how to learn from experience while minimising the risk of potentially irreversible harm? Prior work suggests that addressing this dilemma requires two seemingly complementary capabilities. The first is safe exploration: the ability to act cautiously in uncertain environments to gather information while avoiding external threats.
The second is self-preservation: the capacity to detect, represent, and respond to deviations from homeostatic grace, i.e., internal threats such as injury in an animal or a component fault in a robot1. While safe exploration concerns external dangers, self-preservation focuses on maintaining internal integrity. Both safe exploration and self-preservation are priorities for animals and other intelligent machines wishing to maximise their chances of survival in an uncertain physical world.
One way to study these capabilities computationally is through the lens of safe reinforcement learning2. Recently, such computational approaches are increasingly relevant to neurobehavioural modelling and human computational psychiatry3, particularly in domains involving fear, anxiety, panic, avoidance, aggression, and rumination4 as well as in applying principles of computational psychiatry to computers or intelligent machines5. Significant prior work has mapped these computations along the threat-imminence continuum preceding a predatory encounter, varying with the spatial and temporal proximity of danger6,7.
Extract — continue reading at the source.
