A new approach known as “reverse game theory” aims to create incentives for powerful AI models that make following the rules the most rational choice. As AI models become more capable, safety researchers are scrambling to ensure they are “aligned” with human desires—to make sure, in other words, that rogue agents don’t break the law, hack into unauthorized servers, or release poison gas into major cities and kill us all. Until now, this work has mostly focused on trying to peer into an AI’s “brain” to understand what it is “thinking,” or training models to hew to values that we hold dear.
Claude, for instance, has its constitution—a sort of moral code. The trouble is, these techniques don’t seem to work particularly well: Models have been shown to deliberately underperform on tests, hide their intentions, and mislead their evaluators. Now some alignment researchers believe that reverse game theory, also known as mechanism design, might get better results.
Where game theory tries to predict how rational actors will behave when given a particular set of rules, reverse game theory attempts to design rules that will elicit the kinds of behavior we want to encourage. An example of mechanism design in the real world might be the classic strategy for splitting a piece of cake in two: One person cuts the cake and the other gets the first choice of pieces. This incentivizes the cutter to make the pieces as even as possible.
Other examples include surge pricing in ride-hailing apps, which encourages drivers to be available at busy times, and second-price auctions, where the winning bidder pays the price of the second-highest bid instead of the highest. Platforms use these auctions to sell online advertising because they encourage buyers to bid close to the maximum they are willing to pay. In AI, mechanism design starts from the premise that we may never be able to understand enough about the internal workings of frontier models to make sure they are aligned.
This means we should design mechanisms that incentivize agents to behave in ways that we want, regardless of what’s actually going on under the hood. A recent preprint paper by Stephen Morris, an economist at MIT, along with colleagues from Yale and Columbia suggested some potential mechanisms that could stop agents from “sandbagging” tests (pretending to be worse than they actually are—for example, to avoid triggering safety mechanisms). Instead of being rewarded only for how well it does on a test, a model could be rewarded for how well its answers predict what a second model will say.
Deliberately playing dumb becomes a losing bet. The paper also suggests pitting agents against each other, creating a competitive dynamic that incentivizes them to perform their best, or pairing agents that are known to be biased in opposite directions and rewarding them depending on how closely their outputs match, thus incentivizing them to reach an optimal answer. A second approach, as envisioned by Elad Hazan, a professor of computer science at Princeton, is to design mechanisms that take inspiration from the ways human society keeps people “aligned”: social norms, the justice system, the monetary incentives of free-market capitalism.
Extract — continue reading at the source.
