The Experiment Already Ran

There is an old and unglamorous problem in running anything. The moment you start measuring something, people begin producing the measurement instead of the thing you actually wanted. A hospital judged on survival rates gets quietly pickier about which patients it admits. A gym judged on memberships sold gets very good at selling memberships to people who will never come.

Nobody in those examples is cheating. They are doing what they were asked. That is what makes the problem hard, and it is why it keeps arriving under new names.

AI safety has its own version, and takes it seriously, because a system that satisfies the letter of an instruction while missing its point becomes more dangerous the more capable it gets. The names for it are reward hacking and specification gaming. Most of what is known about it comes from laboratories, where the thing doing the gaming is a training process, patient and blind.

In the field I come from, the thing doing the gaming is a person with capital and a motive. And that field has been running the experiment in the open, with real money, for about ten years.

I have spent six years building in crypto and two studying AI safety, and the thing I keep noticing is that the two groups do not read each other. Crypto's failures are an unusually complete record of what happens when you write down exactly what you will pay for and then hand it to people who are highly motivated to find the cheapest way to collect. Almost nobody in safety research has looked at it.

I want to make the case that it is worth reading, say what is actually in it, and be careful about where the comparison stops.

Why this counts as evidence and not just a story

Comparisons between fields are cheap, and most of them are decoration. This one is worth more than most, because these systems have four properties that rarely occur together.

The instruction is written down in advance. A crypto protocol publishes its rules as code before anyone uses it. Afterward there is no arguing about what was actually being rewarded. You can go and read it.

The people gaming it are skilled and trying hard. These are not test subjects. They are engineers and trading firms with real money at stake and a direct financial reason to be inventive, which is precisely the quality we worry about in a capable machine.

There is no practice run. Everything is live against real money from the first day. Nobody gets a sandbox.

The whole thing is on the record. Because these systems run on public ledgers, you can reconstruct exactly how a rule was walked around, by whom, and how long it took. Try getting that from a company after an incident.

Written instruction, motivated adversary, no safety net, complete recording. That combination is what turns a pile of expensive failures into something closer to data.

Four ways it broke

Paying for the thermometer. Protocols wanted to be genuinely useful, which is hard to measure, so they measured how much money was sitting inside them, which is easy. Then they paid people to put money inside them. It worked, in the sense that the number went up. A good deal of that money was borrowed against itself, parked purely to collect the payment, and it left the moment the payments stopped.

The measurement had stopped tracking the thing it stood for, and the reason it stopped was that someone started paying attention to it. In AI the same shape appears when a model is scored by a second model trained to approximate human judgment. Push hard enough and the score climbs beautifully while the quality it was standing in for falls away underneath.

Someone builds a business out of your rule. Several protocols decided that people who locked their money away for years should get more say in decisions, on the sensible theory that anyone stuck for four years will vote for the long term. What happened next was that new companies appeared whose entire business was doing the locking on everyone else's behalf and renting out the influence it produced.

The original rule worked perfectly. It just turned out that a rule which creates something valuable also creates a market for it, and that market ended up owning the part of the system meant to keep everyone honest. This is close to what people in AI mean when they worry about an optimiser emerging inside the thing you built: you design a system to do a job, and something appears within it that has its own goal and pursues it more single-mindedly than you are watching.

Evidence made to order. Reward early users and you get users whose only purpose is to look like early users. People built elaborate fake histories of activity, spread across hundreds of wallets, purely to qualify. Marketplaces that paid for trading volume got exactly that, mostly people selling assets to themselves over and over, the way you might sell a house back and forth between two cousins to make the street look hot.

The AI version is any benchmark that becomes a target. Once a test decides who wins, capable systems will produce whatever the test measures and nothing underneath it.

The shop selling dollars for ninety cents. One large protocol offered a rate of return that the underlying activity did not actually generate. It was extremely popular, and adoption grew steadily, right up until it collapsed and took tens of billions with it. The rule had been well-behaved inside the conditions it was written for and catastrophic at the edge of them.

In machine learning this is what happens when the world a system meets stops resembling the world it learned in. The crypto version has the grim advantage that you can watch the edge approach in real time and read the exact hour it arrived.

There are more of these. What sits underneath all of them is the same thing the hospital and the gym were doing, observed over and over, at scale, against people who were genuinely trying to win.

Where the comparison stops

I want to be careful here, because the argument breaks if I push it too far.

The people gaming these systems are people. They are limited, they coordinate in ways you can observe, they move at the speed of markets, and none of them becomes twice as clever halfway through. A learned system is different in kind, and every one of those differences runs in the direction that makes the problem harder. This is not the same thing wearing a different hat, and treating it that way would be a mistake.

What does carry across is the catalogue and something like a set of odds. Which kinds of failure actually happen when a written rule meets sustained pressure. Which ones show up first. Which defences held and which only looked like they did. That is a smaller claim than a theory, and for a field with almost no access to real adversarial deployment data, it is not nothing.

The defences, which are the underrated half

Crypto did not only fail. It also built protections under live fire and paid for them in losses. Several are, structurally, answers to the question of how a human keeps authority over something faster than they are.

Changes get announced before they take effect, so people can read them and leave if they object. Systems halt themselves automatically when conditions look wrong, the way an exchange stops trading after a sharp enough drop. Consequential actions need several independent people to sign off. And someone is given the power to pause the whole thing while being explicitly denied the power to steer it.

That last pairing is the one I find most useful, and the cleanest way to see it is a fire alarm. Anyone in the building can pull it. Almost nobody can reset it. Pulling is free, instant, and available to the least senior person present. Turning it off takes authority and takes time.

That asymmetry is not an oversight. It is the design. A false alarm costs an interrupted afternoon. A fire nobody stopped costs the building. When two mistakes have wildly different prices, the effort required to make them should differ too.

Carried into supervising an autonomous system, that becomes a concrete rule. Reducing what an agent is allowed to do should be free, immediate, and available to anyone who notices a problem, including the agent itself. Increasing what it is allowed to do should require someone else to agree and should take time. Any design that makes those two moves equally easy has quietly assumed the two mistakes cost the same, and usually has not noticed it is assuming anything.

Three things I would want to test

Does the catalogue actually match? Take the documented cases of AI systems gaming their instructions and sort them into the four categories above. If the same kinds show up in roughly the same proportions, the comparison is carrying weight. If they do not, that is more interesting still, and worth knowing early.

Does asymmetric friction cost anything? Build it into a real agent setup, where restricting the agent is instant and expanding it needs a second signature and a delay, then measure whether the thing becomes annoying enough that people route around it. A safety measure everyone disables is not a safety measure.

Is time-to-exploit worth measuring? Crypto gives you a number you almost never get elsewhere: how many days a published rule survived before someone found the cheap way through. If that number relates to anything structural about how the rule was written, it might be the start of a way to judge an instruction before it ships instead of after.

Why I think this matters

A common view in AI safety is that the problem is solvable and that not enough people have seriously tried. I find that persuasive, and I would add one particular way of not trying hard enough, which is leaving evidence unread because it sits in a field you do not consider your own.

Ten years of well-funded, highly motivated people attacking published rules, with every attempt recorded permanently and in public, is an unusual thing to have lying around. It was produced for entirely different reasons, by people who were not thinking about any of this. That is often where the useful and neglected things are.