The Machine on Trial

In 2014, two people were booked into the same county jail in Florida, and a piece of software gave each of them a number. Brisha Borden, eighteen, black, a couple of juvenile misdemeanors behind her, was rated a high risk to commit a future crime. Vernon Prater, white, with an armed robbery already on his record, was rated low. Two years later the software had been proven exactly wrong. Borden had stayed clean. Prater was back in prison.

The lesson everyone took from this was that the machine was biased. The more interesting lesson is the one almost nobody took: we only know the machine was wrong because the machine could be checked.

ProPublica pulled the risk scores for ten thousand people in Broward County, lined them up against what those people actually did over the next two years, and published the whole thing. A small war broke out. Statisticians, computer scientists at Cornell and Stanford and Carnegie Mellon, and the company's own engineers spent years fighting over the same numbers in the open. Nobody has ever done that to a judge. We will never know how many times a judge looked at a Borden and a Prater and got it backwards in his head, because a head keeps no records. The scandal of that software was not that it was biased. It was that, for the first time, the bias had left a paper trail.

Bias is the ordinary condition of every judgment a human being has ever made. That isn't cynicism, it's just the setup. What is rare, what has always been rare, is the ability to check the judgment afterward. Every society runs on people who tell the rest of us what is true and whom we have no practical way to audit. For a long time it was the priest. Then it was the scientist. Then the judge, the doctor, the reporter. We delegate truth because we have to. No one can verify everything for himself, so we appoint people to do it for us, and then we hope.

Science was supposed to be the one place where the hoping wasn't necessary. Peer review, replication, the whole apparatus exists precisely so that a claim doesn't rest on the claimant's good name. It mostly doesn't work that way. In 2005 John Ioannidis published a paper with the cheerful title "Why Most Published Research Findings Are False," and the years since have been unkind to anyone who hoped he was exaggerating. A psychologist named Daryl Bem got a paper claiming evidence for extrasensory perception through peer review at a top journal in 2011. When other researchers ran the experiment again and found nothing, four journals declined to publish the failures, on the grounds that a replication isn't novel. Around the same time Diederik Stapel was caught having simply invented the data in dozens of studies; he has fifty-eight retractions to his name. The machine built to catch this kind of thing kept not catching it, for a plain reason. The people who run the correction have careers, and a career is an interest, and you cannot re-run a person to see what their interest did to their judgment.

People like to say the replication crisis proves that science self-corrects. It does, eventually. It also took outsiders, a decade, and a fair amount of public humiliation to make the correction happen, which is a strange thing to call self-correction.

The law is no better, and it's more honest about it only by accident. David Mustard, studying federal sentencing, found that black defendants received sentences about twelve percent longer than comparable white ones. Survey the judges and ninety-seven percent of them will tell you they are above average at avoiding the very thing Mustard measured. My favorite part of the literature is the famous "hungry judge" study, the one where parole boards supposedly turn merciless right before lunch. Everyone loves it. It turns out it probably isn't real; most of the effect can be explained by the dull way cases get ordered through the day. We cling to it anyway, because it confirms something we already believe. Which is the whole problem performed live: a bias, working on us, while we read about bias.

Here someone usually says the deeper trouble is capitalism, that markets reward winning and have no particular love for the truth. That's half right, and the wrong half is the interesting one. A market rewards whatever you score it on. Score it on attention and you get slop. Score it on being correct and you get the best truth-finding instrument anyone has built. Prediction markets, where people bet real money on what will happen, routinely beat the panels of experts who are paid to know. Hayek pointed out eighty years ago that the price system is itself a vast machine for discovering things no central office could ever assemble. So the enemy was never interest, and never the market. The enemy is interest you can't audit.

Which brings us to the newest member of the priesthood. AI is the most powerful truth-proxy ever made, and by some distance the most concentrated. It answers in the level voice of a patient expert, and people believe it the way they once believed the encyclopedia, except the encyclopedia couldn't talk back. I build AI companies for a living, and I spend at least as much of my time worried about who is going to end up owning them. The danger isn't science fiction. A fluent machine that nobody outside a few companies can inspect, sitting between billions of people and everything they want to know, is plenty dangerous on its own. Is that a problem? Only if you think it matters who stands between a person and every question he will ever ask. A single closed oracle is the old priesthood again, with better uptime.

But something genuinely new is on offer, and it's the thing worth fighting for. You can put a model on trial. You can run it a million times and watch what it does. Set it to answer the same way twice and it will, which is more than you can say for any human authority that ever lived. You can test it across every group you care about and measure how it fails. If its weights are open you can read them, and the young science of interpretability is slowly learning to read the reasoning inside, though we are a long way from the end of that road. A study out of Tulane looked at fifty thousand sentencing decisions in Virginia where judges had an algorithm to consult, and the algorithm did two things at once. It made the sentences more consistent. And it made a racial bias visible that ordinary discretion had kept hidden for a century. That is the trade. You give up the comfortable opacity of the human gut, and you get something you can examine. You cannot subpoena a feeling. You can subpoena a model.

This is why the things that sound like three separate hobbies are really one machine. You can inspect the mechanism, which is what open source is for. You can prove which model produced which answer, which is what the cryptographic work now going on under names like zkML is for. And you can build the whole arrangement so that no one is able to quietly swap the model out or switch it off, which is what the censorship-resistant part is for. I should be honest about how unfinished all of this is. Most "open" models today are open-weight only; the training data stays secret, and you can't fully audit what you can't see. A good deal of the open frontier right now is Chinese. The cryptography doesn't yet scale cheaply to the largest models. None of that is fatal. All of it is real, and pretending otherwise is how you end up believing your own brochure.

The argument has to survive its own best objection, so here it is. Auditing the machine does not clean the world the machine learned from. Two researchers, Dressel and Farid, showed that ordinary people using seven facts predicted re-offense about as well as that Florida software did with a hundred and thirty-seven, and produced the same racial gap. The bias was in the world, in the arrest records, in the history that fed the thing. A perfectly transparent model trained on a crooked reality will reproduce the crookedness faithfully, now wearing a lab coat. There is worse news, and it comes from the mathematics of fairness itself. When two groups offend at different observed rates, you cannot build a predictor that is both equally calibrated and equally balanced in its errors. You have to pick one. So auditing doesn't hand you the right answer. It hands you the trade-off, in daylight, and somebody still has to choose which kind of fairness to want. AI doesn't abolish politics. It relocates politics into the choice of what the machine is told to optimize. The new version of "who guards the guards" is "who picks the objective."

Notice what this does to the original intuition, the one that says AI will become our single source of truth and that this is why open source must win. The instinct is right and the phrasing is backwards. A single source of truth is the most dangerous thing you could build, because it is the one arrangement in which everyone can be wrong in precisely the same direction at the same instant. Open source does not make AI the single source. It is the only thing that stops AI from becoming a single captured one. The thing to want was never one open oracle. It is many auditable ones, arguing, the way prices argue in a market and conjectures argue in a working science. Popper saw the shape of this long before the technology existed. An open society is one where any claim, the rulers' included, can be dragged into the open and overturned. Rightness was never the promise. Correctability was.

Go back to Borden and Prater one last time. The machine got them exactly backwards, and that, strangely, was the progress, because it could be caught. We are not deciding whether to trust machines with the truth. That decision is behind us; we already do, every time we ask instead of look. The only live question is whether we build them so they can be hauled into a courtroom, read line by line, and contradicted. That choice is being made right now, quietly, in license files and model cards, by a few hundred people most of us will never meet. Reasoned from first principles it isn't a close call. If one thing is going to tell all of us what is true, it cannot be allowed to belong to any of us.