There’s a seductive argument being made for artificial intelligence in the legal system. It goes like this: courts are slow, lawyers are expensive, judges are overworked, and ordinary Americans can’t afford justice. Therefore, we should let machines help settle disputes.

At first glance, that sounds practical and even compassionate. But institutions built over centuries of trial and error deserve more scrutiny than a shiny tech fix. The justice system is one of the central institutions of a free society, and relying on artificial intelligence for justice will undermine it quickly.

The case for AI in courts and arbitration relies on real problems. The federal judiciary’s own records show that more than a quarter of all federal civil cases have someone representing themselves without a lawyer. Court statistics estimate that figure rises to around three-quarters of the tens of millions of civil cases filed in state courts each year.

These numbers show that the legal system costs too much for the average person or small business to work with, moves too slowly, and has strayed too far from the people it is meant to serve. The decline goes deeper than just cost: Gallup found that trust in America’s judicial system dropped to 35% in 2024. The National Center for State Courts found that only 44% of Americans now believe state courts offer equal justice for everyone, down from 62% in 2014. Replacing an inaccessible human system with an inaccessible machine-run one won’t fix the problem.

Nonetheless, the American Arbitration Association is already moving in that direction. Its president, Bridget Mary McCormack — the former chief justice of the Michigan Supreme Court — has built an AI Arbitrator that uses AI agents to parse claims and draft awards, with a human arbitrator signing off at the end. This reportedly resolves cases significantly faster and at lower cost. That’s the “responsible” version of this promise, but arbitration is just the beginning. Courts in California are already piloting an AI tool called Learned Hand to draft preliminary rulings, and the company claims it’s in use in courts across multiple states, including the Michigan Supreme Court.

Even the most carefully piloted program with good intentions runs into trouble. I experienced the problem firsthand a decade ago when I received a ticket for running a toll in Miami. I’ve never been to Miami. A license plate reader misread one character on the toll runner’s plate and attributed the fine to me. Getting a real person on the phone to explain the mistake took most of an afternoon. It would have been much easier to just pay the fine and forget about it. Many people in my position would have done the same. That was for a $100 ticket. Now imagine the same automatic certainty applied to a custody dispute or a construction contract worth millions.

That’s the first problem, which is really two issues: accountability and error. A judge has a name. A courtroom has a location. You can appeal or criticize a decision. Human decision-makers can be wrong, biased, or incompetent, but they work within a real system. An AI system consists of ones and zeros and is trained on data that most people will never see. It follows rules they didn’t write and runs on infrastructure they can’t check. When it makes a mistake, who is responsible? The vendor, the arbitrator who accepted its summary, the institution that bought it, or the software engineer who tweaked it? Accountability gets lost in confusion.

A 2024 Stanford study found that general-purpose large language models hallucinated up to 88% of the time when asked specific questions about legal matters and federal court cases. This isn’t confined to a lab. Last October, Senate Judiciary Committee Chairman Chuck Grassley opened an inquiry into two sitting federal judges in Mississippi and New Jersey after their chambers issued signed orders containing fabricated quotes, misstated case outcomes, and references to people who were never party to the case. These types of errors trace back to AI drafting with no oversight before it went out under a judge’s name. An error in a vacation itinerary is frustrating. An error in law can cost someone their business, home, or freedom.

The second issue is consent. While AI arbitration may begin with the option to “opt in,” most Americans know how this process usually plays out. Voluntary participation often becomes a requirement. Most people do not (or cannot) negotiate the contracts they agree to. We already click “I agree” on countless terms just to use a phone or download an app. If AI dispute resolution becomes common in agreements like that, most people will find justice-by-machine waiting in the fine print.

And what happens to the human once the machine takes over? A person will supposedly remain “in the loop,” but if an AI system processes documents, organizes facts, frames issues, and drafts reasoning, the human reviewer becomes a rubber stamp instead of a judge (as the cases above illustrate). It’s easy to opt for the machine’s draft, but a citizen in a legal setting deserves a human judgment, not an artificial imitation of one.

Due process is a meaningful opportunity to be heard by someone who can fairly weigh one’s specific situation. The point of a hearing is to be heard by a human who can be persuaded, consider exceptions, and change their mind. An AI model simply pattern-matches your circumstances against everyone else’s.

Some legal scholars, such as the former Chief Justice McCormack, argue that AI could improve the legal system. Immigration judges handle dozens of cases a day. Disability hearings can turn on a judge’s mood. Arbitration clauses sit buried in contracts nobody reads. Measured against that reality, a system that never tires and never has a bad day can look like an upgrade.

But due process protects something more fundamental. Even an overworked judge, in principle, can be persuaded, and there’s a chance for a different outcome with a better judge, a more complete record, or a sharper lawyer, because a human is in charge. Expanding AI’s role risks eliminating the possibility of a human eventually getting it right.

There are valid administrative uses for AI in law, such as reviewing transcripts and searching documents. However, these applications should not replace human judgment. The more AI influences decisions about rights, custody, punishment, or compensation, the riskier it becomes.

AI judges are a bad idea. What we need is a legal system with fewer barriers for routine issues, and human judges who recognize that the courts serve the public. (Tort reform to limit frivolous lawsuits and litigation costs wouldn’t hurt either, while we’re at it.) Technology can help with this work, but it cannot replace the human responsibility at the core of the rule of law.

The law exists to resolve disputes quickly, fairly, and transparently. The promise of AI in the courtroom is efficiency. The peril is everything else — and I, for one, would rather wait on hold for a human than get a fast answer from a machine that’s confidently wrong.