Want to hear something truly Orwellian? Here you go: Schools have decided to use Artificial Intelligence to monitor students’ online messaging, and because AI is a very imperfect tool, students have been arrested for making stupid jokes whose context AI couldn’t understand.

How is this happening? Well, some schools across the nation have decided to use AI as an early warning system for potential shooters. In school email-linked chats, AI monitors their every word. One such AI called Gaggle has a low success rate of identifying threats. “Gaggle alerted more than 1,200 incidents to the Lawrence, Kansas, school district in a recent 10-month period,” according to the Associated Press. “But almost two-thirds of those alerts were deemed by school officials to be nonissues — including over 200 false alarms from student homework, according to an Associated Press analysis of data received via a public records request.” Those that don’t pass muster are immediately handed over to the police.

It’s appalling that schools would immediately resort to law enforcement intervention as opposed to weighing the severity of the situation, understanding the context, and trying to use it as a teaching moment for students. One particularly egregious example of an overly punitive action concerns an eighth-grade female student in Tennessee who made a very off-color and stupid joke that looked bad to an outsider but made sense to her friends. The girl had been getting teased about being very tan and being a “Mexican,” even though she doesn’t have that heritage. Later that day, on the school email-linked chat, a friend asked her what she was planning to do on Thursday. Her response: “On Thursday we kill all the Mexico’s [sic].”

AI flagged it. The girl was arrested, strip-searched, and put in solitary confinement. Additionally, her parents didn’t find out until much later and didn’t get to see her until the next day. The student was placed under house arrest, ordered to have a psychological evaluation, and made to attend an alternative school. As if that wasn’t bad enough, the school then charged the student with truancy — even though she was under house arrest. It’s almost like the right hand doesn’t know what the left hand is doing.

Was the comment the girl made stupid? Absolutely. Did she deserve to go to jail? Not at all. AI is not an appropriate tool for this sort of application because AI isn’t human. It doesn’t understand context; it just looks for certain words or phrases that it has been directed to look for. However, AI wasn’t the only one that needed to understand the context. The school administrators also messed up and did not handle the situation well. Moreover, according to Not The Bee, the judge who ruled on her case reportedly thought the student was “probably mentally retarded” and sentenced her to a psych evaluation.

These children as so highly monitored that they aren’t allowed to be dumb teenagers. And being a dumb teenager is an important part of the growing process. If they are trained to live in constant fear that their words could be misconstrued or used against them, that is going to be terrible for their mental well-being.

Another question I have: Did the parents and the children give consent to the schools and allow this invasion of privacy? Were they given a standard of conduct and made to understand that their content was being monitored? And finally, does AI monitoring in schools violate the First Amendment? Some students have filed a lawsuit claiming that it does.

This situation is similar to what’s happening in the UK. “The police are making more than 30 arrests a day over offensive posts on social media and other platforms,” announced the Free Speech Union, a special interest group in the UK. FSU added, “Custody data obtained by The Times shows that officers are making about 12,000 arrests a year under section 127 of the Communications Act 2003 and section 1 of the Malicious Communications Act 1988. The acts make it illegal to cause distress by sending ‘grossly offensive’ messages or sharing content of an ‘indecent, obscene or menacing character’ on an electronic communications network.”

People in the UK are being arrested for thought crimes, not actions. Now our public school children are being subjected to this. We truly are living in an Orwellian nightmare.