Your AI-dar probably doesn’t work
A report from the Harvard Crimson published earlier this week presents a dire view into how one of the country’s top colleges is struggling to adapt to the AI age.
Harvard students are already using LLMs widely, and some have learned to evade professors’ more technical countermeasures, including hidden text meant to flag AI-generated work. Teachers have given up on reported suspected AI use to the school’s honor council, because there’s no real way to prove when AI was actually used. Some instructors even suggest they may push back on the technology’s proliferation by simply analyzing the—well—vibes of student submission.
“If your submission reads like it might be AI work, I’ll have you redo the assignment in its entirety. I am uninterested in proving whether you did or did not use AI,” one professor’s syllabus states, according to the Crimson. “I’ll just ask for better work, in your unique voice, reflective of your unique interests; that’s all.” (Just in case, students taking this class must submit their Google Doc version history, which may be analyzed, too).
This position reflects an enduring belief that there are some obvious tells that a piece of content was produced by AI. In prior chatbot generations, these AIisms included excessive use of em dashes. More recently, skeptics report that LLM usage is revealed by a sort of tell-tale evenhandedness: ChatGPT loves to tell you something on the one hand, and then something else on the other. A chatbot might tend to suggest that something is also this, but also, that.
These alleged AI proclivities lead plenty of people on the internet to say they simply know when something is written by AI.
But perhaps these people don’t know! The challenge for amateur AI investigators is in the long run, their detection methods might be doomed to fail. This is unfortunately obvious if you use these systems: it’s relatively easy to coax an LLM to perform to meet various standards set out by a professor, whether that’s using more adverbs, or inserting extra-long sentences, or even inserting some very human typos. The same is true of actual content. It’s also pretty easy to push an AI system to express a more unique or individual tone. You can simply coach it to! In fact, these platforms even offer ways to meld how an AI communicates to your own personal preferences.
While there may be some broad patterns to how AI writes, overall, these are just patterns, and it’s very easy to mold AI content for your own purposes—and to whatever standards are set out by instructors.
There is plenty of evidence that the quest to detect AI writing isn’t working, including the reports of Harvard professors. A recent New York Times poll indicated that lots of people can’t really tell if writing was a great work of art, or just generated by Claude. Yes, there are technological tools that show some signs of being able to detect AI prose, but not all of them work, and MIT even cautions people from relying on the tools. (Deepfake detection companies are facing the same problem, and find that their human detectors are increasingly fooled by ever more convincing image generations). These AI detectors could struggle even more as models improve as computerized ghostwriters.
Broadly, it’s hard to believe that there might be an abundance of deterministic methods for identifying the product of a stochastic process. It’s going to be wrenching to accept, but in many cases, there may be no easy way to tell, from looking at a piece of writing, that it was generated by a large language model. Consider that yes, there is a lot of AI slop, but there’s also lots of human slop. We still don’t seem to have a great sense of how to parse the difference. Our false positives and false negatives are all mixed up with one another: Plenty of people are being falsely accused of using AI to write, and plenty of others are simply getting away with it
Students cheat because they either don’t have the time or resources to complete an assignment, or because they don’t want to. These are all higher-order problems than AI itself. In the end, the quest to stop AI writing lies in convincing people that it’s not worth doing, which it really isn’t, on principle. On the other hand, a puritan approach predicated on fear of policing seems to be destined to fail. We may well learn that lesson the hard way.