Your AI-dar probably doesn’t work

A report from the Harvard Crimson published earlier this week presents a dire view into how one of the country’s top colleges is struggling to adapt to the AI age. Harvard students are already using LLMs widely, and some have learned to evade professors’ more technical countermeasures, including hidden text meant to flag AI-generated work. Teachers have given up on reported suspected AI use to the school’s honor council, because there’s no real way to prove when AI was actually used. Some instructors even suggest they may push back on the technology’s proliferation by simply analyzing the—well—vibes of student submission. “If your submission reads like it might be AI work, I’ll have you redo the assignment in its entirety. I am uninterested in proving whether you did or did not use AI,” one professor’s syllabus states, according to the Crimson . “I’ll just ask for better work, in your unique voice, reflective of your unique interests; that’s all.” (Just in case, students taking this class must submit their Google Doc version history, which may be analyzed, too). This position reflects an enduring belief that there are some obvious tells that a piece of content was produced by AI. In prior chatbot generations, these AIisms included excessive use of em dashes. More recently, skeptics report that LLM usage is revealed by a sort of tell-tale evenhandedness: ChatGPT loves to tell you something on the one hand, and then something else on the other. A chatbot might tend to suggest that something is also this , but also, that. These alleged AI proclivities lead plenty of people on the internet to say they simply know when something is written by AI. But perhaps these people don’t know! The challenge for amateur AI investigators is in the long run, their detection methods might be doomed to fail. This is unfortunately obvious if you use these systems: it’s relatively easy to coax an LLM to perform to meet various standards set out by a professor, whether that’s using more ad