In January, artificial intelligence powerhouse OpenAI announced a tool that could save the world — or at least save the minds of professors and teachers — by detecting whether a piece of content was created using AI-generating tools like its ChatGPT.
Half a year later, that tool is dead, killed because it couldn’t do what it was designed to do.
ChatGPT creator OpenAI quietly pulled its AI detection tool, AI Classifier, last week due to its “low accuracy rate,” the firm said. The explanation wasn’t in a new announcement, but was added in a note added to the blog post that originally announced the tool. The link to the OpenAI classifier is no longer available.
“We are working to incorporate feedback and are currently researching more effective provenance techniques for text, and have made a commitment to develop and deploy mechanisms that enable users to understand whether audio or visual content is AI-generated,” OpenAI wrote.
New tools that allow the use of increasingly sophisticated artificial intelligence come online almost daily and have created a cottage industry of AI detectors.
OpenAI announced the release of its AI Classifier claiming it could distinguish text written by a human and an AI. Even then, however, OpenAI called the classifier “not entirely reliable,” adding that evaluations on a “challenge set” of English texts correctly identified 26% of AI-written text as “likely AI-written,” while mislabeling human-written text as AI-written 9% of the time.
OpenAI said limitations of the AI Classifier include being unreliable on text of less than 1,000 characters, mislabeling human-written text as written by AI, and neural network-based classifiers performing poorly outside of their training data.
One sector that is very interested in accurately detecting AI is education. Since ChatGPT launched in November, teachers have raised the alarm about students using chatbots to write essays.
“We recognize that AI text recognition has been an important point of discussion among educators, and equally important is knowing the limits and impacts of AI-generated text classifiers in the classroom,” OpenAI said, adding that the company will continue to expand the scope as it learns.
OpenAI has not yet responded Decrypt request for comment.
