Best AI models could not find grammatical errors

Spanish researchers from Rovira i Virgili University and the University of Barcelona have found that the most advanced artificial intelligence (AI) systems based on large language models significantly lag behind humans in detecting grammatical errors. The study has been published in the scientific journal Proceedings of the National Academy of Sciences (PNAS).

The researchers tested three of the top language models today, including ChatGPT. They gave them a simple task: to determine which sentences were grammatically correct and which contained errors. The same task was performed by a group of humans.

The human participants easily identified grammatical inconsistencies, while the artificial intelligence provided many incorrect answers. In fact, the AI identified all presented examples as grammatically correct.

According to the study's authors, the experiment confirmed that current AI systems are not capable of effectively evaluating texts for compliance with grammatical norms, even when these rules are embedded in the system. In contrast, humans can instantly and effortlessly identify trivial grammatical errors. This indicates that neural networks understand language differently than humans.

Earlier research also revealed that some AIs exhibit bias towards men and women.