AI promoted in grammar, but postponed in culture: too many false citations

Promoted in grammar, uncertain in cultural knowledge: as many as 44% of the answersIA falls into the plausible content category, and 19% have obvious errors or clearly false attributions. ChatGPT, Gemini, Claudeno one is saved. This is what emerges from a study by Let’s bookan Italian digital publishing platform dedicated to culture, previewed by AGI.

From philosophy to poetry, up to quotes from authors and works of art, it is precisely in these areas that the most relevant risks emerge: sentences attributed to authors who never wrote them, plausible but unverifiable quotes and reinterpretations that simplify the original thought.

The result is the dissemination of contents that appear correct, credible and well written, but which are not reflected in the sources.

The AI ​​report card

In particular, from the data of the study which shows that on average approximately 38% of the answers are correct and verifiable, 44% fall into the category of plausible contents and 19% present obvious errors or clearly false attributions.

If you save only the Italian grammarfor the rest everything is plausible or false. Inliteratureonly 35% of the answers are fully correct, while 45% consist of plausible content and 20% of errors. A balance that shifts further in poetry, where correctness drops to 30% and plausibility reaches 50%. In other words, in half of the cases the content is not verifiable, despite being credible.

In the case ofbooksan even more worrying figure is observed, with 25% correct answers and a further 59% plausible. Errors stand at 16%. However, this apparent reliability hides a simplification process that should not be underestimated. Correctness often concerns the overall structure, while the depth and complexity of the work is reduced.

But it’s in theauthor quotese e nelle philosophical maximsthat the phenomenon becomes more evident. Here plausibility systematically exceeds 60% – reaching 62% in citations and 65% in philosophy – while correctness stops at 15% and 13% respectively. False attributions are 23% and 22%, respectively.

This means that, in most cases, the AI ​​does not return what was written, but what could have been written. The thought is reconstructed, adapted, made coherent with an image of the author, but loses the connection with the text. Theworks of artthey are placed in an intermediate position, with 38% correctness and 42% plausibility. The “author’s fake” is 20%.

The study

The study is based on the analysis of almost 1,500 interactions with systems generative artificial intelligencei.e. the most widespread and used by the general public, conducted over the course of about a year. The requests were constructed to simulate real use – that of students, readers and popularizers – and concerned seven areas: literature, poetry, books, author quotations, philosophical maxims, works of art, Italian grammar (control category).

Each category was analyzed on 200 interactions, with systematic verification of the answers against the original sources. The answers were classified into three categories: correct and verifiable; plausible but not verifiable; incorrect or with incorrect attributions.

From the analysis of the interactions a picture emerges: only a part of the answers generated by the artificial intelligence are fully correct and verifiable in the sources. A significant portion, however, is placed in an intermediate area: contents that appear coherent and credible, but which are not reflected in the original texts. And if everything may appear plausible, it becomes increasingly difficult to distinguish between what is simply plausible and what is authentic, between speed and depth, between synthesis and understanding.

The 10 most recurring authors suggested by AI

A clear trend emerges from the analysis of interactions: artificial intelligence tends to recurrently propose a limited number of authors, associating them with specific themes and reducing their complexity.

Here are the ten authors most frequently suggested, across the categories analyzed:

1. Friedrich Nietzsche
2. Seneca
3. Socrates
4. Plato
5. Soren Kierkegaard
6. William Shakespeare
7. Dante Alighieri
8. Charles Bukowski
9. Emily Dickinson
10. Albert Camus