SoylentNews is people
SoylentNews
Submission Preview
Want the news summarized accurately? Don't ask an "AI".
The Beeb decided to test some LLMs to see how well they could summarize the news.
https://www.bbc.com/news/articles/c0m17d8827ko [bbc.com] Turns out the answer is, "not very well".
In the study, the BBC asked ChatGPT, Copilot, Gemini and Perplexity to summarise 100 news stories and rated each answer.
It got journalists who were relevant experts in the subject of the article to rate the quality of answers from the AI assistants.
It found 51% of all AI answers to questions about the news were judged to have significant issues of some form.
Additionally, 19% of AI answers which cited BBC content introduced factual errors, such as incorrect factual statements, numbers and dates.[...]
In general, Microsoft's Copilot and Google's Gemini had more significant issues than OpenAI's ChatGPT and Perplexity, which counts Jeff Bezos as one of its investors.
Normally, the BBC blocks its content from AI chatbots, but it opened its website up for the duration of the tests in December 2024.
The report said that as well as containing factual inaccuracies, the chatbots "struggled to differentiate between opinion and fact, editorialised, and often failed to include essential context."
Normally I'd add a snide remark, but I don't think I need to this time...
