Since last year's disastrous rollout of Google's AI Overviews, the world at large has been aware of how AI-powered search results can differ wildly from the traditional list of links search engines have generated for decades. Now, new research helps quantify that difference, showing that AI search engines tend to cite less popular websites and ones that wouldn't even appear in the Top 100 links listed in an "organic" Google search.
In the pre-print paper "Characterizing Web Search in The Age of Generative AI," researchers from Ruhr University in Bochum, Germany, and the Max Planck Institute for Software Systems compared traditional link results from Google's search engine to its AI Overviews and Gemini-2.5-Flash. The researchers also looked at GPT-4o's web search mode and the separate "GPT-4o with Search Tool," which resorts to searching the web only when the LLM decides it needs information found outside its own pre-trained data.
[...]
Overall, the sources cited in results from the generative search tools tended to be from sites that were less popular than those that appeared in the top 10 of a traditional search, as measured by the domain-tracker Tranco. Sources cited by the AI engines were more likely than those linked in traditional Google searches to fall outside both the top 1,000 and top 1,000,000 domains tracked by Tranco. Gemini search in particular showed a tendency to cite unpopular domains, with the median source falling outside Tranco's top 1,000 across all results.
[...]
For search terms pulled from Google's list of Trending Queries for September 15, the researchers found GPT-4o with Search Tool often responded with messages along the lines of "could you please provide more information" rather than actually searching the web for up-to-date information.While the researchers didn't determine whether AI-based search engines were overall "better" or "worse" than traditional search engine links, they did urge future research on "new evaluation methods that jointly consider source diversity, conceptual coverage, and synthesis behavior in generative search systems."
(Score: 4, Interesting) by mrpg on Thursday October 30, @09:32PM (2 children)
"While the researchers didn't determine whether AI-based search engines were overall "better" or "worse" than traditional search engine links, they did urge future research on "new evaluation methods that jointly consider source diversity, conceptual coverage, and synthesis behavior in generative search systems."
That's the only thing I wanted to read about this.
(Score: 5, Insightful) by JoeMerchant on Thursday October 30, @10:46PM
I want to know what makes domain tracker Tranco the authoritative metric on what is popular.
Next up: why is popular a good metric to be looking at, anyway?
Implied subtext for me: they're manipulating us all. Well, duh - as it has been since forever.
🌻🌻🌻 [google.com]
(Score: 2) by Mojibake Tengu on Friday October 31, @08:14AM
That 'overall "better" or "worse"' compare is computationally a scalar fold over some multidimensional metric. It depends more on a fashion it is computed by than on actual data.
Rust programming language offends both my Intelligence and my Spirit.
(Score: 5, Insightful) by Anonymous Coward on Friday October 31, @01:18AM (5 children)
Twitter's "Grok" AI has taken the place of Google for me. Of course it's using the less popular sites. THAT'S WHERE A LOT OF GOOD STUFF GOT BURIED BY SEO ENSHITIFICATION. Throw in the bit of "reasoning" that AI does, and it's essentially what "Ask Jeeves" pretended to be back in the 90s. I know AI can do other things, but for me getting around SEO is the killer app. I wonder how long it will last. They usually find ways to destroy things that are this good. Yes of course you still have to verify what AI says; but it's got more hits than misses for me, and to reiterate, it's not giving more weight to sites that used SEO and are "top 1000" by some company that's probably got ties to the ad industry.
(Score: 5, Interesting) by anubi on Friday October 31, @05:20AM (1 child)
When I am looking for insight as to troubleshooting problems on my legacy diesel van, nothing beats the three "enthusiast" sites I frequent.
Same with my old computers, DOS, and drivers...the manufacturer's site is useless. I feel I get far better technical support with my 30 + year old machines than my neighbor gets with his "brand new" 2000 model. He asks me to fix his...I can't do a thing about his machine, designed from birth for obsolescence.
The even older ones were designed for upgrades, but for me, designed for interchangeable generic parts. I like the laptops as by now, it's easier to buy a used laptop than it is to buy a part for one.
The factory sites usually. have nothing of value. Just a bunch of marketing for their latest machines.
HP used to run an FTP site with all their old drivers. They even discontinued that. There are still some Russian FTP sites that might have them. Often even AI fails to find them.
"Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]
(Score: 4, Informative) by JoeMerchant on Friday October 31, @11:25AM
It may not work for long, but I have had good success directing Gemini to focus on information found in sailing and cruisers forums posted by actual sailors and cruisers, not product vendors.
Without that refocusing directive, the AI results are basically parroting product description flak from the vendors. After adding the directive I get results more like what I search for myself in the forums.
🌻🌻🌻 [google.com]
(Score: 3, Insightful) by driverless on Friday October 31, @08:54AM (2 children)
You mean that ones that tell us that there's a global conspiracy to kill all the white people and the Jews are behind it all, and we should listen to MechaHitler? It's a good thing we've got Musk's "AI" available to tell us that.
(Score: -1, Flamebait) by Anonymous Coward on Friday October 31, @01:13PM (1 child)
Have you looked at Europe lately? Have you LISTENED TO the imams and their followers? Yes, there IS a nearly global conspiracy to overrun the West.
(Score: 4, Insightful) by janrinok on Friday October 31, @01:53PM
How have you listened to the Imams yourself? Have you visited Europe and spent time in a mosque? Or do you believe what you have been told or have read in your own media?
Do you think that only immigrants are Muslims? Do you know how many people of European descent are also Muslim? Do you know how many immigrants are Catholic, Protestant, Hindu, Buddhist, Jews, Taoists, or something else? Have a closer look at your own country. What are the religions of those many Americans - born in America, raised in America, perhaps 2nd, 3rd generation or even more Americans - who live all around you. I'll bet you are not white, Anglo-Saxon, protestant either are you? You too are of immigrant descent. Do you pose a threat to those around you who have a different religion? We even let Americans resettle here - and we don't ask them their religion before we welcome them. Have you witnessed or even participated in a Muslim holiday that welcomes and feeds everyone for free - regardless of their ethnic background or religion?
What percentage of people in America are illegal immigrants? Not just immigrants of which you have plenty such as yourself, but who are actually 'illegal'? And even those that are were supporting your economy. Ask some of the farmers whose crops are rotting in the fields, or the managers of meat packing companies that suddenly lost a large number of their staff. Are your children going to fill those jobs, or will you expect another ethnic group to do so? Will you ask them their religions too?
You are talking trash and it is clear for anyone to see. It has little to do with AI powered search engines but more to do with your own twisted and erroneous perceptions of the world about you.
[nostyle RIP 06 May 2025]
(Score: 3, Interesting) by Runaway1956 on Friday October 31, @01:34PM
What relationship is there between a site's popularity, and reliability or factuality? Facebook is arguably the most popular site in the world, but I don't go to Facebook for technical data on ANYTHING! I certainly don't go to Facebook for medical advice. Actually, the only reason I EVER look at Facebook, is the Marketplace. It does connect obscure sellers with unknown potential buyers.
Putting Facebook aside, forums in general are great sources of information, as already mentioned above. You often find manuals and datasheets linked on a forum, that might take you a month of searching elsewhere.
I'll give a nod to Youtube here, despite it's popularity. I have a trim panel on an SUV that is broken. Not sure how it happened, but it's broken. A search on Youtube showed me exactly how to remove and replace that panel, and ALSO warned me of multiple ways I could screw things up.
Data is data, and popularity has no relevance, regarding the source of that data.
“Take me to the Brig. I want to see the “real Marines”. – Major General Chesty Puller, USMC