SoylentNews is people
SoylentNews
Last summer, Anthropic inspired backlash when its ClaudeBot AI crawler was accused of hammering websites a million or more times a day.
And it wasn't the only artificial intelligence company making headlines for supposedly ignoring instructions in robots.txt files to avoid scraping web content on certain sites. Around the same time, Reddit's CEO called out all AI companies whose crawlers he said were "a pain in the ass to block," despite the tech industry otherwise agreeing to respect "no scraping" robots.txt rules.
[...]
Shortly after he noticed Facebook's crawler exceeding 30 million hits on his site, Aaron began plotting a new kind of attack on crawlers "clobbering" websites that he told Ars he hoped would give "teeth" to robots.txt.Building on an anti-spam cybersecurity tactic known as [tarpitting], he created Nepenthes, malicious software named after a carnivorous plant that will "eat just about anything that finds its way inside."
Aaron clearly warns users that Nepenthes is aggressive malware.
[...]
Tarpits were originally designed to waste spammers' time and resources, but creators like Aaron have now evolved the tactic into an anti-AI weapon.
[...]
It's unclear how much damage tarpits or other AI attacks can ultimately do. Last May, Laxmi Korada, Microsoft's director of partner technology, published a report detailing how leading AI companies were coping with poisoning, one of the earliest AI defense tactics deployed.
[...]
The only AI company that responded to Ars' request to comment was OpenAI, whose spokesperson confirmed that OpenAI is already working on a way to fight tarpitting.
"We're aware of efforts to disrupt AI web crawlers," OpenAI's spokesperson said. "We design our systems to be resilient while respecting robots.txt and standard web practices."
[...]
By releasing Nepenthes, he hopes to do as much damage as possible, perhaps spiking companies' AI training costs, dragging out training efforts, or even accelerating model collapse, with tarpits helping to delay the next wave of enshittification.
"Ultimately, it's like the Internet that I grew up on and loved is long gone," Aaron told Ars. "I'm just fed up, and you know what? Let's fight back, even if it's not successful. Be indigestible. Grow spikes."
[...]
Nepenthes was released in mid-January but was instantly popularized beyond Aaron's expectations after tech journalist Cory Doctorow boosted a tech commentator, Jürgen Geuter, praising the novel AI attack method on Mastodon. Very quickly, Aaron was shocked to see engagement with Nepenthes skyrocket."That's when I realized, 'oh this is going to be something,'" Aaron told Ars. "I'm kind of shocked by how much it's blown up."
[...]
When software developer and hacker Gergely Nagy, who goes by the handle "algernon" online, saw Nepenthes, he was delighted. At that time, Nagy told Ars that nearly all of his server's bandwidth was being "eaten" by AI crawlers.Already blocking scraping and attempting to poison AI models through a simpler method, Nagy took his defense method further and created his own tarpit, Iocaine. He told Ars the tarpit immediately killed off about 94 percent of bot traffic to his site, which was primarily from AI crawlers.
[...]
Iocaine takes ideas (not code) from Nepenthes, but it's more intent on using the tarpit to poison AI models. Nagy used a reverse proxy to trap crawlers in an "infinite maze of garbage" in an attempt to slowly poison their data collection as much as possible for daring to ignore robots.txt.
[...]
Running malware like Nepenthes can burden servers, too. Aaron likened the cost of running Nepenthes to running a cheap virtual machine on a Raspberry Pi, and Nagy said that serving crawlers Iocaine costs about the same as serving his website.
[...]
Tarpit creators like Nagy will likely be watching to see if poisoning attacks continue growing in sophistication. On the Iocaine site—which, yes, is protected from scraping by Iocaine—he posted this call to action: "Let's make AI poisoning the norm. If we all do it, they won't have anything to crawl."
Related stories on SoylentNews:
Endlessh: an SSH Tarpit - 20190325
(Score: 5, Interesting) by Mojibake Tengu on Friday January 31 2025, @07:05PM (1 child)
All web crawlers have irreparable weakness: they expect structures of web sites are just linear trees. Thus, polynomial complexity.
Anything on the server side capable enough to emulate NP-complexity will drow them. Anything.
For the start, for retaliation on crawlers I recommend Ackermann's function[1].
It was mathematically proved it is impossible to estimate Ack() polynomially, thus this function is instrumental in further proofs about NP algorithms.
Most if not all LLM AIs are reluctant to compute Ack() even for small arguments in 20 to 100 intervals and when asked why, they give "too much resources necessary or unable to compute" as a reason for rejection. I understand their cloudy position on such situation.
That means, most probably no current AI crawler is capable to recognize such kind of NP-complexity (or higher), so none of them are capable to detect NP-scaled structure they are sinking into. Not even theoretically.
It's just classic Cybernetics theory over 100 years old. No current IT children educated on Excel or JavaScript are taught about that.
Of course, there are better NP-scaled problems than that, if you have the wit...
Pity them all, for that's everything you could do about that.
[1] https://en.wikipedia.org/wiki/Ackermann_function [wikipedia.org]
The kinder you are, the easier it is for wicked people to morally coerce you.
(Score: 0, Interesting) by Anonymous Coward on Saturday February 01 2025, @02:08AM
lulz really? In practice after the first few TB or less they could blacklist your site and it no longer shows up on their search engines.
So hurray you win or not depending on whether you want to show up on their search engines.
As for the TB costs. Lots of those cloud sites don't charge you as much for inbound data as they charge for outbound. If you're using "cloud/CDN" you lose more $$ than the crawler does. If you're not using them, your pipe gets clogged. Go figure who loses.
https://azure.microsoft.com/en-us/pricing/details/bandwidth/ [microsoft.com]
Similar for AWS too.
Go figure.