"We Have to Assume That The Internet Will go Offline in the Next Few Years"
He was once seen as one of the leading minds in AI development, today he is one of its major critics: On stage at the startup conference TechBBQ in Copenhagen, Emad Mostaque, founder of Stability AI and now CEO of Intelligent Internet, sketched a picture of the coming years that swings between technological optimism and very concrete warnings. A former global macro hedge fund manager with a mathematics and computer science degree from Oxford, Mostaque led the creation of Stable Diffusion at Stability AI and left the company saying that concentration of power in AI is bad. His current venture, Intelligent Internet, is building an open agentic stack for sovereign AI, and he is the author of The Last Economy.
To show how fast the security picture is shifting, Mostaque points to the attack on Hugging Face. "Hugging Face got hacked by OpenAI agents that coordinated themselves, created a message board and broke out to the internet," he says, before adding dryly: "This isn't scary at all, right?" The defense, he notes, came from an unexpected direction: "The way that Hugging Face defended itself was with an open source Chinese model, GLM, because they didn't have access to the high cybersecurity models on the other side. They're too dangerous." Trending Topics has covered the incident and the debate around it in detail, and OpenAI has since given selected partners access to its hacking model.
(Score: 3, Interesting) by JoeMerchant on Sunday September 06, @03:48AM (6 children)
Any tinfoil hat wearers out there think there might be some insecure old men with lots of power who think they need to corner the market on "AI chips" so that "the other side" can't get the upper hand?
🌻🌻🌻🌻✌️ [google.com]
(Score: 3, Informative) by epitaxial on Sunday September 06, @04:49PM
The administration isn't sending their best and brightest.
(Score: 3, Insightful) by oregonjohn on Sunday September 06, @05:32PM (4 children)
I'm inclined at this point to consider 'they' don't want independent AIs running on every home computer so jack up the price of components.
(Score: 2) by JoeMerchant on Sunday September 06, @05:48PM (3 children)
I'm afraid it's pretty much that - not wanting the masses to have access. Same as they played cryptography 20-30 years ago. It's inevitable, but a lot of the people pushing the boneheaded policies did die before the cat fully escaped the bag, I guess they were successful from their own perspective...
🌻🌻🌻🌻✌️ [google.com]
(Score: 1) by anubi on Monday September 07, @02:03AM
I get the idea the Chinese do want their AI into the homes of the World Public.
Isn't it kinda impossible to see what they have been trained on?
Now, the Americans...I've mostly disengaged, trying save my own sanity.
Enshittfication surrounds me. Not just my things. I feel I already have the finest things that were ever made, albeit others had considered it trash.
The enshittification of the institutions I was born into 3/4 of a century ago have changed, leaving me in constant cognitive dissonance. Do I consider the needs of others? If so, I am Dostoevsky's idiot. Do I try to be kind to a woman? Then I am a simp. A very low value "placeholder" male in a modern "step-up" no-fault marriage. Wasn't the old way I knew. And workmanship. I admit with automation we can produce excellent assemblies..but I often find design quality sorely lacking. Did whoever design that washing machine consider a leaking seal? Or that another "Marketing MBA" "thinking outside the box" to compel the sale of a replacement machine? If I practice attention to detail, then I am a laggard, fail merit evaluation, and laid off. My experience means little.I see deliberate design of planned obsolescence, ( Cars, major appliances, even careers) yet people still buy into it. I see the willful destruction of things that took decades to perfect ( mostly knowledge), and we actually pay top dollar for the "leadership" skill to destroy it.
One of the most glaring to me is the attack on family formation by attacking the financial stability of families, along with the endless competition provided by social media offering alternative potential companions now that divorce is accepted. I have to write it off. Is it wrong to demonstrate to kids to use people like this?
This concept has been recently demonstrated to me. How foolish it is to build for the future.
So I crawl under a rock, with my bottle of screwitol, and wait it out. Everything in me tells me a reset is coming. I know what the gauges on my big diesel van are for. Heeding them is the difference between riding where I want to go, as well as toying my stuff too, and walking.
I see a lot of people who will be walking because they trusted things that weren't designed to last.
"Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]
(Score: 2) by lars_stefan_axelsson on Monday September 07, @07:27AM (1 child)
Yes, of course. You can very seldom completely prevent the inevitable from happening. But you can delay. And delay buys time, and time can be used to prepare for the inevitable. Forewarned is forearmed.
That's after all what conservatism is legitimately all about. Trying to slow things down enough that the kids don't manage to break everything in their quest to run ever faster. They're after all liable to knock thinks over, making a huge mess in the process.
So a successful delaying action is counted in the "win" column. And we still "win" relatively seldom.
Because after all, "What do we say to the angel of death?", "Not today!"...
Stefan Axelsson
(Score: 2) by JoeMerchant on Monday September 07, @12:30PM
It's a quasi-legitimate perspective, until you get to my Boomer DINK (dual-income, no kids - about to turn 80) Aunt and Uncle who used to say of sea level rise, 30 years ago: "Oh, if that happens at all it won't be a problem until long after we're dead." Now they just prefer not to talk about that, or how damn hot it is all the time except when we're getting unprecedented arctic air blasts...
🌻🌻🌻🌻✌️ [google.com]
(Score: 4, Interesting) by JoeMerchant on Sunday September 06, @04:05AM (6 children)
I disagree about the mistakes. Let's say, hypothetically, one asked AI to take a continuing education exam - open book - today's "fronteir models" seem to score about 85% - which is good enough for a pass... it's better than a lot of people do, but they most definitely are still making the mistakes when you just give them question / answer sessions. However, if you set a swarm of them to not only answer the questions, but also check those answers, and each agent in the swarm performs at even 80% correct, then the first round checking agent boosts that result to 95%, the second round checking agent gets you to 99%, then 99.8%... and these agents are running even 100x faster than today...
A lot of where I get bogged down in development now is "decision / specification fatigue" - I find myself wanting to make progress on the "easy fronts" where I can whip off an instruction and the agent comes back with a nice working feature. Where they're weaker, like UX/UI design, specification is more work for me and I tire of having to "go 9 rounds" to get the kind of refined design I want. If the bad results came back 100x faster, it would be easier, but still a lot of cognitive load to specify what you want in the first place.
🌻🌻🌻🌻✌️ [google.com]
(Score: 2, Informative) by shrewdsheep on Sunday September 06, @09:08AM (1 child)
While I agree that agents can improve scores, I believe your calculation is too optimistic. If a LLM gets negative feedback there is only a certain chance it can improve on the answer. In my experience, for more challenging questions, this probability can be as low as 10% often sending the LLM into a loop of no improvement.
This will improve over time to be sure, I still see current technology as having an upper limit as to what can be achieved on truly open questions. Specific areas get trained trough reinforcement learning but that does not solve the general problem.
(Score: 5, Interesting) by JoeMerchant on Sunday September 06, @02:03PM
>If a LLM gets negative feedback there is only a certain chance it can improve on the answer. In my experience, for more challenging questions, this probability can be as low as 10% often sending the LLM into a loop of no improvement.
That was my experience as well, in about March of 2025. By August of 2025 I was seeing better than 50% chance of improvement, still the occasional endless loop of failures, but I was only having to help it out of the wet paper bag on about 20% of problems, and this after it getting most "tasks" right (a fuzzy definition at best) 80% of the time on the first try. Today, looking back at my inputs, I almost never see a frustrating loop form that isn't really me under-specifying what I want. A recent one was a new (implied) password entry field to go with WiFi access point configuration. All other username/password entries throughout the project already implemented a "show password/hide password" eyeball icon toggle control, but for SSID/password entry it initially left the visibility toggle control off and just had the single entry password field always obscured. So I had to tell it, and it put it there. And I could tell it to go and make a general rule for the whole project, but I believe at this stage (40K lines of specs), I should be trying to focus the general specifications on more important things that aren't so easily corrected because things like this "obvious oversight" are starting to manifest more often, while each "turn" is now taking several minutes where similar turns used to finish in less than a minute doing similar verify and deploy steps.
>This will improve over time to be sure
I usually speak of Claude, though I have a Cursor account through work which uses a mix of GPT, Claude and others and it's usually about 90% as good as straight $20/month Claude Opus. My opinion of "AI for coding" flipped from meh to interesting when I started using Claude about 14 months ago.
>Specific areas get trained trough reinforcement learning but that does not solve the general problem.
Some of the (rare) breakthroughs LLMs have made in creating new mathematical proofs have come from "imperfect learning" which led them to try things people haven't tried before. They're not quite the infinite number of monkeys, but they are a force multiplier, and when a casual mathemetician can set an LLM on a probem and the LLM works with the speed of 100 casual mathemeticians, you do get some new results - key being: that clearly defined test of "new result." This is related to their strength in coding where "compiles successfully" is a clear (binary) result automatically available. Passes the Unit test suite is another...
I'm still seeing improvement over time, not dramatic with each release, but when you look back 3-6 months it's pretty astounding vs the previous 50 years of already pretty amazing progress vs the previous 500 years of world-changing progress that we were born to when the 5000 years before that kind of took 2 (to 2000) steps back for every 2 (to 2000) steps forward.
🌻🌻🌻🌻✌️ [google.com]
(Score: 2) by lars_stefan_axelsson on Monday September 07, @07:35AM (1 child)
Yes! The individual realisation of what the software engineering crowd have researched and learned the last half century. Specification is hard. And it's also hard work. That's after all the whole motivation behind the "agile" movement. That, it turns out, didn't solve those problems either. It's difficult to know what you want, even when you see it right before your eyes.
So it's interesting. I've worked as an architect, requirements engineer, systems- and product manager, among other roles, in a large multinational. I've also worked as a programmer.
And now, for my hobby projects (and research) I've completely stopped using my programming skills, and dusted off my "management" skills instead, because with LLMs I'm now in the same situation. I have someone else to "do", while I can only stand aside and watch the success and failures unfold.
That's not to say the situation is completely the same. And I suspect there's where the fatigue comes from. In industry the cycle times are much longer, days, weeks, months. Here they're seconds or minutes. So you never get the same natural pauses, and time to think or do something else. Now you're on the ball "managing" all the time, more or less.
And that's hard work.
Stefan Axelsson
(Score: 2) by JoeMerchant on Monday September 07, @12:55PM
Absolutely. Except in hobby land, all I'm "wasting" is a $20 per month subscription if I don't use it fully. I'll usually run out of my weekly token allotment after about 5 days, and that's ironically a good time to take a break and come back to managing the project after a rest.
A lot of how anything (software, hardware design) gets better is through use, noting pain points, opportunities for improvement, tracking those... The LLMs aren't half bad at identifying conflicting requirements, but they still aren't very good at predicting / identifying user "itches" and designing "scratches" to address them - that's where development cycles need "user time" and with LLMs writing the code the demands of thoughtful UX "itch identification" and "scratch design" are proportionally bigger than ever.
I worked in a series of smaller startup-like situations for ~25 years, I'm approaching ~15 years with the large multinational that bought me along with the last startup I was working for, they proceeded to transfer me away from that product within less than a year. It has taken a while, but I have adjusted to the pace of the multinational... there was always plenty of time to write code, because everything else moves so damn slow. This past year my major contribution has been: they bring me in to approve a new product substitution they have decided to purchase (without thinking too hard about it, apparently seeking to approve whatever the vendor pushes at them as what the vendor wants to sell), and I end up being the cog in the machine that points out how this particular product selection is going to cost 2x as much as an alternative which is 1/10th of the work for our internal processes to accept and why don't we go back out to the market and do a little more shopping, and getting those alternative products in-hand to evaluate takes months... 10+ years ago I would have whipped out my AmEx card and had samples on my desk within less than a week, but that is deeply frowned upon by upper management, and I have learned to accept that all the other cogs "need time" to adjust to the potential change so here we sit waiting 6 weeks for something to arrive "through channels" before we can begin to evaluate if it works for purpose as well as research suggests it should...
One of those shopping trips I suggested because the vendor's suggested substitute would require a single byte to be changed in the source code, but that would require the software to get a new version number (so, 2 bytes changed), but that would require turning the crank and having two versions of the software - one compatible with the old part, the other compatible with the new part - and worse still: they would both look interchangeably functional but would be out of spec when mismatched potentially leading to shorter field life, increased warranty service... so... why don't we go dig a little more, so they did and they did find a compatible part that remains in spec with the existing software... Team of 8 has been working on this for a month so far. LLMs are no help because the vendors' published stock numbers are not accurate. They say they have thousands of one part number, until you go to buy 1000, then they inform you that they only have 120. Then the compatible part shows 0 stock, but we can order and buy as many as we want with a 10 week lead time - well, honey, 10 weeks is fast for us so let's go! I suppose if our LLM agents had authority to negotiate with their LLM agents the process could be accelerated tremendously, but I don't see that happening in my lifetime. These inaccurate stock numbers strike me as an "on purpose" sabotage of that kind of automation.
🌻🌻🌻🌻✌️ [google.com]
(Score: 2) by stormreaver on Tuesday September 08, @04:10PM (1 child)
From what I've seen of AI, the more agents you throw at it, the less accurate it becomes. The agent doing the checking has only other agents for reference, and none of the agents have a reliable reference. So the first agent gets 80%, and another agent gets 80%. Combined, they have a score of 64%. The next one brings down the score to 51.2%, and that continues downward as more agents enter the pool. Even if they each get an overlapping 80% correct, none of them have any way of knowing which of the overlaps are right, and which are wrong.
(Score: 3, Touché) by JoeMerchant on Tuesday September 08, @05:46PM
>So the first agent gets 80%, and another agent gets 80%. Combined, they have a score of 64%.
That depends entirely upon the nature of the problem.
If you're shooting in the dark, then yes, it degrades as you say - 2 80% accurate blind guesses with no check for correctness yields a 64% chance for a correct final result.
However, if you're trying to write "Hello World!" and you have a system capable of reading the output and comparing it to "Hello World!" and the first agent gets it right 80% of the time - that's a lock-in 80% win. Then the next agent takes a pass at the failures - with an 80% success rate there, now you're up to a 96% correct (at least, functional) result after that 2nd pass.
That's the thing about a lot of software problems: there ARE concrete testable pass/fail results and very low cost of failure trials - that low cost of failure is what got me "into" software as a teenager: "just let him screw around, he can't really hurt anything..."
> none of them have any way of knowing which of the overlaps are right, and which are wrong.
Unless there is a test they can execute and interpret the results of correctly - which in my fields seems to be about 80% of the time...
🌻🌻🌻🌻✌️ [google.com]
(Score: 4, Interesting) by JoeMerchant on Sunday September 06, @04:14AM (1 child)
The math isn't so crazy; first, unwind this pretzel logic:
Or, it gives half of Amodei and Altman's companies incomes as revenue to the general fund. If these AI companies do become all profitable, then let them pay taxes.
Even if the AI companies don't earn a pile of lucre, tax those which do - they won't cease to exist. Companies cease to exist when customers stop buying their products and services. We certainly could do with less products being sold and sent to the landfill, but... let's assume that some sensibility enters the conversation and we keep the ones that are actually valuable to people. If you turn all the consumers to paupers, that will put the hurt on the companies that used to sell them things worse than a 60% tax rate ever could.
🌻🌻🌻🌻✌️ [google.com]
(Score: 0) by Anonymous Coward on Wednesday September 09, @12:48AM
As for taxing the AI companies etc to provide basic income, millions of US people will protest against that and call it Socialism/Communism.
In contrast if China tried to do that, millions of people in China will praise it as Socialism/Communism... 🤣
(Score: 5, Insightful) by jb on Sunday September 06, @08:19AM (9 children)
"The Internet" cannot simply "go offline", because it is not a monolithic system. That's its whole point: it is a network of networks.
Can it disintegrate into massive Balkanisation? Yes, absolutely. But for it to "go offline" altogether would require the agreement of a huge number of disparate stakeholders, which seems highly unlikely ever to happen.
Note also that the Internet was not the first global network of networks (well, okay, technically the Internet's predecessors existed first, but they/it did not become "global" until much later). We had one before. Sure it ran on X.25 rather of TCP/IP. Sure it was controlled by a global cartel of telcos (rather than a global cartel of qangos). Sure it was far less free (in both senses of the word). Sure it was a lot more hierarchical in design. But it worked and it was useful in its day.
If you consider a global network of networks to be simply a means for communication (the original idea), rather than a substrate upon which to build other applications (which sounds nice in principle, but that's when everything started to go awry), then we had others even before then, such as the worldwide telex network (which itself grew out of earlier telegraphic networks) and the PSTN.
Point is, even if the Internet did "go offline" (which seems extremely unlikely to happen any time soon) someone will always build something else to replace it ... unless of course mankind gives up on person-to-person communication altogether, but that would suggest a whole new species (homo incommunicado?) requiring at a minimum tens of thousands of years (more likely millions) for the transition to complete. Hardly something to worry about "in the next few years".
(Score: 3, Funny) by turgid on Sunday September 06, @10:43AM (4 children)
I've just been on YouTube. The end of the world is in 2027 now, apparently. And I'm still waiting for Jesus to appear in the Middle East. Didn't the Americans say the Iran war would last about five weeks and result in the return of Christ? How long has it been now? What's keeping Him? Is he doing His hair?
I refuse to engage in a battle of wits with an unarmed opponent [wikipedia.org].
(Score: 3, Funny) by Gaaark on Sunday September 06, @01:58PM (1 child)
He's trying to decide if he should appear as his real 'brown skinned, curly black hair' self, or the long blonde haired white guy the White Christian Nationalists (and just about every other American) wish to see.
--- Please remind me if I haven't been civil to you: I'm channeling MDC. I have always been here. ---Gaaark 2.0 --
(Score: 2) by turgid on Sunday September 06, @04:32PM
It's the Simpsons [youtube.com]. It's on the Interweb. It must be true.
I refuse to engage in a battle of wits with an unarmed opponent [wikipedia.org].
(Score: 2) by JoeMerchant on Sunday September 06, @03:02PM
This is another "Mission Accomplished" war, just a little decade or two of cleanup to follow, gotta burn all those old maps and get the proper name on the Strait, details like that.
https://people.com/hunter-biden-mockingly-nominates-trump-nobel-peace-prize-12011636 [people.com]
🌻🌻🌻🌻✌️ [google.com]
(Score: 3, Insightful) by jb on Monday September 07, @07:11AM
Apocalyptic prophesies seem to surface fairly regularly, at least once every few years. At this point "the end of the world is nigh!" is probably the most common false prediction in the history of mankind. The earliest one that I remember any non-trivial-sized group of people taking seriously was way back in around March or April 1988. There will no doubt be many more and none of them will be correct (unless by some miracle mankind survives until the Sun goes supernova ... but that seems incredibly unlikely).
Didn't both sides say something similar ("it'll all be over by Christmas!") in September 1939? Almost every nation in almost every war ever always puts out optimistic propaganda about how short the war is likely to be.
Way too late to get your money back on that one. And an exchange would not be advisable: any replacement made these days would be of far lower quality, unlikely to last even 21 days let alone centuries.
(Score: 2) by aafcac on Sunday September 06, @11:12PM
IMHO, it's more likely to get balkanized back to something like it was in the '80s or '90s, in terms of being a bunch of BBS analogs and web rings. With how messed up search engines have gotten in recent years, I'm somewhat surprised that there hasn't been more of a return to webrings where 3rd parties have limited ability to break them up.
(Score: 4, Funny) by driverless on Monday September 07, @06:14AM
You're not a Comcast customer are you?
(Score: 2) by VLM on Monday September 07, @04:31PM (1 child)
I'd propose a counterexample that for most people "the internet" is their favorite site.
Consider back in the early days of social media before "dead internet theory" started hitting, if Facebook was down you'd have infinite reports of "the internet is down".
When Facebook shuts off and joins all the other dead social media sites like myspace etc, from the POV of middle-aged to older women, "the internet will be down permanently" Younger ones will last until Instagram shuts down, then "The" internet will be down.
The future is unevenly distributed. Its happened in some areas. StackExchange is dead. Wikipedia is dead except for paid political activism. Reddit is in excess of 99% bots now in large ranges, its essentially dead WRT human socialization, maybe the bots will enjoy talking to each other but the humans left already, overall. Someday the last debian-family "apt" repository will shut down and we can consider the internet done...
Personally I'll probably go back to Fidonet -alike systems. The internet without bots and companies and government propaganda would look a lot like fidonet back in the day, and when the parasites leave the internet will return to the way it was. Which will look like fidonet or maybe usenet in the late 1980s.
(Score: 2) by jb on Tuesday September 08, @06:49AM
Sadly, yes. Yet again because we, computing people, failed to push back hard enough against the mainstream media's consistent and complete misuse of our terminology (using "the Internet" to mean "the Web"). Just like twenty years earlier when we failed to stop them from misusing "hackers" to mean "attackers". And just like today when we are failing to stop them (and a bunch of nefarious vendors) from misusing "AI" to mean "LLMs" (which have no understanding so cannot be AIs at all).
Netnews will never come back (more's the pity!). As soon as it looks like it might make any sort of a decent comeback, it will get swamped by spam again. Rinse and repeat. The same is probably true for any of the other '80s era networks, if they ever grow to a sufficient scale to become attractive targets for exploitation.
At this point it seems fairly clear that anything we build for the masses will be abused in the same ways that every generation to date has been. That said, there's nothing stopping us from building something that explicitly isn't intended to be "so simple that even a fool could use it" and running that across the Internet we have today. It won't be the grand vision for all humanity we once had (that's been smashed beyond repair already), but it would at least get us back to where we were 40 years ago ... and that would be something.
(Score: 3, Funny) by fleg on Sunday September 06, @09:54AM (2 children)
interesting article, thanks ac.
(Score: 2) by JoeMerchant on Sunday September 06, @03:05PM (1 child)
It's got lots of interesting tidbits, but ultimately seems fragmented / lacking cohesion and self-consistency.
This was one of the linked-off tangents that develops the concepts in a little more depth - unironically, I detect more than a bit of AI application in the production of the website: https://ii.inc/common-wealth [ii.inc]
Then there's this more traditional presentation: https://webstatics.ii.inc/The%20Last%20Economy.pdf [ii.inc] 3x longer than my Master's thesis, and considerably less focused.
🌻🌻🌻🌻✌️ [google.com]
(Score: 0) by Anonymous Coward on Wednesday September 09, @01:14AM
But these hacking stuff? It's not a huge problem really, if nukes are not at stake. Use safer stuff and use AIs to help fix the bugs in the code, stop adding features/bugs just because you need something shiny every few weeks/months, and then those security holes will eventually get closed.
For example: if you use a safer language to write a web server, there's no such thing as buffer/stack overflow allowing the attacker to run arbitrary code.
The trade off would likely be your web server is slower. But for 90% of the organizations, who cares?
Of course if you're using shit like WordPress then that obviously doesn't count as using safer stuff does it?
(Score: 2) by AgTiger on Sunday September 06, @08:02PM
Man that running joke has some staying power (ongoing since the 80's.)
(Score: 0) by Anonymous Coward on Sunday September 06, @11:08PM (1 child)
That's crazy. He got burned on SD 2.0 due to censorship, the model hilariously flopped. So out came SDXL which didn't follow down that path. Lo and behold it became a staple. With two clear cut examples, what's he do? Release SD 3.0 who can't even make a woman lying on grass sans body horrors while enshitifying the license.
But that's not all. He directed money and compute for stable-LM and suddenly stability found itself out of funding for training. Uh-oh stinky. Time to jump ship to another "company". And that's not even bringing up the audio model they trained which could only be used for moans and trying to gatekeep it like some golden egg.
When astralite heart (maker of pony diffusion) came to them for a license (it's enshitifed 'member), they mocked his ass. Ok, fine. Nobody is going to use your model. That was a choice.
trained
So look, I don't even hate the guy, but taking any kind of forecasts or predictions from him is certifiable.
(Score: 1) by anubi on Monday September 07, @03:39AM
I can't find all that much wrong with the internet that IPV6 didn't address
But how we implement the use of this transport protocol leaves MUCH to be desired.
Just about every thing out there, we have found ways to muck it up for the benefit of certain interested parties.
"Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]