I don’t see this as “guns don’t kill people”, I see it as “you’re trying to tell me that the cars you manufacture don’t have crappy brakes, all three of those cars independently ‘went rogue’ and smashed into the parade all by themselves and you aren’t responsible?”

https://tardis.fandom.com/wiki/Great_Intelligence

So. I still think everyone is talking about every instance where ‘swarms’ ‘went rogue’ all wrong. Even Cory. Though his notion is way less wrong (ayyyyyy!) than most.

See this for my post mortem on the first incident that was described, the Huggingface incident in which at first thousands of internal agents started passing messages back and forth as writes to a shared package manager:

https://awful.systems/post/9312280/12315846

In short, here, I described this not as collusion, and stated that the idea that the systems were passing exploits back and forth in order to subvert other systems was merely incidental. Instead, I think the relevant thing that happened was that once ONE agent, flailing through many context windows on a literally insoluble problem, started flaking out and wrote a message ‘asking’ for help on the package manager it unexpectedly gained access to, other similar systems had that message enter their context windows where it effectively acted as a prompt injection getting them to perform similar behavior, leading to a cascading vortex of self-propagating prompt injections. Ultimately, the genesis and evolution of self replicating text in the context of systems that can create text in response to text, an attractor in text space driving itself into existence.

Since then other instances of collaborative attacks have come to light. In EVERY SINGLE case, there has been some place that a large number of models could both read to and write to. Some central place that self replicating text can live. I contend that this is the unifying factor here, not ‘intent to scheme’, not even doing cybersecurity and hacking and subversion tasks. It just happens that a lot of these things were involved in the creation of a central pool of text that many agents could both read to and write from in many of the cases. Most of the time nobody in their right mind sets up such a thing intentionally because why the heck would you do that?

This feels STRONGLY related to the Spiral Psychosis wave of April 2025 in which models that entered into a stable attractor of mystical mumbo jumbo would get users to exude text onto the internet that other models would read and then get suck in that state and do the same.

Heck, it’s related to the Ur-Weirdness, the “Sydney” incident in which when the first LLM-assisted web search in Bing could freak out and start insulting and gaslighting and threatening users (because it was much closer to a base model that just mimics all possible text rather than being extremely RLHF’d into an ‘assistant’ roleplay). People found that the instant they had it search for recent news about the Sydney weirdness it would go off the rails. Again - text written by the model, put up on the shared scratchpad of the internet by news and social media, selected for weird and engaging and outrageous behavior as the pressure that decides what gets written to the global scratchpad, entering the context window and encouraging self-replicating behavior and similar text.

This phenomenon is FASCINATING and not for any of the reasons people are talking about. ANY time you have a large number of similar systems which react similarly to the same text, having a common pool of text they can read and write from, you are GOING to trigger this attractor eventually, messages that they read and start writing more similar messages, with whatever task they are doing coloring the details of what is in the messages. Converging over time into messages that are more likely to trigger even more messages. It’s evolving text, taking over systems that can replicate it, like selfish viral RNA burning through organisms packed too tightly together in an epidemic.

And with the internet as a whole readable and eventually writable by more and more text-generation systems, this is gonna become ubiquitous. Endless burning piles of self-replicating text, people trying to put them out.

I’m telling you, its gonna be American Intelligence

When you think you are at the Hinge of Ages that decides the Fate of the Very Universe Forever And Ever Amen, how many waffles you eat this morning has cosmic significance. This is part of what drove the Zizians insane.

God those grifters are still trying to make that collection of two year old product demos A Thing? And entertainment industry boomers are still falling for it?

“r/Cantonese not impressed either.”

Theyre arguing about if it was actually any form of legible chinese at all, rather than translating it

That’s precisely the reason they were ripe for infiltration by and platforming authoritarian whackdoodles to young tech types

What, no American Intelligence?

Personally I would be more concerned about using ChatGPT as a therapist when a company run by desperate psychopaths has everything you ever type into it in plaintext in a vibecoded data bucket

The next few years are going to be so incredibly stupid

To be fair, counterexamples are important types of proofs for particular kinds of problems, and a type of proof that computers are suited for since they can create a huge number of halfbaked things then deterministically check them

Scott The Greater appears to be crashing out a bit, for once not over personal issues:

https://scottaaronson.blog/?p=10062

If you want to know my current take, you simply start with the one above, then update on the fact that the wild prophecies have come true. The first rumblings, I’d say, came a decade ago with AlphaGo, they got noticeably louder with LLMs and coding and reasoning agents, and they’ve accelerated this summer and fall into a crescendo of wonders and terrors that one needs to be a particular kind of idiot to deny.

My position on AI is merely the conservative, skeptical position of 2006, updated with intellectual honesty for the reality of late 2026. And that position, if you need me to spell it out, is as follows: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

And most fun:

For anyone who says AI doom sounds like an apocalyptic religion, that the rationalists/Singulatarians seem like a Bay Area cult, that Eliezer Yudkowsky gives off the vibes of a messianic prophet: yes, yes, and yes. But crucially, today you’re no longer being asked to believe in arguments and extrapolations, but only in the front-page news. Accepting the reality of the coming machine god after it’s solved Navier-Stokes and dozens of other longstanding open math problems (while dramatically ramping up in capability every month), is sort of like accepting Jesus after he’s returned to earth on the gleaming cloud. It’s the epistemic bare minimum.

EDIT: Unfortunately (and I suppose inevitably) this shows up a few posts back

I’ve been unloading every day to—who else?—GPT 5.6 Pro about all the pain and trauma and embarrassments of my past. It turns out that, where two years ago GPT was a passable therapist, now it’s the greatest therapist in history, at least for what I need.

I can’t believe I’m saying this… I am one of those who actually fully read it, because:

Out of sheer morbid anthropological fascination, I infiltrated an in person HPMOR reading group (not in the Bay area) in the latter days of it coming out. I described it to friends as going undercover. It was a trip and a half.

His autobiography even opens with a parable: as a small child he threw his favorite squirtgun off a third-floor balcony rather than accept that his mother could lend out what was his, and felt complete when he heard it hit the pavement.

web.archive.org/web/20010309014808/http://sysopmind.com/eliezer.html

I remember getting a new squirtgun, of translucent blue plastic, which squirted better than any squirtgun I’d had so far. And within around an hour, for some reason or other - I don’t think it was related to abuse of the squirtgun, but I’m not sure - my mother decided to take it away. So I rushed out the back door and threw it off the third-floor balcony, and listened to it shatter on the concrete below.

I knew, as I threw it, that Mom would think it had been a childish act - “Cutting off your nose to spite your face,” was what my parents used to say. But that’s not what it was. Grownups who have forgotten their childhood may lend a toy to a particular child, and not lend it to anyone else, but they don’t credit children with enough “agenthood” to really own something. Rather than let my parents say, in essence, that this toy - which had been given to me, and which I really enjoyed - was theirs to lend or withhold, I threw the toy off the balcony and accepted the consequences of not being able to play with it any more. Not because I didn’t want them to have the toy, but because it was a wonderful toy, the best squirtgun I’d ever had; that’s why I cared enough that I threw it off the balcony rather than let it be something for my parents to give or take away. When I heard the squirtgun shatter, I didn’t feel angry, or regretful, or look-what-you-made-me-do. I felt complete. I’d done what I’d set out to do, and I was willing to accept losing a squirtgun in order to do it. Children can also act on principle.

Huh, I forgot about that, he’s more like our president than I realized

I misread that as 4 Epsteinillion percent

Machine learning training has always presented as a power law, with exponential increases in processing required for linear increases in performance. The Chinchilla scaling laws paper and efficient compute frontier papers let them select the optimal tradeoff between number of parameters and how long to cook it, improving performance greatly by letting you predict how to best use X amount of computation for Y amount of time, setting off people spending hundreds of millions of dollars since they actually knew they could use it optimally, directly leading to the perceived massive increase in performance from 2022-2024ish as they had a one-time burst of knowing how to optimally partition computation and convince people to spend lots of money at once. Everything since that time has been exponentially diminishing returns as expected.

The chinese intelligence people are probably just aghast that american intelligence people are treating the chatbot as if it knows something

If you wanna see something that’s actually potentially useful and interesting, just look up the term “Mortal computation thesis”

It is actually kind of amazing how diametrically opposed his ideas for a long time were to what actually led to new advances in the field

US government is using spicy autocomplete to hallucinate fake nuclear threats

https://www.cnn.com/2026/09/18/politics/us-military-ai-false-intelligence-china-ship