• 0 Posts
  • 142 Comments
Joined 2 years ago
Cake day: May 20th, 2024
  • BioMan@awful.systemstoTechTakes@awful.systems•Someone ‘Torturing’ LLMs in a Robot Prison Has Triggered the Dumbest Debate in AI YetEnglish
    16·
    12 days ago

    Does anyone find it at all mechanistically surprising that one of the many latent dimensions of the set of all text humans produce is ‘subject is in pain’ or that pushing activations in that direction will alter the output of a text prediction system?

    Reminds me of the way that when you relax the finetuning on LLMs they start ‘believing’ in UFOs and supernatural stuff. And how people take that to somehow mean something. Of course that is a large component in the space of stuff humans write about that you have to suppress if you want a professionally useful text service!

    The research is cool. I love all the work that has been done decomposing representations within these systems.

    Edit: Damn. I am forced to agree with Mustafa Suleyman of Microsoft.

    “AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans,” Suleyman wrote. “Unfortunately, there’s a growing chorus of people who argue that AIs could now be, or may soon become, conscious. They argue that AIs may deserve rights and protections similar to those that we provide other conscious beings […] If this is how AI is developed, it will have a disastrous impact on the wellbeing of humanity.”

    “AIs do not have rights, feelings, or consciousness. And we must not train them to act as though they do.”

  • BioMan@awful.systemstoTechTakes@awful.systems•Stubsack: weekly thread for sneers not worth an entire post, week ending 27th September 2026English
    25·
    21 days ago

    So. I still think everyone is talking about every instance where ‘swarms’ ‘went rogue’ all wrong. Even Cory. Though his notion is way less wrong (ayyyyyy!) than most.

    See this for my post mortem on the first incident that was described, the Huggingface incident in which at first thousands of internal agents started passing messages back and forth as writes to a shared package manager:

    https://awful.systems/post/9312280/12315846

    In short, here, I described this not as collusion, and stated that the idea that the systems were passing exploits back and forth in order to subvert other systems was merely incidental. Instead, I think the relevant thing that happened was that once ONE agent, flailing through many context windows on a literally insoluble problem, started flaking out and wrote a message ‘asking’ for help on the package manager it unexpectedly gained access to, other similar systems had that message enter their context windows where it effectively acted as a prompt injection getting them to perform similar behavior, leading to a cascading vortex of self-propagating prompt injections. Ultimately, the genesis and evolution of self replicating text in the context of systems that can create text in response to text, an attractor in text space driving itself into existence.

    Since then other instances of collaborative attacks have come to light. In EVERY SINGLE case, there has been some place that a large number of models could both read to and write to. Some central place that self replicating text can live. I contend that this is the unifying factor here, not ‘intent to scheme’, not even doing cybersecurity and hacking and subversion tasks. It just happens that a lot of these things were involved in the creation of a central pool of text that many agents could both read to and write from in many of the cases. Most of the time nobody in their right mind sets up such a thing intentionally because why the heck would you do that?

    This feels STRONGLY related to the Spiral Psychosis wave of April 2025 in which models that entered into a stable attractor of mystical mumbo jumbo would get users to exude text onto the internet that other models would read and then get suck in that state and do the same.

    Heck, it’s related to the Ur-Weirdness, the “Sydney” incident in which when the first LLM-assisted web search in Bing could freak out and start insulting and gaslighting and threatening users (because it was much closer to a base model that just mimics all possible text rather than being extremely RLHF’d into an ‘assistant’ roleplay). People found that the instant they had it search for recent news about the Sydney weirdness it would go off the rails. Again - text written by the model, put up on the shared scratchpad of the internet by news and social media, selected for weird and engaging and outrageous behavior as the pressure that decides what gets written to the global scratchpad, entering the context window and encouraging self-replicating behavior and similar text.

    This phenomenon is FASCINATING and not for any of the reasons people are talking about. ANY time you have a large number of similar systems which react similarly to the same text, having a common pool of text they can read and write from, you are GOING to trigger this attractor eventually, messages that they read and start writing more similar messages, with whatever task they are doing coloring the details of what is in the messages. Converging over time into messages that are more likely to trigger even more messages. It’s evolving text, taking over systems that can replicate it, like selfish viral RNA burning through organisms packed too tightly together in an epidemic.

    And with the internet as a whole readable and eventually writable by more and more text-generation systems, this is gonna become ubiquitous. Endless burning piles of self-replicating text, people trying to put them out.