When Bad Data Leads to Good Models
arxiv.org
In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of "quality" from the perspective of pre- and post-training co-design. Specifically, we explore the possibility that pre-training on more toxic data can lead to better control in post-training, ultimately decreasing a model's output toxicity. First, we use a toy experiment to study how data composition affects the geometry of features in the representation space. Next, through controlled experiments with Olmo-1B models trained on varying ratios of clean and toxic data, we find that the concept of toxicity enjoys a less entangled linear representation as the proportion of toxic data increases. Furthermore, we show that although toxic data increases the generational toxicity of the base model, it also makes the toxicity easier to remove. Evaluations on Toxigen and Real Toxicity Prompts demonstrate that models trained on toxic data achieve a better trade-off between reducing generational toxicity and preserving general capabilities when detoxifying techniques such as inference-time intervention (ITI) are applied. Our findings suggest that, with post-training taken into account, bad data may lead to good models.

In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of “quality” from the perspective of pre- and post-training co-design. Specifically, we explore the possibility that pre-training on more toxic data can lead to better control in post-training, ultimately decreasing a model’s output toxicity. First, we use a toy experiment to study how data composition affects the geometry of features in the representation space. Next, through controlled experiments with Olmo-1B models trained on varying ratios of clean and toxic data, we find that the concept of toxicity enjoys a less entangled linear representation as the proportion of toxic data increases. Furthermore, we show that although toxic data increases the generational toxicity of the base model, it also makes the toxicity easier to remove. Evaluations on Toxigen and Real Toxicity Prompts demonstrate that models trained on toxic data achieve a better trade-off between reducing generational toxicity and preserving general capabilities when detoxifying techniques such as inference-time intervention (ITI) are applied. Our findings suggest that, with post-training taken into account, bad data may lead to good models.

  • katy ✨@lemmy.blahaj.zoneEnglish
    0·
    2 months ago

    can we stop referring to llm’s as if they’re capable of thought? they don’t make decisions; their programming just responds to patterns.

    • MangoCats@feddit.itEnglish
      1·
      2 months ago

      Do you make decisions, or are you just 1300 grams of synapses responding to stimuli?

  • Naevermix@lemmy.worldEnglish
    0·
    2 months ago

    I envision a Gemini powered bot that cracks captcha and posts “woke” replies on 4chan. If you’re an antivaxxer, antisemite, nazi, racist, sionist, or otherwise, it will debate you. It will not get tired. It will not get mad. It will maintain a sense of decorum indefinitely and it will never ever stop. If some far right extremist decides to do the same, it will have the advantage that academia is left leaning, meaning the model can cite widely recognized studies.

    Dead internet theory and so on, but I’ll gladly completely and utterly destroy the internet if it means the filth dies with it.

    • Sabin10@lemmy.worldEnglish
      1·
      2 months ago

      I dislike that people are relying on them to do all their thinking for them while also being incredibly interested in the tech behind them.

      • L0rdMathias@sh.itjust.worksEnglish
        0·
        2 months ago

        I recently realized it’s a non-issue. The people doing this have already been looking for decades to find new ways to rot their minds. LLMs are just the latest in a long line of tools that help them tune out.

        • Plebcouncilman@sh.itjust.worksEnglish
          0·
          2 months ago

          I’ve said this a few times in a different way and I always get downvoted. The fact is that the people who will use the LLMs to think for them, were not gonna think a lot in the first place.

          • Dale@lemmy.worldEnglish
            0·
            2 months ago

            This is true, but we don’t need people putting glue on their pizza. These people used to have a person to ask now they’ll be asking Sam Altman

            • Plebcouncilman@sh.itjust.worksEnglish
              1·
              2 months ago

              Well I would make the argument that someone stupid enough to do such a thing kinda deserves whatever consequences their actions have. I find that people learn faster when actions have consequences instead of everything being babyproofed.

  • jsomae@lemmy.mlEnglish
    0·
    2 months ago

    Headlines should not say “scientists,” they should name the institution. (Harvard in this case.)

    • Unbecredible@lemm.eeEnglish
      1·
      2 months ago

      Headlines should not say “Harvard”, they should name the researchers. (Rachel Greene in this case.)

      I don’t know why I had to write this.

      • jsomae@lemmy.mlEnglish
        2·
        2 months ago

        Who’s Rachel Greene? But we all know Harvard and have an idea of their respectability. Name of the researcher if not well-known should be in the body instead.

  • LainTrain@lemmy.dbzer0.comEnglish
    0·
    2 months ago

    They taught it toxicity so it knows what they mean by “don’t be toxic”. It’s only a shame so few flesh and blood models take the same lesson away from it.