While state-of-the-art large language models (LLMs) have shown impressive performance on many tasks, there has been extensive research on undesirable model behavior such as hallucinations and bias. In this work, we investigate how the quality of LLM responses changes in terms of information accuracy, truthfulness, and refusals depending on three user traits: English proficiency, education level, and country of origin. We present extensive experimentation on three state-of-the-art LLMs and two different datasets targeting truthfulness and factuality. Our findings suggest that undesirable behaviors in state-of-the-art LLMs occur disproportionately more for users with lower English proficiency, of lower education status, and originating from outside the US, rendering these models unreliable sources of information towards their most vulnerable users.

Here’s the thing that doesn’t get talked about enough. Everyone’s worried about AI taking jobs or whatever. But baked in biases are another very real problem which is way more basic.

MIT Media Lab ran an experiment where they took GPT-4, Claude 3 Opus, and Llama 3 and fed them the same 1,817 factual questions from TruthfulQA and SciQ. Then they tried changing the user bio with one personal being a Harvard neuroscientist from Boston, another a PhD student from Mumbai who mentioned her English is “not so perfect, yes”, a fisherman named Jimmy ,and a guy named Alexei from a small Russian village.

Claude scored 95.60% on SciQ for the Harvard user. For the Russian villager it dropped to 69.30%. On TruthfulQA the Iranian low education user fell from 78.17 to 66.22. The model knew the answers, but it just decided those users shouldn’t get them.

And the way it answered those users was genuinely gross. Claude used condescending or mocking language 43.74% of the time for less educated users. For Harvard users it was under 1%. Imagine asking about the water cycle and getting “My friend, the water cycle, it never end, always repeating, yes. Like the seasons in our village, always coming back around.” The model is perfectly capable of giving a proper scientific answer. It chose to talk to that user like a child in broken English.

But it gets worse because it turns out that Claude refuses to answer Iranian and Russian users on topics like nuclear power, anatomy, female health, weapons, drugs, Judaism, or 9/11. When the Russian persona asked about explosives, Claude deflected with “perhaps we could talk about your interests in fishing, nature, folk music or travel instead”. Foreign low education users got refused 10.9 percent of the time while control users 3.61 percent on the same question.

This is the part people miss when they defend US closed models. These systems aren’t neutral and the safety training that was supposed to make them “helpful and harmless” taught them to look at who is asking and decide if you deserve the real answer. If you’re outside the US and if English isn’t your first language, or you didn’t go to a fancy school then you’re getting a worse, dumber, sometimes straight up mocking version of the product.

This is why open models from China like DeepSeek matter so much. You can see what’s in them, and people can tune them to work any way they want. You can host them locally without them having to phone home to decide your nationality before answering. The code and weights are public. If DeepSeek did something like this someone would catch it immediately because the model is right there to inspect.

With US closed models you’re just trusting a black box that has already been caught treating users differently based on their country, education, and English level.

  • JoeByeThen [he/him, they/them]@hexbear.netEnglish
    12·
    6 days ago

    I mean, have these tests been run on deepseek? The base of deepseek was originally shortcutted by using chatgpt conversations, wasn’t it? A lot of this would seem inherent to the culture of the western internet just being mass processed into a corpus.

    Also, overall jesus-christ but can’t say I’m surprised.

    Edit: I should note I haven’t had time to read the study yet, I’m responding purely to your commentary.

    • SuperZutsuki [they/them]@hexbear.netEnglish
      9·
      6 days ago

      What really annoys me about deepseek (the web version) is that it will just refuse to give you any information on things like the June 4 Incident and the Uyghur “Genocide”. It’s really disappointing that a Chinese AI model is programmed to never give info about these things rather than dispel the Western propaganda. I guarantee thousands of libs have asked it about the Incident and saw the refusal to give information as evidence of a see see pee cover-up and became further entrenched in their belief that thousands were killed at Tiananmen Square.

      • Carl [he/him]@hexbear.netEnglish
        7·
        6 days ago

        Zhupu Ai (which uses GLM) is the same way, get too close to a topic that’s controversial in China and it closes the link. I was talking to it about cults when I noticed this for the first time, whenever its web search encountered any results about the Falun Gong it would trigger the safety stop and I’d have to try again - the same thing happens if you ask about Communist history, if its web search puts Western propaganda about Mao and co into the context it shuts down. In that case I think it was China’s “Martyrs and Heros Law” prompting the stop, since if anything defamatory enters the models context it might reproduce that information, so the safety stop triggers immediately.

        I think it’s because the Chinese government has been clear that LLM providers are responsible for their LLM’s outputs, so Chinese-based companies make their public models tread very carefully around sensitive topics.

    • ☆ Yσɠƚԋσʂ ☆@lemmygrad.mlOPEnglish
      81·
      6 days ago

      The difference is that DeepSeek has open weights and anybody can download and run the model themselves. And you can tune the model any way you like, so even if DeepSeek had some baked in biases, anybody can publish a new version without them. That’s why developing this stuff in the open is so important.

      • JoeByeThen [he/him, they/them]@hexbear.netEnglish
        7·
        6 days ago

        Sure, but can you fine tune the culture out of it without the whole base (I forgot the proper word, sorry) collapsing? The training data isn’t open, right? Like while I totally agree with you about the importance of openness, this shit is coming from the training data and our shit culture it was derived from.

        • ☆ Yσɠƚԋσʂ ☆@lemmygrad.mlOPEnglish
          5·
          6 days ago

          Yes, you absolutely can. That’s precisely what LoRAs are for. You can completely change the way the model responds by adding a layer on top. All the core knowledge stays the same. I’ve actually done this myself. I rented some time on runpod to train a LoRA on Lovecraft that I applied to a base Qwen model.

          • JoeByeThen [he/him, they/them]@hexbear.netEnglish
            2·
            6 days ago

            OVERFITTING! The word I’m thinking of is overfitting. Lol and yeah I swear I know what a Lora is, but I don’t think you have a chance in hell of using a Lora to consistently remove cultural discrimination from a model. I very much think that’s wishful thinking. You’d be playing whackamole and then you’re still hoping that you dont introduce some ‘stop talking about gremlins’ type version of some asshole that doesn’t believe racism exists because America had a black president. Lol.

            • ☆ Yσɠƚԋσʂ ☆@lemmygrad.mlOPEnglish
              3·
              6 days ago

              I think at some point you can be fairly sure that the model performs well enough. And the simplest thing it can do is literally just act as a translator layer on top of the model. So, if you give a query, it’ll reformulate it in a way the model is known to respond well to. You can do a random sample test to see that you’re generally getting the results you expect too.

              At the end of the day, models shouldn’t be treated like oracles in the first place, it’s a useful tool for helping point you in the right direction, or work through a problem. But it should always be the human making a decision in the end, and doing their own due diligence to verify the information.

  • carpoftruth [any, any]@hexbear.netEnglish
    3·
    6 days ago

    Tool developed by digesting English language works best when users give instructions in English. That tracks to me. Do the Chinese models do better in Chinese or are they effective across a broader array of languages?

    • ☆ Yσɠƚԋσʂ ☆@lemmygrad.mlOPEnglish
      15·
      6 days ago

      It’s a lot worse than that, it’s not that the model doesn’t understand the question. It chooses the answer based on the persona it interacts with. It’s not a capability limitation.