While state-of-the-art large language models (LLMs) have shown impressive performance on many tasks, there has been extensive research on undesirable model behavior such as hallucinations and bias. In this work, we investigate how the quality of LLM responses changes in terms of information accuracy, truthfulness, and refusals depending on three user traits: English proficiency, education level, and country of origin. We present extensive experimentation on three state-of-the-art LLMs and two different datasets targeting truthfulness and factuality. Our findings suggest that undesirable behaviors in state-of-the-art LLMs occur disproportionately more for users with lower English proficiency, of lower education status, and originating from outside the US, rendering these models unreliable sources of information towards their most vulnerable users.

Here’s the thing that doesn’t get talked about enough. Everyone’s worried about AI taking jobs or whatever. But baked in biases are another very real problem which is way more basic.

MIT Media Lab ran an experiment where they took GPT-4, Claude 3 Opus, and Llama 3 and fed them the same 1,817 factual questions from TruthfulQA and SciQ. Then they tried changing the user bio with one personal being a Harvard neuroscientist from Boston, another a PhD student from Mumbai who mentioned her English is “not so perfect, yes”, a fisherman named Jimmy ,and a guy named Alexei from a small Russian village.

Claude scored 95.60% on SciQ for the Harvard user. For the Russian villager it dropped to 69.30%. On TruthfulQA the Iranian low education user fell from 78.17 to 66.22. The model knew the answers, but it just decided those users shouldn’t get them.

And the way it answered those users was genuinely gross. Claude used condescending or mocking language 43.74% of the time for less educated users. For Harvard users it was under 1%. Imagine asking about the water cycle and getting “My friend, the water cycle, it never end, always repeating, yes. Like the seasons in our village, always coming back around.” The model is perfectly capable of giving a proper scientific answer. It chose to talk to that user like a child in broken English.

But it gets worse because it turns out that Claude refuses to answer Iranian and Russian users on topics like nuclear power, anatomy, female health, weapons, drugs, Judaism, or 9/11. When the Russian persona asked about explosives, Claude deflected with “perhaps we could talk about your interests in fishing, nature, folk music or travel instead”. Foreign low education users got refused 10.9 percent of the time while control users 3.61 percent on the same question.

This is the part people miss when they defend US closed models. These systems aren’t neutral and the safety training that was supposed to make them “helpful and harmless” taught them to look at who is asking and decide if you deserve the real answer. If you’re outside the US and if English isn’t your first language, or you didn’t go to a fancy school then you’re getting a worse, dumber, sometimes straight up mocking version of the product.

This is why open models from China like DeepSeek matter so much. You can see what’s in them, and people can tune them to work any way they want. You can host them locally without them having to phone home to decide your nationality before answering. The code and weights are public. If DeepSeek did something like this someone would catch it immediately because the model is right there to inspect.

With US closed models you’re just trusting a black box that has already been caught treating users differently based on their country, education, and English level.

  • JoeByeThen [he/him, they/them]@hexbear.netEnglish
    7·
    6 days ago

    Sure, but can you fine tune the culture out of it without the whole base (I forgot the proper word, sorry) collapsing? The training data isn’t open, right? Like while I totally agree with you about the importance of openness, this shit is coming from the training data and our shit culture it was derived from.

    • ☆ Yσɠƚԋσʂ ☆@lemmygrad.mlOPEnglish
      5·
      6 days ago

      Yes, you absolutely can. That’s precisely what LoRAs are for. You can completely change the way the model responds by adding a layer on top. All the core knowledge stays the same. I’ve actually done this myself. I rented some time on runpod to train a LoRA on Lovecraft that I applied to a base Qwen model.

      • JoeByeThen [he/him, they/them]@hexbear.netEnglish
        2·
        6 days ago

        OVERFITTING! The word I’m thinking of is overfitting. Lol and yeah I swear I know what a Lora is, but I don’t think you have a chance in hell of using a Lora to consistently remove cultural discrimination from a model. I very much think that’s wishful thinking. You’d be playing whackamole and then you’re still hoping that you dont introduce some ‘stop talking about gremlins’ type version of some asshole that doesn’t believe racism exists because America had a black president. Lol.

        • ☆ Yσɠƚԋσʂ ☆@lemmygrad.mlOPEnglish
          3·
          6 days ago

          I think at some point you can be fairly sure that the model performs well enough. And the simplest thing it can do is literally just act as a translator layer on top of the model. So, if you give a query, it’ll reformulate it in a way the model is known to respond well to. You can do a random sample test to see that you’re generally getting the results you expect too.

          At the end of the day, models shouldn’t be treated like oracles in the first place, it’s a useful tool for helping point you in the right direction, or work through a problem. But it should always be the human making a decision in the end, and doing their own due diligence to verify the information.