• 0 Posts
  • 76 Comments
Joined 1 year ago
cake
Cake day: June 11th, 2023

help-circle



  • It’s just how machine learning has been since ever.

    We only know the model’s behavior by testing, hence we only know more or less the behavior in relation to the amount of testing that was done. But the model internals has always been a black box of numbers that individually mean nothing and if tracked which neurons fire here and there it’ll appear just random, because it probably is.

    Remember the machine learning models aren’t carefully designed, they’re just brute-force trained for a long time and have the numbers adjusted again and again whenever the results look closer or further away from the desired output.

















  • I don’t even think it’s correct to say it’s querying anything, in the sense of a database. An LLM predicts the next token with no regard for the truth (there’s no sense of factual truth during training to penalize it, since that’s a very hard thing to measure).

    Keep in mind that the same characteristic that allows it to learn the language also allows it to sort of come up with facts, it’s just a statistical distribution based on the whole context, which needs a bit randomness so it can be “creative.” So the ability to come up with facts isn’t something LLMs were designed to do, it’s just something we noticed that happens as it learns the language.

    So it learned from a specific dataset, but the measure of whether it will learn any information depends on how well represented it is in that dataset. Information that appears repeatedly in the web is quite easy for it to answer as it was reinforced during training. Information that doesn’t show up much is just not gonna be learned consistently.[1]

    [1] https://youtu.be/dDUC-LqVrPU