you can make up some game which creates words the LLM has never seen before and assigns them meaning
Tracking these sorts of "X means Y" mappings is precisely what the Q-K-V matrix of a transformer (or rather, Schmidhuber Fast Weight Programmer) does. This particular capability isn't even learned -- it's programmed in by the human who wrote the model evaluation code!
Whatever we call it there clearly is some amount of emergent abstraction in the models
I really, genuinely question this. I see extremely-high-dimensional interpolation over an extremely large dataset. Take away the dataset and what's left is gradient descent. And the token embedding, I guess. I'm not sure how you would "unlearn" something (like King-Man+Woman=Queen) from the embedding, or even what that would mean.
Doesn't have to be something that is directly solvable in K-V lookup style:
"In neothorpic algebra words for seasons take the place of even integers and words for food take the place of odd integers, arithmetic generally works as usual. What can you tell me about the result of summer + cake in neothorpic algebra?"
Perhaps we could agree that you can get pretty far-- further than what people would have expected prior to LLMs-- with pretty dumb linguistic reasoning, and that that's mostly (or all) the LLM is doing.
But how confident can we really be that our thinking is categorically different? :P
But how confident can we really be that our thinking is categorically different?
I know that humans are doing more than interpolating, because at the rate we read and for the typical lifespan we have, we simply cannot ingest enough text to perform the sorts of tasks we perform by simple interpolation.
I also know that whatever our brains are doing, it isn't backpropagation, nor is it even remotely related to it. The inventor of backpropagation, Geoff Hinton, frequently points this out. Backpropagation is egregiously nonlocal.
Tracking these sorts of "X means Y" mappings is precisely what the Q-K-V matrix of a transformer (or rather, Schmidhuber Fast Weight Programmer) does. This particular capability isn't even learned -- it's programmed in by the human who wrote the model evaluation code!
Whatever we call it there clearly is some amount of emergent abstraction in the models
I really, genuinely question this. I see extremely-high-dimensional interpolation over an extremely large dataset. Take away the dataset and what's left is gradient descent. And the token embedding, I guess. I'm not sure how you would "unlearn" something (like King-Man+Woman=Queen) from the embedding, or even what that would mean.