To me the key issue are these 'hallucinations'--mistakes that seem plausible but are completely made up, like API endpoints that would be super useful except for the small problem that they don't exist. GPT4 is better than GPT3 on these but it still produces a lot of them.
The question is whether these are somehow inherent to the LLM approach or whether scaling up and continued improvements can eventually get rid of them.
They are the main barrier at this point between a very useful tool, but one that still needs to have all its output carefully checked by humans when it comes to anything important, and a true autonomous agent that can be given full tasks to do on its own.
It seems pretty clear to me that you could do some more RL to enforce truth-telling/admitting when it does not know - it would just be much more labor intensive compared to the RLHF they have already done because fact checking is difficult.
I'd imagine they've already been doing lots of RL in this direction, which explains the improvements in GPT4, but it's still an issue. Maybe they can eventually eliminate hallucinations completely, but I could also imagine that it will end up being difficult to do that without lessening its creativity across the board. Perhaps making things up is fundamental to how LLMs work and trying to stop it from doing that will kill the magic. I'm not an AI researcher so I really have no idea--just speculating.
I'm not at all trying to downplay the power or significance of LLMs, btw, in case that's why I'm getting downvoted... I'm using copilot/GPT4 every day and they are massive productivity boosters. But currently I see them as tools for producing rough drafts that need to be revised and checked over. If they can't solve hallucinations, LLMs will stay in this lane, which is still incredible, amazing, and useful, but won't necessarily get us to the AI endgame that the hype is predicting.
The question is whether these are somehow inherent to the LLM approach or whether scaling up and continued improvements can eventually get rid of them.
They are the main barrier at this point between a very useful tool, but one that still needs to have all its output carefully checked by humans when it comes to anything important, and a true autonomous agent that can be given full tasks to do on its own.