> ChatGPT produces some shockingly good text, but the rate of hallucinations and its inability to reliably reason about either correct or incorrect statements would be enough to mark a human as untrustworthy.
It's not even the rate, which is troubling enough. It's the kinds of things it gets wrong too. For instance, you can say to ChatGPT, "Tell me about X" where X is something you made up. Then it will say "I don't know anything about X, why don't you tell me about it?" So you proceed to tell it about X, and eventually you ask "Tell me about X" and it will summarize what you've said.
Here's where it gets strange. Now you start telling it more things about X, and it will start telling you that you're wrong. It didn't know anything about X before, now all of a sudden it's an authority on X, willing to correct actual an actual authority after knowing just a couple things.
It will even assert its authority and expertise: as "As a language model, I must clarify that this statement is not entirely accurate". The "clarification" that followed was another lie and a non sequitur. Such clarity.
What does ChatGPT mean by "As a language model, I must clarify". Why must it clarify? Why does its identity as "a language model" give it this imperative?
Well, in actuality it doesn't, it's just saying things. But to the listener, it does. Language Models are currently being sold as passing the bar, passing medical exams, passing the SAT. They are being sold to us as experts before they've even established themselves. And now these so called experts are correcting humans about something it literally said it has no knowledge.
If a 4-year old came up to you and said "As a four year old, I must clarify that this statement is not entirely accurate", you would dismiss them out of hand, because you know they just make shit up all the time. But not the language model that can pass the Bar, SAT, GRE, and MCATS?. Can you do that? No? Then why are you going to doubt the language model when it's trying to clear things up.
Language models are going to be a boon for experts. I can spot the nonsense and correct in real time. For non experts, they when LLMs work they will work great, and when they don't you'll be left holding the bag when you act on its wrong information.
I'm concerned that they'll prevent non-experts from becoming experts. Most of my learning is done through observation: if I'm observing an endless stream of subtly-wrong bullshit, what am I learning?
My wife and I were just talking about this exact thing earlier today. I was using an AI to assist in some boring and repetitive “programming” with yaml. It was wrong a good chunk of the time, but I was mostly working as a “supervisor.”
This would have been useless to the point of breaking things if a junior engineer had been using it. It even almost tripped me up a few times when it would write something correct, but with a punctuation in the wrong place. At least it made the repetitive task interesting.
It's not even the rate, which is troubling enough. It's the kinds of things it gets wrong too. For instance, you can say to ChatGPT, "Tell me about X" where X is something you made up. Then it will say "I don't know anything about X, why don't you tell me about it?" So you proceed to tell it about X, and eventually you ask "Tell me about X" and it will summarize what you've said.
Here's where it gets strange. Now you start telling it more things about X, and it will start telling you that you're wrong. It didn't know anything about X before, now all of a sudden it's an authority on X, willing to correct actual an actual authority after knowing just a couple things.
It will even assert its authority and expertise: as "As a language model, I must clarify that this statement is not entirely accurate". The "clarification" that followed was another lie and a non sequitur. Such clarity.
What does ChatGPT mean by "As a language model, I must clarify". Why must it clarify? Why does its identity as "a language model" give it this imperative?
Well, in actuality it doesn't, it's just saying things. But to the listener, it does. Language Models are currently being sold as passing the bar, passing medical exams, passing the SAT. They are being sold to us as experts before they've even established themselves. And now these so called experts are correcting humans about something it literally said it has no knowledge.
If a 4-year old came up to you and said "As a four year old, I must clarify that this statement is not entirely accurate", you would dismiss them out of hand, because you know they just make shit up all the time. But not the language model that can pass the Bar, SAT, GRE, and MCATS?. Can you do that? No? Then why are you going to doubt the language model when it's trying to clear things up.
Language models are going to be a boon for experts. I can spot the nonsense and correct in real time. For non experts, they when LLMs work they will work great, and when they don't you'll be left holding the bag when you act on its wrong information.