This is a valid point, but the “correct” behavior in that instance should be to communicate the constraint, instead of claiming that it would be followed. This is certainly more innocuous than most confident BS-ing that these models do, but it’s still an instance of it. In the analogy, it’d be like standing in front of the car and having it stop, but still tell the passengers that it’s moving at 60mph. The constraint itself isn’t the problem