Port scanning is generally automated (edit: and scans all ports, not just the handful you listed), so it doesn't actually matter which port you pick as long as it's not 22. Port scanners like nmap are widely available, so the time to actually figure out which port is running SSH is quite short in practice.
Basically, there are two classes of people:
1. Those who use port scanners.
2. Those who do not.
If you are being attacked by someone in class 1, then moving your port gives you absolutely no protection. Thus moving your port is only worth anything at all if the percentage of class 2 people is significant.
However, if you also consider the probability a person in each class has of actually compromising your machine, then the security looks less convincing. Yes, it might be true that 95% of people don't bother to use a port scanner, but the most competent hackers are almost certainly going to be in the 5% that do use one.
Anyway, those 0.999999 that don't scan the ports are doing lame attempts of guessing the password of your box, while the 0.000001 is doing that, but also trying new exploits that have a chance of working.
As a consequence, the chance of one of those 0.999999 bots invading your computer is zero, the chance of one of the 0.000001 doing the same is non-zero.
Who is going to bother with a portscanner when there are plenty of targets on port 22. The targets on other ports are more likely to have been set to use keys, or at least have better passwords.
When you set the system to use keys, all those attempts at accessing just don't happen, and the log does not get polluted. Since the log pollution is the main complaint of people that change the port, I think you are wrong.
There are indirect scanning techniques that induce and measure traffic from ordinary non-compromised hosts. When the attacker notices that he gets banned for scanning he'll do that.
Basically, there are two classes of people:
1. Those who use port scanners.
2. Those who do not.
If you are being attacked by someone in class 1, then moving your port gives you absolutely no protection. Thus moving your port is only worth anything at all if the percentage of class 2 people is significant.
However, if you also consider the probability a person in each class has of actually compromising your machine, then the security looks less convincing. Yes, it might be true that 95% of people don't bother to use a port scanner, but the most competent hackers are almost certainly going to be in the 5% that do use one.