Compare a swedish guy to a samoan to an aboriginal australian to a mongolian to a subsaharian. It's pretty clear that humans are very diverse, the issue isn't in classification, it's in using the classification to support nefarious ideologies.
The false classification is the foundation of the nefarious ideology. So yes, the issue is the classification.
Humans are very diverse. The extent to which people are different or similar is not accurately captured by the race category. It is dangerously inaccurate.