Many boys spend nearly their entire first 18 years of their lives stewing in an extremely toxic media/community/family ecosystem where sexual virility and your ability to have sex with women is _directly_ correlated with status and success in life. Similarly, your in-abililty to attract members of the opposite sex is _directly_ correlated with your status as a loser and station in life.
Short answer: because we're genetically programmed to care about it. And no amount of reasoning or handwaving or logical thought will change what the genes encode.
Sensitivity to status exists in all primates - not just humans. It's one of the most basic instincts of an animal.
We are just vessels for the reproduction of our genetic code (although we believe in our own free will). Status is directly linked to the probability of the genes reproducing, ergo short of some weird chemical drugs, we will never be able to ignore it.