Looking exclusively at death rates is silly. There are many diseases which otherwise don't kill but make life terrible for those affected and result in huge costs to healthcare systems.
This virus doesn't just kill or leave people unscathed, it also leaves people with long term damage or knocks them out for a couple of weeks. Sitting in bed for weeks is not appealing at all to most people.
Which e.g. EU country is currently tracking the long-term sequelae of COVID? Where is this information available?
So far we've only seen individual reports about lung damage (including potentially life long) in e.g. divers (including for non-stationary disease courses!), reports about kidney damage, mid-term loss of smell, etc.
We're going to get a clear picture in years, so it's irresponsible to say it happens rarely.
The people for which social distancing is a major problem will also have a huge issue with becoming bedridden for weeks on end. Think single-income earners, single parents, etc. Even for a couple with kids this could be a nightmare with one partner taking care of the kids and their sick significant other. Remember that the sick person has to isolate too!
This virus doesn't just kill or leave people unscathed, it also leaves people with long term damage or knocks them out for a couple of weeks. Sitting in bed for weeks is not appealing at all to most people.