I know people will say things like "oh see the tax is actually a loss to <country/state>".

But what loss exactly? he was barely paying taxes anyway? How many jobs was his "family office" providing? how many are lost by him changing a postal address to Delaware?