This seems a strange criticism--of course entries in a map don't disappear for no reason at all. Consider using an atomic cache if you are going to stick so many keys into the map that it becomes an issue. Or else bucket the keys into a known set, if you can tolerate the occasional collision.
String.intern does remove references (it used not to and it was considered a major issue). Imagine parsing documents and keeping the words - it works well as long as the input is not malicious. Otherwise words would pile up with no one cleaning them, give it enough time/input to slow the application due to memory pressure, crashing with OOM.
It's leak prone as it'd hold the references forever. It's a cheap and easy way to do it but far from ideal. I'd not recommend it.