It's very hard lesson to imbibe, as I've recently learned from interactions with people I'd worked closely with. What do you do when being nice only results in the person getting worse? Some people don't respond to kindness. Some take advantage of the fact that they are physically stronger. What do you do in those situations?

Then you leave. If needed there are a host of other techniques to also use, see my psychopaths book. Doesn't require you to become like them.

Holy shit I can't believe the amount of venom these guys came up with just because some guy wrote a book!

After reading all these war stories from Pieter I'm definitely going to weigh his book on the subject of psychopaths far higher than any controlled experiment psychology BS (those guys can't even reproduce their results most of the time.)

Why did the reddit AMA go badly?

You rough them up.

There is no silver bullet in social interactions either.

That seems a bit at-odds with the advice to "be nice to them", does it not?

To be sure, but that's the 'no silver bullet' clause. You can compromise and gently escalate, though. Be nice, yet firm. Or be ostensibly nice but with a subtext of consequences ("speak softly and carry a big stick").

Or perhaps the ideals fail and you just need to bare your teeth at them. But return to civility; no reason to senselessly burn bridges. (What if they're just horrible? Then leave. What if you just can't walk away quietly? Well, just try not to burn bridges as a first resort, at least not while you're standing on it ;)

I would add to this that being "nice" doesn't mean to cave or placate others.

To me it means to try not to act out of malice, hurt, or anger towards others, even when they seem to be acting that way towards you. You can calmly and rationally explain your position, and hold firm, while trying to avoid negative emotions. Easier said than done, but it can be incredibly powerful.