It's a legitimate concern. Although the fuels in RTGs (PU 238) will not engage in a chain reaction, like a traditional nuclear reactor, they could be used to fashion dirty bombs. No such uses are known, but the material has caused serious radiation burns. https://en.wikipedia.org/wiki/Radioisotope_thermoelectric_ge...
The real damage of these dirty weapons is very limited, it's mostly physchological/fear. This stuff itself is not easy to spread with an explosion and it's not a deadly exposure either. You can't make a real nuclear reaction from this.
There are ways to make dirty bombs orders of magnitude more dangerous. The problem is that radiation will spread and contaminate a large area for long time, so the bomb cannot be thrown over border then.
Interesting, thanks for your comment. I just did some further digging and it looks like the fuel is essentially a puck of metal alloy and radioactive material. Nuclear chemistry is so cool!