Most hyperthermophilic archaea die well below many bacterial sporulation temperatures already. Additionally, most hyperthermophilic organisms can't live at body temperature, as that's too low, let alone reproduce AND fight off a host immune system. Prions are a special case and not particularly virulent because they aren't genetically tractable.
When you look at the bacteria that generate spores, the only population that is usually affected is neonates; which is why it's advised not to give honey to anyone under the age of 1-2. For spore forming bacteria, they're not prevalent in the environment and their transmission is usually limited.
