Well before tooling is considered, it has to involve people and process. At the highest level, you must have a culture of "blame the process, not the people" or people will do what is natural when things go wrong: try to cover it up and avoid being blamed.

There are procedures in various safety-conscious industries for handling this kind of development. I like that you used the word "systemic" because it is literally a systems issue, not a software, or electronics, or mechanical issue. The entire system has to be considered and analyzed for potential faults.

I spent over a decade writing code for medical devices and while the software aspect of these systems was the most advanced in terms of development process (unlike what many on HN seem to think :-), everything we did had to be considered from a system perspective because even if the individual parts were designed properly, it was possible for the interactions between them to cause problems.

I strongly agree with your emphasis on systems.

Procedures and documentation seem to work well for the aviation industry. Things will still go wrong, but only very rarely twice in the same way. It makes development a lot more expensive but it does work and probably is the only way that we are aware of right now that will get this done in a way that leads to acceptable outcomes.

This leads to glacial progress but I find that is preferable over the 'move fast and break shit' mentality that pervades the software industry.

I think that it leads to glacial progress only because it is done badly. (It is done badly pretty much everywhere). I'm trying to develop tooling to make developing safe software fast: https://github.com/wtpayne/hiai (Long way from being finished, unfortunately).