I can restart window manager and compositor just fine in X. Also it is not generally true that X apps crash when the server goes away. This is a limitation of some client libraries, but I wrote X apps myself that could survive this (or even move their display to a new server). It is of course sad that popular client libraries never got this functionality under Linux, but this is a problem of having wrong development priorities.