The irreplaceability of the ice cores is why in my opinion, an independent backup refrigeration system (without common points of failure) is a necessity, along with an independent system that can isolate the primary chillers and activate the backup system should the primary chillers misbehave.
An emergency refrigeration system that works with consumable cryogenic material (say, dry ice or liquid nitrogen https://news.ycombinator.com/item?id=14127835) would be totally feasible and utterly inexpensive compared to the tragic costs of these cores melting.
I don't know the state of readiness of phase-change materials with appropriate transition temperatures (would need to be lower than -40 C) but if such a material exists, it'd work as well.
So I just learned my suggestion of using LN2/CO2 in an emergency refrigeration system isn't something I came up with -- LN2/CO2-powered backup refrigeration is in fact a standard option on ultra-low temperature freezers (the kind that get used to store biological samples and stuff):
Given that it's commercially-available on freezers that get used to store critical biological material, there is no reason that a sample vault that stores irreplaceable ice cores can't have a similar, scaled-up version of this as well.
Curious what sort of redundancy these systems have. I mean, you think of a web production environment... you don't just get an alarm, you typically have redundant systems you can turn on if primary servers fail. Seems like they only had one system running, and when it failed they were boned.
Wonder what it would take to build more redundancy -- seems like samples that are this rare would deserve freezers inside of freezers? Multiple AC units for the room? Or is it the case that these samples can be easily restored by re-drilling in various places? Or is it the case that despite age and rarity, they don't have that much value after being catalogued?
There's more details about the causes of the failure at that URL:
> the refrigeration chillers shut down due to “high head pressure” conditions. Essentially, the chillers were not able to reject their heat through the condenser water system—heat instead of cold circulated through the freezer.
> Compounding matters, the system monitoring the freezer temperatures failed due to a database corruption. The freezer’s computer system was actually sending out alarm signals that the temperature was rising, but those signals never made it to the university’s service provider or the on-campus control centre.
To properly increase ability to respond to failure conditions use of an in-situ system (that doesn't depend on any additional infrastructure) that monitors the temperature before and after the evaporators and if they are adding heat rather than removing it (like what happened in this failure case), would be extremely useful -- indeed, this is one of the proposed changes: "The refrigeration system will also be modified to improve performance during failures, such as shutting down the evaporators in the event of a condenser failure."
As for redundancy, a secondary set of chillers would provide less protection than expected -- they'd likely be running on the same condenser water system and exposed to the same conditions and abuse as each other. Since the melted cores were so expensive to get, implementing an independent refrigeration mechanism (the same way nuclear reactors have alternative cooling mechanisms) that isn't vulnerable to common-cause failures is an utterly sensible approach and should have been done.
A method of providing such a backup source of refrigeration would be liquid nitrogen cooling -- universities/research labs tend to already have bulk cryogenic tanks of liquid nitrogen (that get refilled regularly from a Praxair or Air Liquide liquid nitrogen truck), so provision of LN2 is no issue. In the worst case, an additional LN2 tank would have to be installed to ensure that there's always enough LN2 on-site to last long enough until either alternative emergency refrigeration can be arranged (via emergency chiller rental) or the normal chillers can be repaired. Such a reserve tank would not incur prohibitive expenses and its normal gaseous boil-off can be used as gas nitrogen supply just like the main tank's boil-off.
All that would be needed is a connection from the LN2 tank to the ice core storage facility, a proper set of evaporators/heat exchangers/fans that work with LN2 as cooling fluid, and also a control system that, upon detection of anomalous conditions, shuts off and isolates the normal chillers and activates the LN2 backup cooling system.
There's very few common points of failure between the normal chillers and the LN2 system, and the LN2 system doesn't cool with electrical power and so wouldn't be affected by electrical outages.
Let me take a wild guess here, this new 3-4M dollar freezer with it's expensive shiney new thermostat was somehow for some stupid reason internet connected?
A set of expensive icicles melted. They're going to go collect more samples once they collect enough money again.