Microsoft 365 outage leads to chlorine gas poisoning of 15 hotel guests
381 points - yesterday at 6:36 PM
SourceComments
These systems can easily fail, and they tend to fail in a way that causes them to measure low or zero chlorine and thus overdispense chlorine. More below on why.
First, this means that an operator must always ensure safety, both against excessive chlorine and against insufficient chlorine. In a commercial context, that means that a real human (or maybe a totally different sensor design, e.g. a colorimetric sensor) need to measure the pool chlorine level, preferably with a high quality titration (drop test) every day before opening the pool. They should probably record the result in pen on water resistant paper, but this part is icing on the cake.
Second, hoping that an email gets noticed is ridiculous. If you do not have positive confirmation that the pool is safe to open, donât open it.
Third, the systems should be sized so that they cannot overdose quickly. The maximum dispensing rate should be such that hours of continuous operation would be required to reach an uncomfortable level, let alone a dangerous level. (Hot tubs are sort of an exception here.)
Fourth, everyone should be using a reasonable amount of CYA in their pool. Even at a lowish level of 30ppm CYA it takes a lot of chlorine to become dangerous unless the operator has also lost control of pH. I realize that some codes are confused about this.
Fifth, chlorine is smelly. Someone should have noticed, even just casually. Admittedly, lots of swimmers may be conditioned to stinky pools, and checking for safety isnât their job. But the staff should notice.
Sixth, the systems Iâve seen beep when thereâs an alert. Itâs easy to add flashing lights, too. (In fact, the system Iâm familiar with automatically shuts down if it feeds for more than X minutes without reaching the setpoint, and you cannot turn this feature off. You can, however, defeat it by switching the power to the whole controller.)
â- sensors â-
Actual free chlorine sensors intended for pools use an amperometric probe immersed in iodide solution or gel, separated from the pool water by a fancy membrane. Free chlorine oxidizes iodide to iodine, and the sensor measures the iodine concentration. If the membrane fails (or general wear and tear or excessive pressure swings) cause the iodide to be lost, then the chlorine canât react and the sensor reads zero (or low if thereâs tiny bit of iodide left).
Interestingly, these sensors are rather expensive devices that appear to mostly come from France and Germany. The Chinese vendors seem to be rebranding European parts (!), and the major US vendor is very clearly reselling a French companyâs product.
There are some vendors who sell a cheap (or absurdly marked up) potentiometric ORP-based closed-loop system. In my considered opinion, these are worse than useless.
Disclaimer: I work in Azure, this is not a defense of the outage and I didn't read the submitted article.
* 13 guests complaining about irritation of airways (+2 later)
* 4 taken to hospital, 1 described as medium injury, rest discharged
* Dual failure - dosing system had a defect (dosing too high) and the email of the monitoring failed (M365). Article sounds like it's two distinct systems.
Overall the German text sounds a lot less dramatic than the hn title.
Anything that could result in that sort of harm should have a more resilient and redundant architecture.
> 15 guests injured at Phantasialand
> Due to the (Microsoft) disruption, no emails were received, as the company informed WDR on Saturday. As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool.
It should be illegal to build a safety system that depends on someone reading an e-mail. But there's no regulation saying "no safety system should depend solely on someone reading an e-mail." There's no generic regulation covering software used for safety. A few select industries and specific cases have limited regulations; but mostly it's the wild west. So we end up with these tragic, preventable accidents.
Governments need to pass regulation about how software can be used in safety-critical applications. At minimum, software should pass a systems engineering safety review, use standard approved methods, and pass independent lab testing. Every other safety-critical device has to follow these requirements. Software should not get a free pass. You can see now what happens when they do.
This event is just one story. There are thousands more like this, buried in the news, or never reported at all. Nobody notices because software is everywhere. You see the symptom but not the cause. CrowdStrike's shitty software ended up forcing airlines to cancel 7,000 flights, causing $500M in damage. Two days ago, shitty software/systems (related to AI) grounded all flights in the USA. In the UK, the BPO Horizon Scandal caused deaths of dozens and ruined hundreds of lives. Regulation can put processes in place to prevent these things. Without it, the problems will continue; with AI involved, it's going to get worse.
People will continue to be harmed by badly designed, badly tested, unregulated software until we do something. Next time it might be your kids poisoned at a water park; by then it'll be too late to act.
> During the incident, some of the gas monitors installed within the Yenkin-Majestic facility detected the flammable solvent release approximately one minute after the release began and sent an email to an offsite employee, but the monitors did not sound audible alarms to warn employees of the hazard.
1 dead, 8 non-fatally injured.
It appears that the hotel is trying to duck liability by latching onto the outage as the root cause even though this happened before the outage.
Why does it matter that the service someone is paying for is an api or something else versus email? The email service failed. First and foremost Microsoftâs service failed. It doesnât matter HOW it should be engineered.
This was bad system design. Hopefully people learn that safety critical services can't rely on Microsoft which they also tell you not to do.
Story: early days of Windows, they invented a power-save mode. We updated our furnace controller PC with the lastest. The customer complained - it would stop regulating the furnace in the middle of the night shift, just sit there without moving the electrodes for hours.
Upon investigation we discovered (of course) that it went into power-save mode. That cost the customer, but more interestingly, considering a 30MW furnace ran for 8 hours without producing a result meant that it would have been better to have never invented the 'power saving' ecological feature that year - the savings for the nation was burned up in that 8 hours.
Posting a bunch of semi-fake out of context images on New York Times should be investigated, really. Same goes for that "classified" leak they've reported. This wiped out billions from the market, disparaged multiple woman (who likely wanted nothing to do with the reporters), affected travel for tens of thousands, now this gas poisoning.
Use of Microsoft products and services confer no warranties. Written explicit warranties are for entertainment purposes only. Use at your risk. Use of Microsoft products are dangerous to your mental and emotional health. If you experience thoughts of deep frustration, call 1-800-MSCares. All inquiries will be promptly processed in the order they will be received or when the sole contractor can get to it.
What caused what? There is a story in Mahabharatha where Arjuna was claiming that he achieved killing of Karna. Krishna, the guide, enlightens him about the causal chains and shows how Arjuna' contribution was minimal.