Microsoft 365 outage leads to chlorine gas poisoning of 15 hotel guests

381 points - yesterday at 6:36 PM

Source

Comments

amluto yesterday at 10:43 PM
I personally operate a fairly advanced closed-loop chlorination system, and this is beyond screwed up.

These systems can easily fail, and they tend to fail in a way that causes them to measure low or zero chlorine and thus overdispense chlorine. More below on why.

First, this means that an operator must always ensure safety, both against excessive chlorine and against insufficient chlorine. In a commercial context, that means that a real human (or maybe a totally different sensor design, e.g. a colorimetric sensor) need to measure the pool chlorine level, preferably with a high quality titration (drop test) every day before opening the pool. They should probably record the result in pen on water resistant paper, but this part is icing on the cake.

Second, hoping that an email gets noticed is ridiculous. If you do not have positive confirmation that the pool is safe to open, don’t open it.

Third, the systems should be sized so that they cannot overdose quickly. The maximum dispensing rate should be such that hours of continuous operation would be required to reach an uncomfortable level, let alone a dangerous level. (Hot tubs are sort of an exception here.)

Fourth, everyone should be using a reasonable amount of CYA in their pool. Even at a lowish level of 30ppm CYA it takes a lot of chlorine to become dangerous unless the operator has also lost control of pH. I realize that some codes are confused about this.

Fifth, chlorine is smelly. Someone should have noticed, even just casually. Admittedly, lots of swimmers may be conditioned to stinky pools, and checking for safety isn’t their job. But the staff should notice.

Sixth, the systems I’ve seen beep when there’s an alert. It’s easy to add flashing lights, too. (In fact, the system I’m familiar with automatically shuts down if it feeds for more than X minutes without reaching the setpoint, and you cannot turn this feature off. You can, however, defeat it by switching the power to the whole controller.)

—- sensors —-

Actual free chlorine sensors intended for pools use an amperometric probe immersed in iodide solution or gel, separated from the pool water by a fancy membrane. Free chlorine oxidizes iodide to iodine, and the sensor measures the iodine concentration. If the membrane fails (or general wear and tear or excessive pressure swings) cause the iodide to be lost, then the chlorine can’t react and the sensor reads zero (or low if there’s tiny bit of iodide left).

Interestingly, these sensors are rather expensive devices that appear to mostly come from France and Germany. The Chinese vendors seem to be rebranding European parts (!), and the major US vendor is very clearly reselling a French company’s product.

There are some vendors who sell a cheap (or absurdly marked up) potentiometric ORP-based closed-loop system. In my considered opinion, these are worse than useless.

N19PEDL2 yesterday at 7:28 PM
This is 100% fault of the hotel, not of Microsoft. It’s unacceptable that the communications regarding such a dangerous thing totally rely on email, without any backup or double check.
bketelsen yesterday at 8:26 PM
I'm not even going to read the article. If your hotel depends on some cloud service to prevent chlorine gas poisoning you're doing something terribly wrong. Full stop. I don't know anything about the laws in Germany but this is dangerously close to libel in the US.

Disclaimer: I work in Azure, this is not a defense of the outage and I didn't read the submitted article.

Havoc today at 12:31 AM
Comments seem to be losing a fair bit in translation so rough translation:

* 13 guests complaining about irritation of airways (+2 later)

* 4 taken to hospital, 1 described as medium injury, rest discharged

* Dual failure - dosing system had a defect (dosing too high) and the email of the monitoring failed (M365). Article sounds like it's two distinct systems.

Overall the German text sounds a lot less dramatic than the hn title.

discordance yesterday at 7:30 PM
I know it’s fun to blame Microsoft for things, but that is a poor and dangerous design.

Anything that could result in that sort of harm should have a more resilient and redundant architecture.

k310 yesterday at 7:18 PM
https://www1-wdr-de.translate.goog/nrw/rheinland/rhein-erft-...

> 15 guests injured at Phantasialand

> Due to the (Microsoft) disruption, no emails were received, as the company informed WDR on Saturday. As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool.

labrador yesterday at 9:01 PM
Chlorine gas is no joke. In high school I was making it by running electricity through salt water and then diffusing the gas through pool water to chlorinate my parents pool. My cousin took a whiff when I was distracted and had to go the emergency room for breathing problems. Fortunately he recovered fully.
spaldingcactus yesterday at 9:43 PM
Maybe the information is being lost in translation, but I'm trying to understand where the "chlorine gas" is being talked about. The articles I found just say "too much chlorine", which is hazardous but not the same as chlorine gas. Can anyone shed some light here?
jasonm23 today at 12:17 AM
Not that I have any affection for Microsoft, whatsoever, but THIS, has basically nothing to do with Microsoft.
dekhn yesterday at 7:36 PM
Perhaps as a proximal cause, but not a root cause.
mentalgear yesterday at 7:31 PM
Core services must be as minimal and stable as possible, with safe fallbacks and never be part of a huge internet smart device big tech clusterfuck.
WalterBright yesterday at 11:55 PM
Dangerous systems should always fail-safe.
ctrlaltdileep yesterday at 9:12 PM
its interesting in the sense that emails have sort of become the de-facto standard for sending alerts and keeping paper trails. Even for systems like this. However, I agree with what other folks are saying in that email should never have been the ONLY way someone gets notified about an issue.
0xbadcafebee today at 12:28 AM
Some people are saying it's Microsoft's fault, some the hotel's fault, some the sensor manufacturer's fault. But even if it were just those individual actors at fault, fixing it there won't prevent it happening at another place, and another, and another.

It should be illegal to build a safety system that depends on someone reading an e-mail. But there's no regulation saying "no safety system should depend solely on someone reading an e-mail." There's no generic regulation covering software used for safety. A few select industries and specific cases have limited regulations; but mostly it's the wild west. So we end up with these tragic, preventable accidents.

Governments need to pass regulation about how software can be used in safety-critical applications. At minimum, software should pass a systems engineering safety review, use standard approved methods, and pass independent lab testing. Every other safety-critical device has to follow these requirements. Software should not get a free pass. You can see now what happens when they do.

This event is just one story. There are thousands more like this, buried in the news, or never reported at all. Nobody notices because software is everywhere. You see the symptom but not the cause. CrowdStrike's shitty software ended up forcing airlines to cancel 7,000 flights, causing $500M in damage. Two days ago, shitty software/systems (related to AI) grounded all flights in the USA. In the UK, the BPO Horizon Scandal caused deaths of dozens and ruined hundreds of lives. Regulation can put processes in place to prevent these things. Without it, the problems will continue; with AI involved, it's going to get worse.

People will continue to be harmed by badly designed, badly tested, unregulated software until we do something. Next time it might be your kids poisoned at a water park; by then it'll be too late to act.

MallocVoidstar yesterday at 10:30 PM
Here's a US Chemical Safety Board report involving something vaguely similar: https://youtu.be/35FkCArjLU0 https://www.csb.gov/assets/1/6/yenkin-majestic_report_for_pu...

> During the incident, some of the gas monitors installed within the Yenkin-Majestic facility detected the flammable solvent release approximately one minute after the release began and sent an email to an offsite employee, but the monitors did not sound audible alarms to warn employees of the hazard.

1 dead, 8 non-fatally injured.

gamblor956 yesterday at 10:21 PM
Anyone else notice that the date given by the hotel doesn't match the date of the Office 365 outage?

It appears that the hotel is trying to duck liability by latching onto the outage as the root cause even though this happened before the outage.

thr0waway001 yesterday at 8:48 PM
C'mon can we really blame Microsoft for this? This is stupid.
metalman yesterday at 11:02 PM
screen by pool door and hotel closed circuit pool is open or pool is closed readout of current water conditions, temp, ph, chlorine, turbidity, whatever is possible. simple logic to dissalow abrupt changes without an override. anything goes to zero or infinity, closes the pool.
anjel yesterday at 9:02 PM
@ Risks Digest
bendergarcia yesterday at 8:46 PM
Buncha Microsoft bootlickers in here.

Why does it matter that the service someone is paying for is an api or something else versus email? The email service failed. First and foremost Microsoft’s service failed. It doesn’t matter HOW it should be engineered.

lazzurs yesterday at 7:40 PM
For anyone that doesn't read the link they were hurt but seriously.

This was bad system design. Hopefully people learn that safety critical services can't rely on Microsoft which they also tell you not to do.

rambojohnson yesterday at 9:22 PM
I'm not microsoft shill but that is such a lazy dogshit title for an article about what can easily be seen as dogshit hotel infra management.
JoeAltmaier today at 12:37 AM
Astonishes me that a PC is trusted with lives. My PC should not be trusted.

Story: early days of Windows, they invented a power-save mode. We updated our furnace controller PC with the lastest. The customer complained - it would stop regulating the furnace in the middle of the night shift, just sit there without moving the electrodes for hours.

Upon investigation we discovered (of course) that it went into power-save mode. That cost the customer, but more interestingly, considering a 30MW furnace ran for 8 hours without producing a result meant that it would have been better to have never invented the 'power saving' ecological feature that year - the savings for the nation was burned up in that 8 hours.

tchalla yesterday at 8:46 PM
Since this is Germany, of course there will be no learning because we can do no wrong!

https://how.complexsystems.fail/

dlcarrier yesterday at 7:52 PM
Who needs a horn or a flashing light, when you can just send an email?
mmaunder today at 1:10 AM
Hotel dead man’s switch launches chemical weapons attack on guests. FTFY.
dangoodmanUT yesterday at 11:58 PM
A correct title would be "Hotel negligently manages chlorine gas on Microsoft 365, poisons 15 guests during Nov 13th(?) outage."
refresh89347 yesterday at 10:44 PM
It is 100% fault of New York Times. They've distracted everyone in the IT industry with their "reportage", when Outlook stuff started failing - engineers were reading the news - so no one at Microsoft paid attention. And that wiped out Outlook for everyone. Outlook wipeout caused "no email" and "no office" for everyone for 24-48h, I imagine that people were complaining to IT. So we've got attention spreading thin for the whole IT industry. The result - cascading failures.

Posting a bunch of semi-fake out of context images on New York Times should be investigated, really. Same goes for that "classified" leak they've reported. This wiped out billions from the market, disparaged multiple woman (who likely wanted nothing to do with the reporters), affected travel for tens of thousands, now this gas poisoning.

mococa yesterday at 9:24 PM
Sure, trust on a company that’s 10000% on AI, nothing bad will happen
lwansbrough yesterday at 7:45 PM
I suppose if you really think about it, butterflies are responsible for everything.
leni536 yesterday at 7:58 PM
I assume the backup pigeons didn't arrive in time either.
creatonez yesterday at 9:53 PM
Yeah, and everyone who dies in a hurricane was killed by a particular pattern of wind
ErroneousBosh yesterday at 8:01 PM
BRB, off to read the Therac-25 paper for the third time this month.
rawgabbit yesterday at 9:40 PM
MICROSOFT TERMS OF SERVICE

Use of Microsoft products and services confer no warranties. Written explicit warranties are for entertainment purposes only. Use at your risk. Use of Microsoft products are dangerous to your mental and emotional health. If you experience thoughts of deep frustration, call 1-800-MSCares. All inquiries will be promptly processed in the order they will be received or when the sole contractor can get to it.

londons_explore yesterday at 7:19 PM
Employee finds pool water has dangerous amounts of chlorine, yet instead of locking the door to the pool they just send an email into the void?
joshu yesterday at 11:27 PM
Therac ‘26
jon128 yesterday at 10:01 PM
This time someone absolutely should be fired for buying Microsoft.
MadrasThorn yesterday at 10:14 PM
Dumb as shit, sue the hotel
kittikitti yesterday at 9:38 PM
Microsoft bullies enterprises into their products and targets tech workers who are against it. This is 100% Microsoft's fault and the amount of corporate goons on YCombinator is ridiculous. I don't think this will change at all when the designated response is gaslight and deny.
johnnyApplePRNG yesterday at 7:42 PM
Hot take.
yukIttEft yesterday at 8:38 PM
[flagged]
userbinator yesterday at 7:47 PM
Surprising this happened in Germany of all places; don't they have a huge emphasis on Linux and OSS?
zkmon yesterday at 8:07 PM
> This was because the automatic dosing system had pumped too much chlorine into the pool.

What caused what? There is a story in Mahabharatha where Arjuna was claiming that he achieved killing of Karna. Krishna, the guide, enlightens him about the causal chains and shows how Arjuna' contribution was minimal.

lifestyleguru yesterday at 7:58 PM
lol Germans and the internet. Of course it's the internet's fault, not theirs.