by nik282000
7 subcomments
- I work at a plant with a site wide SCADA/HMI (Siemens WinCC) system, every alarm is displayed on every HMI regardless of its proximity to the machine or even its ability to address the issue. And any given minute a hundred or more alarms can be generated, the majority being nuisance messages like "air pressure almost low" or my favorite " " (no message set) but scattered among those is the occasional "no cooling water - explosion risk".
This plant is operated and deigned to the spec of an international corp with more than 20 factories, it's not a mom-and-pop operation. No one seems to think the excessive, useless, alarms are an issue and that any damage caused by missed warnings is the fault of the operator. When approaching management and engineering about this the responses range from "it's not in the budget" to " you're maintenance, fix all the problems and the alarms will go away".
The only way for this kind of issue to be resolved is with regulation and safety standards. An operator can't safely operate equipment when alarms are not filtered or sorted in some way. It's like forcing your IT guy to watch web server access logs live to spot vulnerabilities being exploited.
- I learned about the consequences of overloading the human operator when working on the primary UI for a manufacturing business. A natural inclination is to put things like confirmation dialogs around dangerous activities. I had managers telling me that one confirmation wasn't enough and that we had to add additional because people were still missing.
Eventually, we tried removing the dialogs altogether and the incident rate approached zero. If you take away the guardrails completely, it radically alters the psychology and game theory around user interaction. Imagine climbing a tall building with multiple layers of protection vs having none at all.
I strongly believe in ideas like "safety 3rd". It's not that I want the humans to be maimed by the machines. Quite the opposite. The difficulty is in understanding higher order consequences of "safety" and avoiding the immediate knee-jerk satisfaction that first order resolutions may provide.
- Excessive alarms aren't just a problem at sea.
There have also been fatal aviation accidents where there's a problem with a common system (a dip in the power supply or hydraulic pressure, or a problem with a critical sensor) and dozens of systems sound alarms at the same time [1].
And for a technology example, a database server disappearing might raise a single alarm, but the applications that rely on that database might raise countless alarms as attempts to connect fail over and over again.
[1] https://en.wikipedia.org/w/index.php?title=Air_France_Flight...
- When everything is important nothing can be important. I was particularly struck by the reduction of crew rest to service alarms. I'd sort of thought this was a problem for watch standers only but it makes sense that some alarms require a specialist or the crews are too thin. Heck, I bet the captain, engineer and first mate all have monitors in their staterooms.
by chriskanan
1 subcomments
- This has been a major problem in hospitals where there are many false alarms and often concurrent alarms for hospital beds: https://pmc.ncbi.nlm.nih.gov/articles/PMC3928208/
- Same sort of problem we have in modern cars? Speed, lane assistance, blind spot, etc, sometimes apparently beeping for the hell of it.
For some it's distracting and frustrating, even increasing aggression and thereby increasing the risk. For others it breeds complacency, a "boy who cried wolf" scenario such that the alarms become meaningless. Either way, it doesn't work as intended.
Interesting to know ships have followed the same pattern, apparently to a worse extent. I wonder how many more walks of life, and industries, are suffering in the same way.
by general1465
0 subcomment
- That was one of the reasons why Three Mile Island melted down - Alarm overloading operators who were then distracted by mundane issues and did not noticed that nuclear core is melting down.
https://en.wikipedia.org/wiki/Three_Mile_Island_accident
- on around 300 days per year i see a "severe weather alarm" on the iPhone Weather app, although nothing at all is happening, completely ridiculous.
- This is how oil pipeline owners get away with leaking millions of petroleum products all over our natural landscape. They legislated to be able to keep using their 60 year old pipeline leak detection methods, which are a false alarm hell, despite better methods existing and being cheaper to maintain. Hire a guy whos job is to stand around and press a button if things go wrong, give them 99 false signals out of 100, then fire anybody who presses the button too often. And pretty soon, nobody ever sounds the alarm when bad shit happens, because ignoring the problem and letting people die is more profitable.
- It's a problem in every field. In aviation, notices about life-threatening conditions can be buried in pages about taxiways closing during certain hours and other nonsense.
- Direct link to the report PDF: https://maritime.lr.org/AM-report-2026
- Follow the incentives. If I’m in charge of setting alarms, and something bad happens without an alarm going off, it’s my fault. If the alarm goes off and the operator ignores it, it’s their fault.
by __turbobrew__
0 subcomment
- When I have over 10 alarms firing on systems I run I find that overwhelming, I cant imagine thousands. There is no way you can operate a system with that many alarms.
- If you want a real life simulation of this, just go into the lobby of a local McDonald's near the kitchen. It's a cacophony and the employees have tuned it out.
by funkyfiddler69
0 subcomment
- > Improvements were delivered through traditional marine engineering interventions, including correcting valve installations, replacing faulty sensors and tuning existing systems.
This sounds like engineers wanted to go traditional routes in the first place but the chain of command wanted to reduce production/constuction time and save money.
by The_President
0 subcomment
- Excessive alarms from machinery could result in the alarm being bypassed. This is an insidious issue as both the alarm condition and the jumper clips could go unnnoticed and result in an event.
- The main purpose of alarms is to relieve automation of liability.
- The solution is called 'log levels' and you should be able to filter out anything you think appropriate.
- For the aviation equivalent, see also the excellent https://fixingnotams.org/
- This was my issue with platforms like Sentry. Turns out, in a decently used prod web app with a lot of dependencies, there's a lot of noise. After spending hours trying to debug some of these only to conclude that they were not debuggable, I ended up kind of letting the Sentry reports fade into the background, and eventually I turned it off entirely to reduce payload sizes.
by fallingfrog
0 subcomment
- Any equipment eventually fails. Designing it to fail safe- in a way that does not threaten the human operator- costs money. But a little warning message costs nothing and ensures that the human can be blamed when something goes wrong. The company can point to the message and say, "see, you should have known."
The message was never intended to help the human operator. It was intended to allow the company to avoid responsibility for cutting corners.
If the goal of the message was to communicate something important to the human operator, extraneous messages would be a serious problem. But if the goal is simply to cover the ass of the company, then extra error messages are not a problem at all. Thats why they never get fixed or pruned.
by paulnpace
2 subcomments
- I didn't look at the PDF, but the sample size reported in the article is 11 ships(!), which makes me wonder how this might look across a larger population of ships.
I wonder how much labor expense has to be saved to make up for a future catastrophic event?
by pwjtnlskshc
0 subcomment
- No need to solve the underlying problem, let's just send all the alerts to AI to deduplicate and prioritize them! /s
Hacker News