- > Automation doesn't make operators more careful. It makes them forget how to be. The more reliable the system, the less ready the human.
The entire premise of a system is that it removes the need for careful attention.
system: signal lights tell me whether or not I can pass through an intersection, so that I do not have to attend to potentially high speed traffic from a variety of directions.
system: the side my knife blade sits on my arched guide fingers, so that I do not have to attend to the edge of the blade or the location of my fingers.
etc etc.
- There's been many studies around the concept "how do you get expertise? In what field can you actually develop expertise?" such as Naturalistic Decision Making [1] [2]. The wisdom from there can be applied to the question of automation.
If your task is for example, to pilot a preset route with stable condition and very low surprise, you will fall for the "getting too comfortable" trap and we tend to start to get lazy(or efficient) and offload the mental effort and skills atrophy. A common workaround to this is to have regular training(deliberate practice) that introduce the "tricky" situation to keep the skill up. Problem ofc arises when people don't keep this up.
This can be seen in the diagnostic performance differences between junior and senior doctors, not always in the favor to the senior [3]. If you add a layer of automation but the insight gathered by working on that layer is great (and falls off) then deliberate practice start to become a requirement
[1] https://commoncog.com/putting-mental-models-to-practice/
[2] https://youtu.be/5eW6Eagr9XA?si=Y9exacaW-F4PDOKF
[3] https://pubmed.ncbi.nlm.nih.gov/26375267/
by singpolyma3
4 subcomments
- I don't think we believe that the automation makes the human operators safer, but that it is overall safer than the human operator alone.
by supriyo-biswas
0 subcomment
- Also, ironies of automation[1], etc.
[1] https://static1.squarespace.com/static/644321e78cd2dd37613af...
- Automation should always be the serialization of understanding of the system, which can execute on its own under normal conditions. When it breaks, your reaction should not be removing the automation and asking humans to get back to manual operation, that is the wrong approach. Instead, you should ask humans to leverage their understanding to step in, reason about the automation system, and fix it.
This is how I have been doing SRE for the past decade. And the inability to practice this under-the-hood reasoning is the main reason why I don't trust any modern AI based automation systems. They are helpers for some manual work, but they can't be automation.
by eternityforest
0 subcomment
- There are two other solutions, make the automation so reliable that the human skill is not needed, or make the consequences of failure low enough that you can just accept failure.
As far as I know, Places that actually care about safety apply all of them at once.
They add layers of failsafes that don't rely on humans, they make the automation better and better, and for the most critical stuff, they continue training for things that will almost certainly not happen.
And for the rare(depending on personality) cases where we care about the pilot's ability more than the result, just don't automate at all.
- From my manufacturing experience on both ends of the spectrum of highly automated versus manual, the best analogy that comes to mind is driving a manual/stick versus an automatic car. If you know how to drive a manual, you can adapt to an automatic very quickly. The reverse, of course, is not true.
There are, of course, many benefits to automation such standardisation, measurability and the list goes on. Plus cuurrently we have this sweet spot
where the workforce contains several generations who have experienced both very manual and highly automated processes. This dual experience is invaluable for investigation and continuous improvement. It makes me wonder what will happen when the workforce consists entirely of operators and engineers who simply press start most of the time.
- It's interesting living through all this civilizational flailing. Remarkable. I read that when people invented writing it was very controversial because the facility of memory was the prized intellectual capability of the time. I never thought I'd see this kind of thing in person
by analogpixel
1 subcomments
- They should get an auotpilot to fix their website to render on desktops and not just phones.
by ninalanyon
1 subcomments
- I'm not convinced that this is necessarily always a bad thing on balance. It seems to me that automation in safety critical situations might make things worse when they finally go wrong but also that the number of times that they go wrong at a lower level is so much reduced that the result is net positive.
Also if you know that automation induced complacency is a thing then it must surely become a target for training, surveillance, and adaptation not mere hand wringing.
- In this case the only reason I'm a pilot is the existence of autopilot haha
by jonstewart
1 subcomments
- Am I missing something? Is TFA only 2-3 paragraphs of a generic metaphor, with no actual data/research from aviation (or other fields) to back up the core thesis?
by mschuster91
0 subcomment
- In Munich, the U-Bahn subway normally drives semi-automatically (i.e. the driver only opens and closes the doors, then hands over control to the computer), but every driver has to regularly drive without automation assistance ("Fahren nach ortsfesten Signalen") to practice.
- The crew and the plane are a single system. It is meaningless to imagine pilot “skill” without the plane. Further, we killed so many pilots in the 20th century through impossible workloads which we have now automated, it’s almost cruel to be wistful for it.
I know this is an analogy to AI, but I wish we would dispense with this idea that there’s some appropriate level of machinery which was reached just a hair before right now. There is no appropriate level of machinery, no point at which the nature of the system itself will unambiguously say “that’s enough.”
- And that's the whole freaking point. If the humans have to obtain and maintain the exact same skills as before than what's the point of automation?
Actually this could be a side channel to measure the efficiency of automation. If the human operators are just as good in every aspect as before, we know all the money put into automation is wasted.
- This is the concern I have with all the driving aids: adaptive cruise control, automatic braking, lane departure assist, et cetera. It all leads to drivers paying less attention to the road and more attention to their @#$%&*! phones because "Muh car drives itself."
by swordlucky666
0 subcomment
- [dead]
- American Airlines captain Warran VanderBurgh once called this phenomenon "the children of the magenta"[1] in his talk on automation dependency in the 90s.
I wonder what we'd call the children today in hindsight and what line they're chasing now...
[1] https://www.youtube.com/watch?v=5ESJH1NLMLs
- Yet another AI written slop article hitting the top of HN.
Evidence: Look at the most recent article on this blog: https://julienreszka.com/blog/difficult-conversations-don-t-...
"Memory is reconstructive. When someone recalls events differently, it feels like gaslighting. It usually isn't. Document first, then negotiate."
Bleugh
by 21asdffdsa12
1 subcomments
- The more the pilot flies, the worse the pilot- long distance vs short distance, can create the absurd problem that you forget how to land and start, the most important pilot tasks, while flying long haul.
Hacker News