Fun fact, there is no historical evidence of an adult human ever dying from a cheetah attack. They are naturally shy, and a lot smaller than you may realize.

It is tempting for anyone raised in the West, and immersed in Judeo-Christian culture. And for anyone, in general, as it offers an epic narration of a personal entity.

Yet, the reality might be messier - IMHO closer to biology than to a weird mixture of computer science and theology. There is no ultimate intelligence (see Karpathy’s starfish shapes), just a collection of adaptability, learning, generalization and self-reference. Also, even an extremely smart being (or process) can be fragile.

So, less God, more WAU from SOMA or the Ocean from Solaris.

https://youtube.com/watch?v=BETHWKaXX4k

We need better scifi! And like so many things, we already have the technology.

This is Stanislaw Lem, the great Polish scifi author. English-language scifi is terrible, but in the Eastern bloc we have the goods, and we need to make sure it's exported properly.

It's already been translated well into English, it just needs to be better distributed.

What sets authors like Lem and the Strugatsky brothers above their Western counterparts is that these are people who grew up in difficult circumstances, experienced the war, and then lived in a totalitarian society where they had to express their ideas obliquely through writing.

They have an actual understanding of human experience and the limits of Utopian thinking that is nearly absent from the west.

Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=34257025 - Jan 2023 (1 comment)

Superintelligence: The Idea That Eats Smart People (2016) - https://news.ycombinator.com/item?id=18499973 - Nov 2018 (248 comments)

Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=13240811 - Dec 2016 (580 comments)

Maciej Ceglowski – Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=13120213 - Dec 2016 (4 comments)

None of that was predicted.

You can even get a literal tiger into a carrier, even though it can kill you easily. You just drug its food and wait till it passes out. This is because you are smarter than it, and know that tranquilizers exist and how to obtain them, which is a strategy that cats of any size are not even able to conceive of, and probably can't understand what happened after it's been done to them.

I like imagining similar discourse when a more basic tool was invented: "A hammer is like a genie, it's all powerful, but, when you hit something with it, it interprets that super-literally, and it hits it."

In fact, if we consider the strongest version of the safety argument for AI, namely one in which the danger is not coming from robots but rather from a disembodied AI controlling our global finances and/or infrastructure, the assumption still does not correspond to reality.

>Such skull-and-dagger behavior by the tech elite is going to provoke a backlash by non-technical people who don't like to be manipulated. You can't tug on the levers of power indefinitely before it starts to annoy other people in your democratic society.

How right the author was.

2026: hold my molt beer

(I love how "connect an ai to the internet" is always the precursor to doom in pre-2022 scifi scenarios, and then as soon as we get something we call ai we hit that big red button)

1. It's hard to put a cat in a box despite us being smarter than a cat, so we're safe. (Counter: we're pretty good at putting cats in boxes when it matters.)

2. It was hard for Australia to kill Emus, so we're safe. (Counter: Australia could probably kill all Emus if it mattered enough, and we definitely accidentally kill off species when one of their inputs for life matters enough to us.)

3. Some smart humans get paralyzed by hedonism or existential angst instead of optimizing for arbitrary goals implied by their arbitrary value sets, so we're safe. (Counter: others overthrow the Czar, land rockets, etc.)

4. Modern AI is data-trained, so recursive improvement requires more data, so we're safe. (Counter: AI-crafted, synthetic data is a thing.)

5. We don't (yet) know how to improve our brains with brain surgery, so we're safe. (Counter: same as #4 above, which unlike us/evolution AI is being deliberately trained to understand and perform.)

6. Children take a long time to grow up, so we're safe. (Counter: the author's own "Premise 5: Computer-Like Time Scales", where they correctly note that computers can be arbitrarily faster than us.)

7. Individual smart humans on a desert island would be cooked, so we're safe. (Counter: nothing says the capability of a single AI must stop at that of an individual human, or that of a small group of smart humans; humans brains got dropped into a savannah and eventually they launch rockets.)

8. If AI doom is not a real threat, believing in it makes you believe some other not-real things that seem crazy or distasteful. (Counter: do we have a clear argument why it is not a real threat yet, in the list above?)

This is the premise I rejected immediately and, if you agree with me, it takes down the whole house of cards. Let me explain. The rationale has nothing to do with "quantum shenanigans."

I have been called religious but will readily concede that of course a physical brain is possible without a soul. What is impossible is to replicate a soul with purely physical matter. Therefore we may understand that "superintelligence" is possible, and maybe inevitable on the long thread of time, but - crucially - it will never be able to approach that supernatural element present in us (the spark of the godhead) and therefore never be able to replace humanity.

In that sense it is like any other natural disaster that threatens to make us extinct, but it is not some "superhuman" nor anything close.

It's a horror game and it explores all kinds of fascinating and disturbing scenarios. Simulations of human minds. Artificial worlds. Human minds in robot bodies. Genetically modified humans. Man-machine hybrids etc.

(A great exploration of the substance/structure matrix, by the way. My favorite question in AI and consciousness. Is the special sauce in the material, or its shape, both, or neither?)

The very question of aligning the AI with humans assumes that we have a very robust definition of what human means in the first place.

Ostensibly the AI was aligned. It did succeed in keeping humans alive! But it did that in all sorts of ways that mostly made them wish it hadn't.

We don't have a good definition of intelligence

Also, the premise that this thing would take over, it's hard to reason why it would do so

Anthropocentrism is also problematic in this article.

But even if you find them persuasive, there is something unpleasant about AI alarmism as a cultural phenomenon that should make us hesitate to take it seriously.

First, let me engage the substance. Here are the arguments I have against Bostrom-style superintelligence as a risk to humanity"

--

The framing here seems to me to equate "AI risk" and "AI alarmism" with buying in to belief in "Bostom-style superintellgence".

I'm not sure if the author meant to put anyone who is alarmed by developments in what we're calling "AI" into the same bucket as "AI obsessives want to make it into a programming problem, by designing a God-like machine", but I think this conflation is unfair and, frankly, dangerous.

I don't know what superintelligence is. I don't even know what intelligence is. And I don't really know what either "artificial" or "general" mean either when talking about "AGI".

You can believe, as I do, that these things can be, and will inevitably will be if we don't radically correct course, used to do very bad things independent and short of being "God-like". When you have systems which can hypothesize, synthesize, and test thousands if not millions of potential infectious agents in bulk [0], and can then order the ingredients for you from dodgy websites via some "claw", and then when you put these systems under the unsupervised control of millions of people with varying levels of stability and altruism, something extremely bad is exceedingly likely to happen.

I understand that 2016 is ages ago and things change, but I came away from the article with the impression that if I'm worried about AI risk then I'm a clown like the three pictured in the "Outside Argument" section (you're a Google-Glass-wearing cringe nerd if you're alarmed). Maybe that's my fault and I'm not smart enough to understand the actual point of the article. If I have misinterpreted, I welcome the correction.

[0] https://www.nature.com/articles/s41467-024-53759-4

The former is no challenge to the premise, but the latter? That is a different story.

EDIT : For S&G I asked Claude about it. It replied :

The talk groups Penrose with the religious doubter, as if the two objections were the same species and could be dispatched by the same gesture: most of us find this easy to accept. But that's a headcount, not an argument. The religious objection can be set aside because it rests on a premise (the soul) the materialist simply doesn't share. Penrose's can't, because it's pitched entirely inside the materialist frame — Gödelian limits on algorithmic understanding, non-computable physics in the substrate. You don't get to wave that away; you have to show it's wrong. The talk does the former and pretends it's done the latter.

The entire superintelligence thesis is a wager on the authority of intelligence — that smarter minds see further, judge better, and that this is precisely why we should fear or defer to them. If you take that seriously, then dissent from the very smartest humans on the exact question of whether minds are substrate-independent is the most expensive dissent available. You can't venerate intelligence as the thing that settles everything and then file your most intelligent objector under "outliers, moving on." The move is self-undermining on the argument's own terms.

We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators.

We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intelligence paradigm to cross.

What I am afraid of is the existing superhuman agents (companies, governments and religons) will produce AGI or superintelligence and then proceed to use it as cognitive mitocondria, even further deepening thier supremacy in the cognitive ecosystem.

This article is from 2016; now it doesn't feel like backlash is strictly a function of manipulation.

The monkey's paw. You know, you don't need superintelligence for that.

Civilization was already doing this. "What if we just gave ourselves exactly what we wanted." Well, it turns out often that's not so good!

What would be a way to recursively self-improve algorithms for matrix multiplication (foundations of machine learning and inference)?

Well we can do the wager. If it's a nothingburger, then the worst case scenario is that we approached AI too cautiously. (Ha. What are the odds of that?)

If it's not a nothingburger, then we all die, unless the whole world agrees on the correct course of action in advance and coordinates perfectly. Hmm.

Well, maybe we don't all die, but the world is irreversibly transformed into something incomprehensible and repulsive.

Although, I don't really think we needed AI's help for that one. We should probably figure out how to align ourselves before we try to preach to the next species. I'm not exactly holding my breath though :/

2016 https://news.ycombinator.com/item?id=13240811

2018 https://news.ycombinator.com/item?id=18499973

2023 https://news.ycombinator.com/item?id=36098332

Let's talk about Billionaire Alignment, Economic alignment, Human alignment.

Classware should be M.A.D. -- in that it shouldnt even happen.

no we don't...

not sure where this notion comes from that if enough public figures are worried about something, then we must also

Afaik no-one that is actually working on AGI is anywhere close atm.

It starts of interesting and then goes into lots of nonsense and non-sequitars when it start its takedown. (note: I'm not an AI alarmist, just reading the talk)

Arugment from Wooly Definition: An irrelevant argument - we don't need more intelligence. All we need is human intellegence + duplication and communication. An AI can clone itself immediately with its existing knowledge. A human can't. And AI can transmit thoughts perfectly "I know kung-fu style" a human can't

Argument From Stephen Hawking's Cat: This is also irrelevant. The arugment is supposed to be against Superintelligence but this argument is against controlling it, not against it happening.

Argument From Einstein's Cat: more of the same

Argument From Emus: more of the same - we can't control it

Argument From Slavic Pessimism: also not an argument against superintelligence.

Argument From Complex Motivations: Not an argument against superintelligence. Only an argument that some intelligences have mental issues

Argument From Actual AI: this didn't age well

Argument From My Roommate: not an argument against superintelligence. Only that some intelligences aren't motivated.

Argument From Brain Surgery: Not even sure that this is saying? It seems to be saying you need to learn stuff? Yea, people learn, AI can learn.

Argument From Childhood: Not an argument against AI. (1) unlike humans, AI can duplicate with full knowledge. (2) AI can learn faster than humans. Already proven.

Argument From Gilligan's Island: It takes a village is not an argument - AI can also specialize if it needs to.

Grandiosity, Megalomania, Comic Book Ethics: These argument that the people who believe in it often feel they should be charge. I agree that's true and bad. This is not an argument against superintelligence.

Transhuman Voodoo: This is an appeal to "these ideas sound too incredible therefore you should not believe them". Not sure how that's an argument

Religion: Agree, people who beleive and seem and maybe are religious. That's not an argument against superintelligence.

Simulation: Non-sequitar. This is "some of these people believe other crazy stuff QED no superintelligence". That's not an argument against superintelligence.

Data Hunger: This is actually an argument supporting the superintelligence believers. They believe sucking up all the data is bad. Not sure what argument is being made here relativel to superintelligence.

... and I stopped ... What a waste of time

If there exists a path of runaway superintelligence, the trajectory we've experienced has been following it to a tee. Their predictive power was affirmed.

All the "AI is a nothingburger" predictions of the last decade, including many here even in the last year, have aged incredibly poorly.