It's possible that human morality is not sufficient to solve the problems with the world. It wasn't designed to operate at that scale.

It's also quite likely that a "superior" morality — the one that's actually built for a global scale — would be incomprehensible (likely requiring superintelligence to even access), and alien or disturbing from our vantage point.

i.e. it would look immoral to us.

(There's also the thing about civilization already being aligned against the ecosystem and against the happiness of its constituents, but that's kind of a separate discussion.)

In other words, what is good for our sanity or our lakes may be bad for GDP. (Well I guess you don't need superintelligence to tell you that.) We already know what we need to do and we're already not willing to do it.

https://i.imgflip.com/2s99vy.jpg

There are a few related concepts being conflated here.

The quote above is a Transhumanist position; they think that humans need to evolve. I don’t think most transhumanists are AI successionists, though one path is “merge with AI” whatever you think that means. Genetic engineering being the other obvious path. You could argue in some sense that “post-humans” succeed humans, but I think most would argue for a gradual transition where it feels more continuous and values are transmitted/preserved somewhat.

e/acc just says “accelerate AI”. Many VCs seem to think AI will remain a tool, they don’t want to be dethroned. Some e/acc are of course successionist, this is Musk and Beff with their talk about “all that matters is the light of intelligence is spread through the universe”.

Conflating all this is the fact that true successionism is way outside the Overton window and so people likely won’t be honest about their wishes. Larry Page famously espoused the view that it was “speciesist” for Musk to not want machines to replace humans. Most people find this position repugnant of course.

We thought it was a waste of time because the 1990s were relatively peaceful and tech optimism surged.

But you need to teach the classics, philosophy and even religion. No tech people should be allowed near politics in any capacity.

Is it a spiritual question? We’re constantly transforming ourselves with technologies. Taking the most direct example, medicine, how many people considered it a spiritual question when they decided to start taking semaglutide or tirzepatide for weight loss? Or to take SSRIs?

I think the most likely outcome here is that people will continue to alter themselves to meet their immediate needs (hold a job, find love, stay healthy) without a lot of foresight or long term planning. This will likely lead to us increasing our own intelligence, it’s a useful capability across nearly every human endeavor. Whether or not this will happen at a pace that allows us to keep up with and retain control over artificial systems is the first big question, and what the relationship will be between augmented and unaugmented humans is the second. As a humanist, I hope we’re able to answer both questions in such a way that the fundamental dignity of all human beings is respected.

1984. Brave New World. We.

First, it's an extension of meritocracy (in the capitalism sense where poverty is a personal moral failure) as well as Protestant prosperity gospel. That is, you are superior if you're rich. Period. So Homo Sapiens 2.0 is really just "my progeny" from the billionaire's perspective. It's also why the likes of Elon has so many children.

Second, it's just eugenics.

There is an alternate future where humanity gets to enjoy the fruits of automation and AI where we have to work less and have richer lives. That's not where this is going. We are instead bouldering towards vastly increased wealth inequality.

How I see this going is towards the cutting of all public services except policing. AI will enable and enhance the police and surveillance state to protect wealth while everyone else sees lowering real wages and increased desperation. This will give rise to more fascist, apartheid states as some out-group (eg migrants) will be blamed for these ills. And as always happens, the in-group will increasingly shrink.

Ultimately, this leads to neu-fuedalism where the only jobs and housing are on the estates of trillionaires. People own almost nothing.

And if it doesn't go this way, it's going to be very, very violent.