I'm not afraid of rouge agents nearly as much as I'm afraid of us building a permanent underclass of people dependent on whatever scraps the people who devalued their labor decide the grace them with. I'm afraid of the security apparatus that will be built to keep this underclass in line.

The article lists diplomatic actions which might help to manage the risk, starting with "An agreement between America and China" - they all sound like impossible dreams. We had ~80 years of relative peace and prosperity in which to construct a framework of unity to face challenges like AI (and global warming, which until GPT I thought was the bigger risk); international unity is weaker than ever. In geopolitics and defence, capability of other nations is the concern rather than intention; the capability curve of LLMs is heading off our charts. We're backed into tight corners on nuclear proliferation, global warming, and I can see LLM-enabled conflicts (cyber warfare, infrastructure terrorism) pushing us over those other edges. Our democracies seem weakened, and I expect LLMs will empower those using social media to create conflict and control opinion. We're familiar with the cycle of inventing new technology which benefits people, then seeing how long before people invent ways to misuse it. There is a possibility here that LLMs could be used to solve the problems we are juggling, but I struggle to imagine that people won't misuse it even faster.

The article is a start on thinking and talking about managing our risks. The best outcome would be it is so well managed that, like the Y2K "bug", people say "after all that hoopla, nothing happened". I'm not seeing a smooth path to there.

The Fermi paradox could actually also be taken as an evidence that it's rare (at least) for artificial intelligences to take over a civilization and sprawl and survive for very long times

Isn't the preponderant bottleneck in improving the models the need to train them at scale to verify the hypotheses, and the time and cost that it takes?

Or does someone think that they could get magically able to predict big improvements without training?

Or to have AI regulated in their favor

I don't see it. You get explosions when there's a bunch of material that can suddenly ignite or detonate.

AI progress is limited by the data centers and power supply and those are maxed out and not suddenly doubling. The intelligence gradual increase seems more likely.

I've long thought human level AI kind of inevitable, probably since Turing wrote his papers and said computers might pass his test in about 50 years. It actually took about 75 years. It's a steady and gradual process.

Nothing will be able to stop you from pulling the plug.

Humanity has almost never been prepared for anything and almost never successfully prepared when it has chosen to.

almost all progress has been in reaction to change that has already happened.

while on its face this piece is a logical, reasonable stance, it is more likely to become a political foil than deliver anything tangible.

we can either have superintelligence or others will have it and we will not. guns, germs, and steel and all that.

it is bizarre how convinced we are that we can take coordinated action in advance of a threat as a body politic or a species.

Why not, for example, Geoffrey Hinton, the scientist who won the nobel prize for LLM architecture? Or a sci-fi author?

> Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest ai laboratories.

I don't know, they also have an incentive to make their technology seem transformative and powerful, and saying that your technology has the power to cause a massive catastrophe is a way to promote that idea.

Love talking about 'recursive self-improvement' of a prediction engine. What about 'recursive self-degradation'?

The ones talking about how hard it would be to chose which person to run over.

Additionally, I find it hard to believe that this would be a case of the future just not being distributed evenly.

But sure. The AI labs relying on hype stating 15-50% risk of building a magic entity is certainly a reliable number.

The predictions of these "experts" have been drastically wrong for the last 3 years. At what point does someone lose their "expert" title?

Because something AI did or because a human in the loop blindly trusted the result of AI or the promises of AU companies?

This is naïve. The goals of both men have nothing to do with protecting humanity, but rather furthering their own personal agendas.

In Trump’s case, it’s all about amassing more wealth and power.

For Xi, it’s realizing an ethno-nationalist dream where China under the CCP is at the center of world power, the independent nation of Taiwan as well as disputed border areas that are currently controlled by India and Russia and the Philippines are annexed by China, and Xi’s eternal legacy is remembered as the savior of the Chinese people.

International cooperation and touchy-feely rhetoric about saving humanity from AI have no place in either man’s worldview.

> Humanity simply does not have a strategy to ensure it remains safe through RSI.

Turn off the power. It's pretty simple. Leave it to an economist to forget about input costs. Your "super intelligence" only matters if it's actually more energy efficient than a human being and for a million years of evolution humanity is a much harder target to beat than this author seems to realize.

A good chatbot, image/video generator, coder, etc is not going to enslave us all (not in actual sense, although maybe in the instagram sense) or propel us into a golden era.

They are nice tools for some pretty specific tasks and not much more. And we are only seeing the heavily subsidised version of the tools.

I think [AGI] will be an enormous transformative technology, it's going to effectively be a new human era...

We can feel this year, I would say, even though I've been working towards this for 30 years, I think this year with the way the agents are working and tool use, it started to become really useful, still early days of it, but genuinely useful in people's workflows...

And it's not any one thing, it's several different technologies, several use cases, several things that I thought were maybe a bit further out, turned out to be now, that are coming together that make me feel that in aggregate.

I think society needs to hear that because we don't have long to prepare for what that means." -- Demi Hassabis, CEO of Google DeepMind & Nobel laureate

Source with interview clip: https://x.com/deredleritt3r/status/2062223035940139253

EDIT: Ten years ago, most of the skeptics in these threads would have said the success of AlphaFold 3 was impossible.

Still, LLMs are extremely powerful pseudo-AI [2] and will bring a pseudo-singularity. But the impact is still scary if a tiny fraction of humans are augmented 1000x. And as better models become exponentially more costly, only the money people will be able to afford the new models. This is a very likely scenario and scares me to the point I dropped all my projects to work on affordable LLM-based tools to make the difference at most 10x instead of 1000x.

To my elder relatives I explain it like: imagine we are farmers in the 17th century and suddenly out of nowere John Deere tractors, combines, etc. become available. But they cost more to run than all you and your fellow farmers have, so only a tiny handful of rich people take over everything.

[1] https://en.wikipedia.org/wiki/Chatbot_psychosis

[2] https://en.wikipedia.org/wiki/Chinese_room

I don't think it's that high but can we put that aside and focus on the 100% chance that it's being used to enshittify every part of our lives?

'AI Experts', 'superintelligence', and hand-waving doom scenarios.

As an example-

> Within a couple of years, possibly much sooner, AI may achieve so-called closed-loop recursive self-improvement (RSI): the capacity to rewrite its own code to become more capable, without human intervention. Should that happen, the result could be an intelligence explosion of a kind for which there is no precedent and no map.

I've heard these same objections in my lifetime about the internet. And I've read similar arguments against TV, radio, the phonograph, and the printing press.

Honestly, this is getting extremely tiring. Every new invention that has happened has affected the world in both good and bad ways.

Ultimately, what counts is what we do with it.

My aching joints enter the chat...

There is no evidence that we are currently on a path that leads to the Singularity.

There is not yet any evidence that the Singularity is more than science fiction.

It is effectively an article of religion—the Rapture, but for techbros. Indeed, it is what some of them are pinning all their hopes on; after all, if they can recursively-self-improve their way to artificial superintelligence, then and only then could the absurd investments of companies like OpenAI actually prove worthwhile.

The rhetorical structure of talking about uncertain risks and then trying to concentrate the authority to manage those risks in their own hands sounds utterly ridiculous to ordinary people like me.

It's just a simple hypothesis that AI will become uncontrollable to humans once it becomes superintelligent.

Isn't the fact that a reinforcement learning agent improves itself in a specific domain completely different from it recursively improving its own code in a 'better' way? It's just a tool to create a justification for regulation and control using sci-fi fear.

The comparison between nuclear power plant risk and AI risk is also absurd. Where exactly can you define and measure the probability of AI exterminating humanity? It's as unquantifiable as 'I, human JDW64, will become a successful programmer.' What is the measurement standard? Why dress up AI researchers' concerns as objective probabilities? Is it because numbers make it look logical?

The current US-China relationship is in the middle of an AI arms race. The US is strengthening export controls to limit China's AI development, and China is building its own ecosystem. In this situation, I don't understand the idea of cooperating for the common safety of humanity. RAND is an organization that presupposes cooperation—isn't it just a well-written research proposal from an institution that wants to position itself for that role?

Isn't the claim that 'government must step in' ultimately about protecting their own interests? 'A strong government that will protect us' is an authoritarian government. If they were East Asian, they would understand that such regimes have always been used as tools for surveillance and control.

And I don't understand why Fermi's paradox is being brought up here. Why package a software problem as something that inevitably requires strong control? Whenever I see articles like this, I think about what 'intelligence' really means. This person would probably be called 'intelligent' by others. But no matter how I look at it, the holes in the argument are too obvious. It really makes me think that there are different tiers of intelligence.

AI labs have every incentive to overstate the risks so that they can get lucrative government contracts, especially since it’s clearly not profitable going the public consumer route. And if you’re Anthropic you’re even more incentivised to overstate the risks because at this point in the game regulation hurts potential competitors more than it hurts you.

Well, seems it already did.

(Also, Cameron was not quite right with regards to skynet. It is much much dumber but also more effective than displayed in his movie really. Kind of a weird combination if you look at the current AI slop out there.)

Most people out there are literal Normies, and operatelike a robotic vacuum cleaner, or NPC.

   Is it per plant? there aren't a million.
   Is it per year? notably have been at least 2 major ones (arguably 7).

That said, the rest of the analysis and proposal also greatly disappoints me. The idea that the current administrations of US and China could do anything constructive seems hilarious. They're so paranoid and self-serving they couldn't come together, even if there was an alien invasion. Then the idea that LLM safety is somehow as easily traceable as nuclear isotopes and bomb tests seems equally ludicrous. I am sad.