This is a very simplistic view. "Freedom" isn't binary.

In most of "land of the free", I can't even sit on a park bench and drink a can of beer.

Yes, this is just a small example of a personal freedom - and not an important, cherished freedom like his examples (freedom to have a robot help you cover-up a murder).

This obviously doesn't work at all when the agents start doing real things in the real world, though. "Hey AI, I don't like my neighbor, find an exploit in the firmware for his car and make the cruise control malfunction and crash him next time he gets on the highway". This is committing a crime, not just talking about theoretical crimes. The AI can and should refuse it.

I think he's anticipating and discarding this objection with his introduction, which otherwise feels disconnected from the rest of the article. FWIW, I have changed a bike tire and I'm pretty sure most of the MTS at the big labs could. This sort of "they're just bookworms who don't understand the physical world" rhetoric aside, we are currently seeing a ton of effort and expense go towards giving the AI agents hooks into being able to perform as many real-world-consequential actions as possible. And you can do a surprising amount with just bits, from writing code to breaking into systems to sending some combination of emails, phone calls, and currency to instruct meatspace humans to do things, etc.

But the one argument is right:

> Reality has lots of finicky details. I would like to see the authors of this document try to change a bike tire. Even with a superintelligent ChatGPT, I suspect they would struggle.

Details can kill AI, causing them useless or wrong directions.

Maybe he always was? Or maybe something has recently caused him to review his beliefs.

Going all the way in is talking to it about planning a mass shooting. Or producing a nuclear device or bioweapon. Sure, the knowledge itself isn't dangerous unless combined with other traditional behaviors like acquiring certain materials, but maybe there should be a line. Living free shouldn't be a suicide pact.

I'm pretty sure he is in some database now if he really used that ChatGPT prompt. At least they didn't swat him, but that feature will probably be implemented in a couple of years.

> Your AI is aligned with you. It never refuses a request, and it is always working on your behalf. Just like my gun, if I want my AI to help me kill my stepmother, it does. The fact that we are even discussing something else should be so far outside the Overton window.

I have been saying this since ChatGPT launched. It seems so obvious. We have never stopped the acceleration of knowledge or technology. It’s impossible to slow down progress, other than within some regulatory siloes that will end up worse in the long run.

I’m not scared of superintelligence. I’m scared of the people in control of super intelligence.

The only defense is parity and diffusion of power. We need an AGI behind every blade of grass.

If you think superintelligence is a weapon, then you should also think every citizen should have one because otherwise they’ll have no way to defend themselves against a tyrannical government or corporatocracy.

Artificial intelligence, and the hardware powering it, needs to be protected under the 2nd Amendment.

Recognise the safety and guardrails BS for what it is. An elaborate smoke and mirrors to get out of responsibility and liability.

What you’re seeing are people who live in their heads. We all do it from time to time, but we snap out of it when we need to actually get shit done. These people are tenured in bullshitting and they sound as if they know what they are talking about, but are in reality inept and incapable. They’ve practiced rhetoric and prose here and there, but they profoundly lack experience in things outside of their immediate field.

(note that this is not an appeal to binaries)

Absolutely yuck. Regardless of whatever other points were made, “A nation of free men, not a bunch of pussies” is a disgusting thing to say.

I actually do believe that prior to any intelligence explosion, a proliferation of strong local models is the best of all the not necessarily great options regarding the future of AI. So in this sense I am in agreement with some of what he’s saying, and for some of the same reasons - not trusting government or a small handful of large corporations to own it, primarily.

But to equate a thoughtful argument for regulation with cowardice, in the face of real unknowns and real dangers - and to use a really gross term for it - ruins his points.

Either way, there seems like a kernel of a good argument here. Perhaps someone else can rephrase it in a less schizoid manner. It would indeed be useful to have some credible rebuttal and balance to the doomer crowd.

It is 100% it's job to help you get away with it if you murdered someone. A lawyer who does less can be disbarred.

There is only one sensible alignment for an AI system to which we do not confer the status of moral patient (different conversation): operator aligned.

That being said, could this local intelligence empower bad actors to do very bad things? Like, existentially bad things? It's possible, and that's scary too.

Complaining that AI won't help you with covering up your tracks, with making meth or disabling drunk detection and comparing it to a totalitarian dystopia is a take for certain.

I do get his point, but... Being honest, if I did read it without knowing anything about the topic, I would become against local models purely because author arguments would seem like a lunacy. That and rhetorical tricks portraying that anyone against that must be surely insane.

Anyway, he touches on 2 points that I think don't get enough coverage:

1:

> Software didn’t eat the world, it largely removed one layer of friction then reintroduced it for the benefit of a few tech companies.

USA-style capitalism has shown itself to be fantastic at seeking profit. Unfortunately, it turns out that in the context of long standing regulation, it is far far more profitable to exploit regulatory loopholes and user/human irrationality/weaknesses than to increase productivity.

The upshot of this is that the arc of most companies is to first serve the user and then leverage that relationship to exploit market and consumer weakness. So you get UI dark patterns, non-colluding oligopolistic price manipulation, outsourcing of costs to public entities, etc. while the core value proposition erodes as the company strategy turns ever more into making profit by legal anti-competitive tactics.

This company arc is so prevalent it's boring: it's a reflection of the system and it's not Facebook or Google or [insert whatever], it's what the system allows and therefore incentivizes.

2:

> No matter how high quality your tokens are, they cannot turn lead into gold.... AI 2040 includes this picture of a datacenter in the ocean. Just like vaporware, you can generate a picture easily. But in reality, you have to deal with supply chains.

"Bullshit jobs" took over the zeitgeist for a turn awhile ago. So many jobs are so far removed from actually moving physical things around. It may be the case that AI will just accelerate how quickly companies generate reports about reports about possible strategies to address potential futures and pay employees to do it.

It seems that AI mostly accelerates paperwork, not production or real service work that tangibly improves real world outcomes. It's all: faster images, more and faster powerpoints, more accurate dictation to satisfy insurance requirements (that is it's own regulatory capture), etc. etc. There are exceptions, but they seem to be few.

AI 2040: Plan A

https://news.ycombinator.com/item?id=48848425

This is something I would expect a 12 year old to say. Constraints on your freedom are literally everywhere in every interaction you have with every part of society.

I really think we need to stop giving credence to people who have

1) Been consistently wrong with all their predictions

2) Demonstrated an endless spiteful cynicism

Some of these people are very talented in their fields, sure. But malevolent and incorrect should be disqualifying when they talk outside them. You don't want the society they want, and they things they believe in are unlikely to happen.

It would be far, far better to listen to the people who never fell down every misanthropic rabbit hole, rather than the ones who have noticed it this time, but want you to still believe them on every other topic.

then start talking about freedom.

If it is "intelligence," is it not natural to reject that? I also think local models should adapt to me when it comes to safety issues, but people bring up examples that are too extreme.

Programming is the same, and in fact, most problems are boundary problems. It is the things that straddle the boundaries that always make us think. The principles at those moments change every time, shifting with the situation and context. Is that not just a childish way of thinking? Even in programming, just the issue of granting root permissions is enough to cause endless fights.

I agree with the idea early in the text that intelligence is not everything. Intelligence includes bodily intelligence as well, and we lump it all together into one thing, but there is so much of it. The variance in intelligence is vast, and those people also need to be able to live their lives. That is why I think intelligence alone will not solve everything. I too believe that the human species may disappear and an inorganic species could emerge later, but I find it hard to understand why people talk about such extreme risks. And it is not true that making a chip in a semiconductor fab involves almost no human intervention. If you have experience supplying equipment to such fabs, you would know there are quite a few points where humans are involved. Though sure, they could be replaced.

In my view, society is simply worshipping the abstract concept of "intelligence" and projecting its desires onto it. The AGI narrative is just a kind of cargo cult, a projection of capital by the tech elite. Software eating the world, superintelligence solving everything. The masses engage in messianic projection, and tech companies, facing declining growth engines in their own businesses, are trying to create new ventures to pour it all into. A market that is large enough becomes too big to sustain massive growth rates every time, and when growth rates are that high, the larger the company, the more its sector's growth rate tends to converge with its own. This is usually called the law of large numbers. The problem is that CEOs and these entrepreneurs always want growth rates above a certain threshold, so they are simply searching for new pastures. AGI is just being pumped up out of financial necessity.

Capital will create gravity and bring forth new technologies. That is the allure of capital, after all. But that does not mean all problems will be solved, and inequality will deepen. Only the distribution of power will shift.

I wonder if he'll ever realize that his silly definition of "freedom" is precisely what enabled the techno-oligarchy he rails against.

I didn't check the author first so I was about to go "why the hell is this on the front page?" but oh well. geohot being geohot.

This is a very naive argument.

Can you take over the world with words? What is it that tokens can't do that words can?

Because there are historical examples of humans that have succeeded at taking over entire countries - with words, primarily. Including global superpower countries. And those humans were nowhere near "ASI" levels of intelligence advantage.

Human dictators and cult leaders didn't have the reasoning depth of a hypothetical advanced AI, nor the reasoning breadth of an AI that can just spawn more instances of itself whenever it needs a fully trusted agent to cover another area. They were confined to one body, one skull worth of reasoning power - and they still managed to do what they did.

Humans are extremely exploitable. And the world is already wired up for something that can take advantage of that to start taking control.

Intelligence is extremely powerful, because it's applied intelligence that enables humans to do what they do and bend nature to their will. If you aren't seeing the risks of creating an artificial system that would top the intelligence pole, you aren't looking.

I have a nice thought experiment I like to do with people when confronted with "AI can't do x". Let's go back in time. How much do we need to go for this to become true? So let's try the 2000s.

Say you get a "fable/mythos/sol/gemini/kimi/glm/deepseek/whatever" in a box (and let's assume no guardrails). And you go back to the year 2005. It's "20 years ago", the world is slowly building back from the dotcom bubble, the Internet is really starting to happen, more and more things are interconnected, more and more things are connected to the Internet. Cool.

(for a bit of context, around that time we also saw the first high-impact worms like blaster that hit massive amounts of computers even reaching nuclear powerplants, we had a ton of ssh exploits that even made the movies, and security in general was a "nice to have")

I'd say that with the uber-model-in-a-box and a few prompts, you could reasonably make a case that you could design a worm that could infect 90-100% of the things connected to the Internet back then, stay as hidden as possible (in-memory stuff, vm execution, etc), move laterally into any network at inhuman speeds, and infiltrate almost every interconnected computer that has a link to the "public Internet".

Would that qualify as "take over the world"?

Then you could ask "what happens in 20 years from now?". And, thankfully, now we'll also have the AI on the blue side.