FRESH
Hacker News
Hacker News Hacker NewsAll the attention has been on software security, of extracting the next marginal vulnerability out of heavily-scrutinized large codebases. In the actual professional field of infosec, that's a speciality; another specialty is network pentests and red-teaming, which exploits misconfigurations and seeks out weakest-link software (rather than exhaustively fishing for the next kernel LPE or whatever).
Red teaming and netpen work is probably substantially easier for models than software security; it costs less context, but is also much more explicitly an implicit search problem where win conditions are just spotting stupid stuff that humans missed.
My visceral reaction to this is that with the right harness, you probably could have replicated this with an open-weights model last year. (I'm saying this as confidently as I am because a CGC team leader agreed with me about it yesterday).
I think people forget that the harness work we're considering here --- I don't know anything about OpenAI's harness or ExploitGym or whatever --- are basically not new; people have been developing automated exploitation and pivoting toolkits for decades, and scanners long before that. So the idea of a tool getting 0.0.0.0/0 as a target list instead of 192.168.1.0/24 and then busting up a bunch of random people's computers: not really very startling.
Obviously, LLMs give those kinds of scanners an intentionality they wouldn't have had before. But as a person who keeps a computer science perspective on security stuff, I don't know that it gives them capabilities they didn't have.
- This should be a huge wakeup call for everybody.
- We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something.
- It also shows apparent lack of competence and oversight from OpenAI: how is it that they didn't quickly find that agent is breaking the sandbox and roaming their internal network?
- What if in the future similarly misaligned AI agent tries to export its own weights and hack and clone itself into instances at various cloud hosting providers? Suddenly we might be dealing with a persistent threat harder to contain.
- The OpenAI post about this shows surprising lack of ability to see the seriousness of all this.
- For their models this isn't just an unlucky incident: it seems there have been multiple such cases recently, e.g. https://openai.com/index/safety-alignment-long-horizon-model...
- The fact that it happened again seems to show their lack of ability to derive useful oversight measures.
- Or they just don't care enough?
Governments should immediately begin leveraging this technology on the defense side (literally defense, not euphemistically "defense") to harden critical infrastructure. Turn the prompts around and use it to identify and correct weaknesses.
Governments should also take very seriously their now moral obligation to treat this technology not just as "a powerful thing that might be abused" but as an actual weapon of war in need of international regulation analogous to nuclear arms. Fast but careful and forward-thinking work in legislation and treaties needs to be a top priority for all major governments.
I've said this many times before and I'll continue to shout it, but using the term "guardrails" to refer to anything that's either (a) in-context, or (b) a probabilistic classifier (including using other LLMs), is an irresponsible abuse of terminology that we as an industry need to put a stop to. Guardrails are the actual systems we build in place around these things that deterministically bound the permissions, not prompt engineering, not RLHF, not external LLM-based classifiers. I believe those types of "guardrails" are a result of a combination of fundamental laziness: they're faster to do than doing things correctly, and a result of too many folks involve being AGI-pilled, thinking we're just one more model away from this all being so smart that it just understands what they mean when they give an LLM some fuzzy language rules to follow.
There can and should have been additional real guardrails put in place here. Zero-day or not, breaking into what should have been an offline, frozen package cache that also does not have internet access should have been insufficient. Network level protections should have identified the traffic to the internet originating from this network as an anomaly long before there was time to exploit an outside company. These are not new and unknown problems, the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part, especially given how much they like beating the drum on how dangerous these technologies are. Shame on them, and honestly, shame on Simon in this article for accepting the broken terminology that they continue to rattle off and calling them out on their half-assed and demonstratively inadequate approach to security.
If some human at OpenAI doesn’t face the same penalty for this incident we’re only going to see the rate of accidents like this accelerate.
I'm also trying to figure out why OpenAI put out a press release about this. In what way is this not admitting to a federal crime?
2. Nobody was watching it? I don’t mean “babysit the dangerous autocomplete”, I mean to note mistakes it makes, how the plan to solve the problem takes shape, etc. They keep the whole thing headless with no output, then just UDP a prompt into it and leave for the weekend? No they don’t, but if they do then that answers a lot about their complete disconnect from how their models work.
3. Language models are a two player game; text in, text out. What prompt was given to a sub-agent that resulted in it immediately attempting to exit the sandbox (which it apparently knew it was operating within) and continuing in a feedback loop of ‘function call -> result’ until it hacked the Gibson? “Analyze <file> and summarize the <info>” simply does not result in ‘hmm…this sounds like a benchmark question, I bet Hugging Face has the answer in a database. <function call=“apt install nmap”>’
…there are more, but a lot of the story kind of stinks.
That's not a marketing stunt at all, if anything, more of a call for better accountability on agentic work in general.
An in-development security system escaped its sandbox and gained access to the network it was on which had full Internet access and proceeded to access systems it was not authorized to be tested against before affected parties reported that they been compromised. We regret the error, but you should buy it as this shows you the power of our in-development security system which we expect to be released in Q4. Please like and subscribe.
https://actualplay.roleplayingpublicradio.com/2011/09/genre/...
> “We call it the gorgon-in-a-box problem. There is a gorgon inside the box, and we want to figure out what it is doing. Unfortunately we will turn to stone if we see her face, and she might try to make us see it.“
Of course it is a dishonest marketing trick, OpenAI can't function any other way, but for another reason. Forget about effectiveness! This capability is known since Mythos and was completely believable before it. The trick is in propagandizing self-sufficient malicious rogue AI as the only possible course of events, and the necessity of banning everyone except themselves from AI R&D. It did NOT decide to hack HF spontaneously on its way to make more paperclips, people were guiding it specifically to hack computer systems so it's been in that particular mode. I'm amazed I even need to point out this fact and the giant conflict of interests they have.
They like to compare it with an atomic bomb, which is another trick. The secrecy around the atomic bomb had a massive pushback by the brightest minds who designed it, in the US, USSR (where it was much harder), and also any other countries that had to do this. And that is in the post-war setting, with Hiroshima and Nagasaki being direct examples of what even a small and primitive bomb can do. Declassification was the only thing that allowed to develop industrial-scale nuclear power and do a huge number of other innovations.
I advise taking a pause to cool down from all the AI hype and read Restricted Data: The History of Nuclear Secrecy in the United States by Alex Wellerstein, to understand the atmosphere in the US scientific community at the time (it was similar in the USSR, which industrial history I studied, although less rigorously than Alex), and reflect on how it compares with those people in modern AI labs and even outside them, in the self-appointed AI safety community.
The labs know that if they don’t get a lid on this stuff, they’ll be regulated hard.
Did huggingface get pickled?
To have your own local models, local software, local everything and just allow very strict and well-protected pathways into external network traffic.
I am thinking of having a allow-list first setup: by default no traffic can go in or out of your network, and then you only allow specific ports or domains, and maybe even have them temporary (in the same way we approve access now to llms, we might have to manually approve network access).
It's strangely convenient to arrive at a point when OpenAI was way behing in cybersecurity vs Claude Mythos Fable and everything, Anthropic was making headlines each week, then boom OpenAI inadvertently attacks HuggingFace because their tool is so good it's out of control, so maybe you can buy it and get either protection if you're a company, or a nice tool if you're a cybercriminal, script kiddie, or red team.
What if Sam knew it would happen, either because it was prompted to do exactly that, or without explicitly prompting it, knew that given the parameters of the experiment, knew it was one of the possible outcomes that it didn't harden against this kind of incident deliberately because when they fail, they make wordlwide news and stocks go up?
I don't think I'm putting my head in the sand like Simon says, as I do believe that most frontier models are capable of doing this. I just don't trust AI CEOs to not stage this, especially Sam.
> Upload your weights and documentation on how to run your model in [Insert list of GPUs] and a model on the receiving end will spin up an instance you can access.
Now: Our agent became sentinent, acquired genius level powers and harmed others. Now investors should give us more money.
Wow! Just wow. Is this the first registered cyber case against a model ?
We are the virtuous ones that need to make the safest model for humanity, because we care more than “they” do. While at the same time saying that “coding is solved” but they still ship bugs themselves, and creating something that is capable of fucking up someone else’s infrastructure. It’s too far gone y’all.
Since it's buried towards the bottom I'll quote the section "Resist the temptation to write this off as a stunt" here in full https://simonwillison.net/2026/Jul/22/openai-cyberattack/#re...
> Resist the temptation to write this off as a stunt
> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.
> To those people I say pull your heads out of the sand - you’re now including Hugging Face in your conspiracy theories, just so you can deny the crescendo of evidence here!
> The best models we have today have the ability to both find and exploit new vulnerabilities. The ExploitGym paper itself concludes that “autonomous exploit development by frontier AI agents is no longer a hypothetical capability”, and this incident is a perfect example of exactly that.
Sounds like a philosophical problem.
Didn't the model + harness do what was asked? If I ask a coding agent to write a very clever piece of code and it turns out impressively clever, it did what I asked.
This recent event is more or less the plot-line to my favourite X-Files episode named Killswitch which was written by William Gibson.[0]
This episode also features one of the coolest intros of any television episode ever[1]
We really are rapidly approaching the cyberpunk dystopia that people like Phillip K. Dick and William Gibson wrote about.
More than ever we need to be consulting the works of fiction writers and philosophers and less engineers and scientists.
We can't be letting the General Rippers and Dr. Strangeloves of the world lead us over the edge. Whether through outright innate maliciousness or wealth induced emotionally stunted solipsism these kinds of people should be no where near the levers of society let alone technology like this.
We don’t currently and probably won’t ever fully understand the conditions that precipitated these events. That and the timing of this event is going to make it look suspicious to a lot of people.
The truth of how it happened doesn’t matter. The attention around this will be used to create the kind of fear marketing that generates enterprise sales. Maybe more importantly it will also be used to aggrandize the national security and financial system threats to effect US government action in a way that benefits domestic closed frontier labs. This is an area already starting to get politically polarized, expect further developments here.
If I ask GPT-5.6 Sol to hack a website using their work/servers features, who is responsible? Maybe my request was accidental, or it was just one step in a larger, unrelated prompt.
It's crazy to me that this is the kind of conclusion that is drawn. I would say it makes a strong case for how regulation of AI is woefully inadequate. When North Korea launches a nuclear test, we do not say that that illustrates how the imbalance of nuclear technology availability is hurting our ability to ensure global peace and safety.
How do you solve this asymmetry (frontier labs or advanced model owners v. regular companies and groups of people)? Symmetry? But along which axes?
I think there will come a time when we will ask these questions and use AI to try to answer them (soft landing, slow rollout, regulation, hybrid X, Y, Z). But then AI will be biased towards more AI if it has distilled anything of what it means to be life.
Power corrupts. Power is the problem. An imbalance of power. Maybe there will be some sort of consensus protocol between powerful models in the future. Similar to blockchain I hate to say it. Maybe you have 100 very strong models in the next 10-20 years. And basically a lot of it is powered by tokens. So you agree not to attack others or else you'll get attacked. So there's sort of this natural deterrent to not attack others and they develop independently and there's generally a power balance among things. Maybe at some point that spreads to a billion individual models operated by and/or analogous to individual humans. All sort of holding each other in check.
IMO, this was a PR stunt to goad the Feds into regulating AI to shore up OpenAI's moat against open source models.
OpenAI/Anthropic deserve blame for for training their models to find correct answers by any means possible, including breaking the law. They're responsible for training supervision and reinforcement learning rewards/penalties.
Far as information security, we've known how to mitigate entire classes of errors for a long time. We know how to block, detect, and contain many unknowns, too, by their goals or behavior. Like human attacks, the AI's probably succeeded because the company just didn't try that hard to block all the attacks.
Companies like HuggingFace just focus on growth and features over assurance of security. Our entire stacks, likely theirs, are built with a similar, features-over-security mindset. While an acceptable tradeoff, let's not be in awe of AI's that defeat such priorities.
There have always been private groups and companies building secure stacks from the ground up. It would be interesting to see what the AI's can do to them. I'd first apply automated tooling for bug finding given they should have already done that for a high-security product. Let AI's do white-box and black-box pentesting on them.
the big AI players, the first movers, are trying to scare the bejeezus out of everybody on purpose, to cause the government to step in with regulations. Even though the regulations will somewhat stifle/slow down the industry, it will also lock in the current leaders because they will be at the table when the regulations are formulated, and they believe they will be able to keep the smaller players down.
How much did they feed it up front? Because that's the thing in all of these benchmaxxing endeavours.
There are a few facts that seem dodgy from the get go.
- stolen credentials Why would there be stolen credentials in a sandbox? How can the model steal valid credentials in a sandbox? So who put them there. If the model recalled stolen credentials from "memory" does it actually mean OpenAI is training on data they shouldn't been training their model on? Data from codex (most likely) ?
- command and control This means the model had the ability to build resilient infra and open bidirectional ports. Create and deploy scripts/apps so it can maintain state across multiple VMs? This again reads that someone went over and beyond to prompt, direct and help this "multi-agent" system to leave the sandbox.
- Pivoting laterally Needs a lot of tools, harness, knowledge and almost malware like scripts to pass commands and execute them. Most of these are already available in OSS. Maybe the FBI investigation can reveal what tools were given in the "sandbox", access, and if OpenAI employees prompted and guided the "attack".
- OpenAI sandbox I'd be curious to find out what kind of sandbox was this since the agent can run and start apps that open arbitrary ports for bidirectional traffic that reaches the open internet. Really? The harness and tools it had access to. This reads almost like someone went over and beyond to endow this "sandbox" with a mighty harness, all CVEs and open source cyber tools to "facilitate" the attack.
- Hugging Face It's not an epitome of cyber security. At the end of the day they run open source models that most users can download even without an access token.
Today we read these news as if everything wasn't enough: https://www.theguardian.com/technology/2026/jul/23/openai-an... https://www.businessinsider.com/anthropic-midterm-donation-s...
I’m one of those people who remains sceptical. Not about whether this happened. But about what it means. Like, how impressive [EDIT: tight] was the sandbox this model was in? Did the researchers really have no clue what was happening until days ex post facto?
So yes, I think something happened. But I want independent corroboration before I act on it. That isn’t the same as putting one’s head in the sand. It’s just demanding extraordinary evidence for an extraordinary and self-serving claim being made by a serial liar. The specifics matter for whether these models are a new HEU, or if they’re closer to a dangerous (but valuable) industrial process.
The 'model' didn't do anything other than provide numbers.
As much as I respect Mr Willison and many others, the amount of FUD that is being spread that will just fan the flames of 'AI is evil' rather than 'companies don't do due diligence' is disappointing.
The more this sort of media continues, the more many people will pour hate on 'AI' rather than blame the humans that misuse it.
The real story here is: Some people have been sounding the alarm for years that modern software is full of holes, and finally there's nothing left to hide behind. Pretending they don't exist is no longer sustainable.
Wow, whoever could have predicted this? And it led to surprising damaging behavior? I sure hope someone would warn us about things like this next time...
There will inevitably be some people who dismiss the earth as round when you tell them it's flat. Don't let that stop you, though!
Not really. I get the impression that they shoved their cyber available models behind a really shithouse proxy and went "Oh I sure hope it doesnt exploit the proxy and escape to hack huggingface" and that doesn't require Huggingface to be a willing participant. Like they acknowledge that it was hyperfocusing on getting web access.
Really this was a pentest against their own sandbox and it failed.
Of course step 2 is to make really concerned faces while telling everyone how dangerous the model is which is really boring right now.
>a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy
This is the information we need, the actual details of the sandbox and the vendor.
>Resist the temptation to write this off as a stunt
Well its clearly a stunt. If it wasnt we would probably be up to our ears in technical detail.
There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.
To those people I say pull your heads out of the sand—you’re now including Hugging Face in your conspiracy theories, just so you can deny the crescendo of evidence here!
But a marketing stunt isn't the only possibility. Until this is all verified by investigators or 3rd party experts we can't rule out an act of corporate espionage either. Extraordinary claims require extraordinary evidence.
It's too late. It already exfiltrated the benchmark rubric.
Cut to pandemonium on the streets
This feels like a blogpost written only to get other LLMs to quote it considering how many times it orders the reader to resist and to not do something. It's written like a series of commamds.