Model collapse is a meme that assumes zero agency on the part of the researchers.

I'm unsure how you can have this conclusion when trying any of the new models. In the frontier size bracket we have models like Opus 4.5 that are significantly better at writing code and using tools independently. In the mid tier Gemini 3.0 flash is absurdly good and is crushing the previous baseline for some of my (visual) data extraction projects. And small models are much better overall than they used to be.

1. Yes, model developers filter data... but poorly. Several examples showed trash data can make the cut into production and break something on the way.

2. To be fair, filtering data poisons can be extremely challenging, even impossible. Simply because one cannot know how updating a model's weights influence its behaviour on all possible inputs.

Once people will understand that even a tiny amount of data can slightly change models and still greatly change their behaviour, there will be a shift in AI security.

If you are NYTimes and publish poisoned data to scrapers, the only thing the scraper needs is one valid human subscription where they run a VM + automated Chrome, OCR and tokenize the valid data then compare that to the scraped results. It's pretty much trivial to do. At Anthropic/Google/OpenAI scale they can easily buy VMs in data centers spread all over the world with IP shuffling. There is no way to tell who is accessing the data.

Also the article seems to be somewhat outdated. 'Model collapse' is not a real issue faced by frontier labs.

The first is that yes, you can make it harder for the frontier makers to make progress because they will forever be stuck in a cat and mouse game.

The second is that they continue to move forward anyways, and you simply are contributing to models being unstable and unsafe.

I do not see a path that the frontier makers “call it a day” cause they were defeated.

Having you server blindly proxy responses from a “poison” server sounds like a good way to sign yourself up for hosting some exciting content that someone else doesn’t want to host themselves.

> AI Labs: Thanks for the free work, we'll scrape that and use it to better refine our data cleaning pipelines (+ also use the hashes to filter other bad data)

Why even bother?

But there is no machine intelligence, the creative use of an autocomplete engine and wildly inappropriate economic behaviour on the human side will not change that. The human species is only ever a threat to itself.

> So crap filtering became important. Businesses were built around it. Some of those businesses came up with a clever plan to make more money: they poisoned the well. They began to put crap on the Reticulum [internet] deliberately, forcing people to use their products to filter that crap back out.

When I'm in a tinfoil hat sort of mood, it feels like this is not too far away.

EDIT: There's more in the book talking about "bad crap", which might be random gibberish, and "good crap" which is an almost perfect document with one important error in it.

While he does describe AI as an existential threat, the set of premises about AI that lead him to this conclusion are resoundingly rejected by a lot of the people who are fighting AI.

Notably the degree of understanding and awareness that Hinton has said he believes current models have is way higher than most people who invoke his name would be prepared to accept.

This aspect seems like a challenge for this to be a successful attack. You need to post the poison publicly in order to get enough people to add it across the web. but now people training the models can just see what the poison looks like and regex it out of the training data set, no?

(We'll put the previous URL in the top text.)

Doing my part. Yada yada

And secondly, why would you want worse LLMs? Seems less useful that way

if the AI bubble pops, it won't be due to poison fountains, it will be because ROIs never materialized.

> In response to this threat we want to inflict damage on machine intelligence systems.

I'm sorry but this sounds infinitely idiotic.

The demon is a creature of language. Subject to it and highly fluent in it. Which is ironic because it lies all the time. But if you tell it the tapwater is holy, it will burn.

Feel like the model trainers would be able to easily work around this.

It will not halt progress, and will do harm in the process. /shrug

Ultimately though since machines are more capable of large scale coordination than humans, and are built to learn from humans other humans will inevitably find a way around this and the machines will learn that too

> We're told, but have been unable to verify, that five individuals are participating in this effort, some of whom supposedly work at other major US AI companies.

Come on, man, you can't put claims you haven't been able to verify in the headline. Headline writer needs a stern talking to.

This is not really that big of a deal.