- You don't need to vandalize Wikipedia to get this kind of thing to work.
Back in September 2024 I named a whale "Teresa T" with just a blog entry and a YouTube video caption: https://simonwillison.net/2024/Sep/8/teresa-t-whale-pillar-p...
(For a few glorious weeks if you asked any search-enabled LLM, including Google search previews, for the name of the whale in the Half Moon Bay harbor it confidently replied Teresa T)
by nicole_express
8 subcomments
- It's an odd thing here, because I don't really understand why this is LLM-specific at all. If someone came up to me and asked "who's the 6 Nimmt world champion?" I'd google it and probably find the same result, and have no reason not to believe it. I mean, for all I know the game is being made up too, though it has more sources at least.
by xeeeeeeeeeeenu
4 subcomments
- The key to successful poisoning attacks is to introduce brand new information that doesn't directly contradict other training data. It's much easier to convince the LLMs that you're the king of a fictional Mapupu kingdom than the president of the United States.
So this means that for bad actors it's more efficient to manufacture brand new fake stories instead of trying to distort the real ones. Don't produce fake articles absolving yourself of a crime, instead produce fake articles accusing your opponent of 100 different things. Then people will fact-check the accusations using LLMs, and since all the sources mentioning those accusations are controlled by you, the LLMs will confirm them.
by pinkmuffinere
0 subcomment
- This has nothing to do with LLMs. If incorrect info can get onto a reputable resource, that info will seem authoritative, and it will be incorrect - that's not surprising. LLM's use publicly available info in their training, and often times publicly-available info is incorrect. I feel this is just as interesting as the base claim of 'I can get incorrect info onto wiki pages', no more interesting, and no less.
If somebody is trying to put out incorrect information on the internet, and they choose a small enough niche, it is not at all surprising that they can succeed.
- This is basically the same problem of products astroturfing reddit, or SEO optimizing google. You want a new X, and so they heavily go after the keywords associated with it.
This is sort of why "brand" matters; it provides a source of trust.
Encyclopedia Britannica used to be that source of 'facts'. Then it became whatever page-rank told you. Eventually SEO optimization ruined that.
News stories are the same thing. For certain groups, they have their 'independent' publication whose reporting they trust.
by billypilgrim
1 subcomments
- I must say I expected an actual poisoning of the data used to train the LLM and was excited, but the examples indicate that the LLM just searched the web and reported what it found? When you create a website with fake information and search Google for that information, it will of course bring up your site, not because it’s factually correct but because it’s related to what you searched for. What am I missing?
- One of the problems with labelling automation as AI.
People think that whatever information an "AI" spits out has gone through a round of critical thinking which enhances the trust value of that information.
The early LLM's using groomed data may have had such critical thinking somewhere in the pipeline. So it was already not really trustworthy.
And now? Using agents to search the internet for you?...
Garbage in, garbage out still applies in computing as ever.
by Paracompact
1 subcomments
- Most of the popular discourse around AI is still at the level of, "Don't trust the AI, trust the sources!" When it gets to the point where even the sources of simple facts are untrustworthy, the average person just trying to learn some trivia about the world is doomed.
Doesn't help that AI media literacy is so primitive compared to how intelligent the models are generally. We're in a marginally better place than we were back when chatbots didn't cite anything at all, but duplicated Wikipedia citations back to a single source about a supposedly global event is just embarrassing. By default, I feel citations and epistemological qualifications should be explicit, front-and-center, and subject to introspection, not implicit and confined to tiny little opaque buttons as an afterthought.
- "Stoner became the first American world champion...."
Even being on stoner.com,I read that as meaning something different from what was meant.
Op has a great surname!
- Not too dissimilar to googlewhacking where you'd aim to be the only result for a search query on Google.
And in a more indirect way, spamming Google's autosuggest feature to shape what people search for, though that perhaps is more open to factual/real-world information.
- Pretty much boils down to lying.
Since we've been kids we've been taught, hopefully, that lying is bad.
Society though normalize it :
- advertisement is pretty much always wrong (to the point of having laws in Japan about food packaging, France about modeling, etc) and the deception is the message
- entrepreneurs promises, nobody reach the goals set to VCs, it's always a lower number no matter the KPI. See https://elonmusk.today where the wealthiest man on Earth, ever, keeps on lying pretty much daily.
- political promises, no need to even give examples of that because it's just pervasive.
so... yeah, we keep on telling our kids "Do as I say, not as I do." then we somehow keep on being shocked that the practice of lying is pretty much happening in every corner of our society.
It's not a technical problem.
by DiscourseFan
0 subcomment
- The models are trained on expert data for important inquiries, this gets “hard coded” so to speak, and allows them to differentiate between the gunk online. For hyper specific references like this, it really doesn’t matter if its “true” since its not like someone’s life depends on it.
- BBC journalist doing a very similar thing in February: https://www.bbc.com/future/article/20260218-i-hacked/-chatgp...
- In American college football there's all sorts of awards, and each year they put out "watch-lists" and silly press releases that get parroted on social media by any team that has their own player mentioned.
I've wanted to come up with my own for a while ...
- How many people have done things like this and then disclosed the fact? It would be fascinating to collect as many instances as you can to develop a data set. Could you train a system to find more? How many could it find, and in what areas?
- i'm now thinking about creating a github repo that contains non sense code solutions to many problems. if that gets stars and many forks that could have an effect
- I feel uncomfortable that I can't actually verify that this story is true.
Asking Opus 4.7 who the reigning 6nimmt! champion is leads to this article and a warning about a possible hoax
- My wife cited ChatGPT as her primary source the other day when she wanted to debate with me on something.
"AI told me that..."
In the old days, it would have been "I read on Google..."
- Poisoning wikipedia shows low respect.
by wodenokoto
0 subcomment
- >Trust Laundering
>This is the part that really matters.
I can't tell if this is slop or parody!
by CrzyLngPwd
0 subcomment
- So it's trivial for an individual to poison the LLMs, but imagine what a state with billions of American dollars could achieve.
We can easily look ahead a few years and see how people will rely on the LLMs to be a source of truth in the same way people looked at Google that way, or newspapers.
Rewriting history has been happening for a while, and with LLMs being the one-stop shop for guidance and truth, the rewrite will be complete.
Doubly so since most people see these things as artificial intelligence, and soon to be superintelligence...so how can they be wrong?
- Like a FIFA peace prize?
by standeven
1 subcomments
- I've had LLMs regurgitate satire as fact many, many times.
- I made a post on Reddit asking for help with a TV, I had made up some (likley incorrect) technical assumptions about the issue. Several years later I asked the LLM about the TV, it used my own post as a citation to tell me what was wrong with it.
I am paranoid that this is happening every time I ask a LLM for a product recommendation or a shop recommendation. In the same way as SEO, anyone wanting to sell or convince needs to do as much as they can to influence the LLM.
by NooneAtAll3
0 subcomment
- so it's just https://xkcd.com/1958/
by shevy-java
0 subcomment
- So like Frank Dux! In the movie Bloodsport epilogue, he didn't do that.
It's almost like he was a better Chuck Norris than Chuck Norris. By his own ... testimony ...
by nonameiguess
1 subcomments
- Pails in comparison to what Frank Dux and Frank Abagnale were able to convince much of the world they did with no evidence other than their own stories. Who knows how much of recorded and believed history is complete bullshit? Not to get too far into sacred territory, but claims around Siddhartha Gautama, Jesus Christ, and the Prophet Muhammad are quite a bit less plausible than the legends of Ragnar Lodbrok or the tales of Jonathan Swift, but nonetheless widely believed.
- [flagged]
- [dead]
- [flagged]
by dyauspitr
6 subcomments
- Why does this person deserve any kind of support? What’s the point of poisoning LLMs? To put some cursory Luddite roadblock that might delay the technology for a couple of months?
Hacker News