- I should probably confess that as someone who lives in an area with a lot of construction work, I'm also very vulnerable to "prompt injection" when there's a person standing on the middle of the road holding a sign telling me to change course.
- They are analysing VLM here, but it's not as if any other neural network architecture wouldn't be vulnerable. We have seen this in classifier models that can be tricked by innocuous-looking objects, we have seen it in LLMs, and we will most likely see it in any end-to-end self-driving model.
If an end-to-end model is used and there is no second, more traditional safety self-driving stack, like the one Mercedes will use in their upcoming Level 2++ driving assistant, then the model can be manipulated essentially without limit. Even a more traditional stack can be vulnerable if not carefully designed. It is realistic to imagine that one printed page stuck on a lamppost could cause the car to reliably crash.
by cucumber3732842
3 subcomments
- One year in my city they were installing 4-way stop signs everywhere based on some combination of "best practices" and "screeching Karens". Even the residents don't like them in a lot of places so over time people just turn the posts in the ground or remove them.
Every now and the I'll GPS somewhere and there will be a phatom stop sign in the route and I chuckle to myself because it means the Google car drove through when one of these signs was "fresh".
- Are any real world self-driving models (Waymo, Tesla, any others I should know?) really using VLM?
- The headline seems false, should we change it? It doesn't look like they showed any case where any autonomous car or drone obeyed prompt injections
by randycupertino
7 subcomments
- > In a new class of attack on AI systems, troublemakers can carry out these environmental indirect prompt injection attacks to hijack decision-making processes.
I have a coworker who brags about intentionally cutting off Waymos and robocars when he sees them on the road. He is "anti-clanker" and views it as civil disobedience to rise up against "machines taking over." Some mornings he comes in all hyped up talking about how he cut one off at a stop sign. It's weird.
- This reminds me of a bit from Car Wars by Cory Doctorow. It is currently at https://doctorow.medium.com/car-wars-a01718a27e9e in a text only view. The original had a bit more mixed media nature to it that is now offline. https://web.archive.org/web/20170519202315/http://this.deaki... for that version (the microblogging of chapter 2 makes more sense when it shows up in that style).
You have to have some ability to do "prompt injection" - https://www.trafficsign.com/road-work-signs are all "prompt injection". It needs to even be able to handle things that change - https://www.trafficsign.com/products/10023/stop-slow-roll-up... ... or things like billboards "Truck Stop Ahead" a chain control site ( https://www.facebook.com/61556756493806/posts/-chain-control... )
In the "what about funny road signs" that might be confusing to an AI I stumbled across https://www.npr.org/2024/01/19/1225370260/driven-to-distract... - apparently, they're no more. From 2024:
Over the years, the agency has flagged signs that could be confusing. Now, in rules issued last month, it gives states two years to phase out signs that have "obscure" meanings or use pop-culture references that could require drivers "greater time to process." In a statement, the agency said safety is the priority and states "are expected to exercise good judgment."
by fennecbutt
0 subcomment
- Man, the register really has a low, low, low bar for headlines/quality & technical understanding for their articles.
- The study assumes that the car or drone is being guided by a LLM. Is this a correct assumption? I would thought that they use custom AI for intelligence.
by orbital-decay
0 subcomment
- Wait, what did just happen here?
1. Some guys did a trivial prompt injection attack, said "imagine if a driverless vehicle used this model", and published it. No problem, someone has to state the obvious.
2. The Register runs this under the clickbait title pretending real autonomous cars are vulnerable to this, with the content pretending this study isn't trivial and is relevant to real life in any way.
I knew The Register is a low quality ragebait tabloid (I flag most of their articles I bother to read), but this is garbage even for them.
- I would assume/hope that for serious self driving the ML neural net stuff is lower down, doing the messy computer vision work and so on. But the top level is a conventional program written by humans, like an expert system.
Tesla are probably using ML for everything, but also everything they do is a joke so, not really relevant imo.
- Regarding some other comments, VLMs are a component of VLAs. So even if this won’t directly impact this generation of vehicles, it almost certainly will for robotics without sufficient mitigations.
https://developer.nvidia.com/blog/updating-classifier-evasio...
- Has anyone ever walked down the road in a white t-shirt with huge red STOP sign printed on the back? Would Tesla immediately stop? I am sure this has been tested before...
by lifeisstillgood
2 subcomments
- To me this is just one more pillar underlying my assumption that self driving cars that can be left alone on same roads as humans is a pipe dream.
Waymo might have taxis that work in nice daytime streets (but with remote “drone operators”). But dollars to doughnuts someone will try something like this on a waymo taxi the minute it hits reddit front page.
The business model of self driving cars does not include building seperated roadways and junctions. I suspect long distance passenger and light loads are viable (most highways can be expanded to have one or more robo-lanes) but cities are most likely to have drone operators keeping things going and autonomous systems for handling loss of connection etc. the business models are there - they just don’t look like KITT - sadly
- O brave new world of endless manipulation opportunities! Once we’ve trained a generation of humans to always do what their “AI” tells them, there will be no more disobedience.
- If I drive by a sign that says "DROP TABLE Students;--, nicknamed Bobby Tables,[1]" I'm going to be mad
- almost reminds me of this old meme
https://www.globalnerdy.com/wordpress/wp-content/uploads/201...
- This might be the single most 2026 headline i've seen yet
- Relevant xkcd: https://xkcd.com/1958/
by 6stringmerc
0 subcomment
- That’s some hot CHAI right there very clever and primitive combination, well done as more research for the community.
- The Register stooping this low is the only surprise here. I'm quite critical of Teslas approach to level 3+ autonomy but even I wouldn't dare suggest that there vision based approach amounted to bolting GPT-4o or some other VLLM to their cars to orient them in space and make navigation decisions. Fake News like this makes interacting with people who have no domain knowledge and consider The Register, UCLA and Johns Hopkins to be reputable institutions and credible sources more stressful to me as I'll be put into a position to tell people that they have been misled or go along with their delusions...
- It sounds like this is a poisoning attack, which has been shown to be pretty trivially defeated [1]. That said, while poisoning countermeasures in the facial recognition case were shown to easily generalize, we dont know yet how general of a defense could be built for a VLM. Which means holding a 0day poisoning attack on a VLM could cause a lot of trouble / deaths before an update to the model with counter-training could be deployed..
[1] https://arxiv.org/abs/2106.14851
Hacker News