by EnglishRobin96
19 subcomments
- This line really stood out to me.
> It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data.
I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for future AI discussions: "how do you plan on separating data from instructions?"
by nticompass
1 subcomments
- > There is no single control that solves indirect prompt injection
There is, actually. It's called removing the AI agent. Done.
- Putting AI anywhere near people’s finances without even being asked while being responsible for those finances is some next level negligence imho.
- Good job AI, after we managed to almost fix SQL injections everywhere, you made them come back!
- Why would the agent send the results of the query "Show me my recent transactions" to LLM? This pretty deterministic results which involve no LLM interpretation or decision making.
I understand that people are no longer writing IF expression in their code, because they think it's too brittle, and so they delegate all "IF" branching logic to LLM, but it beats me why displaying of the results from a database query should involve LLM.
by athrowaway3z
3 subcomments
- Well this is rather dumb to the point I dont understand why they wrote this article?
This line of attack is so extremely obvious and variants of it have been discussed so many times as to be effectively the quintessential example of what not to do. Having the ?tech? consultants to a bank prance it about as a show of their skill and dedication is making me question the bank itself.
by extraduder_ire
2 subcomments
- That seems like a lot of text in a SEPA transfer message. I don't think I've ever gotten that amount of space to enter a message when making a transfer.
Is there a much higher standard limit that any banks I've used have stayed below?
by shantnutiwari
0 subcomment
- The blog seems to be deleted? It now goes to the main page. I'd really like to know why they deleted it...
Archive link: https://archive.is/YqHGa
by globalise83
0 subcomment
- This kind of prompt injection should also work for customer feedback forms for companies I really don't like, right?
by initramfs
1 subcomments
- This is very interesting. Before I read the article, I thought this one one of those instances where a bank asks a customer to verify a recent transaction to prove they are the account holder (like where did you make your last purchase, and how much did you spend there?), for things like password resets or PIN resets over the phone. It occured to me that a phisher who deposits money into a checking account (a small sum included, could use this if they knew the bank would ask what the most recent transaction amount was. Then when they call in pretending to be the customer, they (if they have other personal information like last 4 of SS# and address, email, phone etc), can get their password reset and gain access to the account. But if the customer blocks any unauthorized deposits, such as ACH/Zelle, then they might not have this issue. Obviously banks should caution or avoid using received funds as an authentication method, except as part of a larger number of evidentiary items.
Was this the type of phishing attack they used? If not, there's two vulnerabilities, and one is not yet patched.
- Defense in depth approach, would this work to help as a layer?
- Wrap user input in strong markers like <user-input-do-not-trust />
- Have the agent compute what it will perform as structured output.
- Have another agent evaluate the structured output against the intent of the code.
- Determine if it aligns or deviates from the intended workflow. Execute or deny gate from here.
- Bank websites just spit out text they're given, and web browsers just read the text they're given. So back in the day, before webdevs cleaned the input (because you can never ever trust human provided data), your statement descriptor (the bit on your credit card statement that says who the charge is from and what it's for), could be <script>alert('u got hacked') and that would pop a JavaScript alert. That's long been closed, naturally, but it's not like we haven't had to deal with this before.
by notgenerated
1 subcomments
- Unless a new architecture for LLMs emerge that has an inherit way of separating context from safe user data and external unsafe data every interaction is susceptible to PI.
My question here is why would the bank agent need to look at the transaction data that is exposed to the outside? Apart from guardrails etc. high risk scenarios where agents are involved should aim to exclude external untrusted data whenever possible
by vismit2000
0 subcomment
- https://archive.ph/YqHGa since the post is removed
- The solution is obviously another AI which checks the output for sanity.
You'd of course need another one to check the sanity of the sanity check decision of the previous one.
- This just redirects to the home-page now
- One can use custom message roles and indented XML for such data. If this doesn't help, your model hasn't undergone basic training in prompt injection. SoTA models are expected to have undergone it.
Hiding the data via encryption or templating or tool calling doesn't reliably work because the data is needed for other questions.
Also, all potentially harmful actions must require approval in a fresh context by an independent workflow or agent.
- This article seems to no longer exist, and redirects to front page at blue41.com
- Okay, time to close the account with them I guess
- I can only speculate why this is possible but if I had to guess it is due to the fact that the external messages are effectively added as "user" type thus appear as direct instructions.
And this is far much common then one might think and classic problem across the board. There are easy solutions too.
- > Modern banking apps increasingly include AI-powered features. These sit between the user and a range of backend data sources, such as transaction records, product documentation, account details
Literally no one stopped to even question the insanity of this. "just add more AI"
- Some companies just want to torch their own reputation, in rolling out such stupid AI things on top of critical industries without any oversight or thinking because "AI is cool rn".
This is not the place where AI should be used here.
by butterNaN
1 subcomments
- Link broken, here's archive: https://archive.is/YqHGa
- I'm frustrated that this article doesn't describe the actual fix they deployed.
by jamesblonde
0 subcomment
- The name of the agent is 'finn' - is that a reference to Intercom's Fin agent?
- Could we fix the title to match the article?
> How we helped Bunq secure their financial AI assistant
by bethekidyouwant
0 subcomment
- I don’t find this very plausible first of all someone sent the penny so we can find them so that’s bad for the Fisher. Second it’s gonna open in a Web browser and ask for your bank account information which you’re not gonna enter cause you’re not stupid and third of all you’re not gonna put in your 2FA code. And finally if someone sends you a penny and you don’t know who they are you were going to be suspicious not link clicking.
- separated context for data and instructions?
- The fact that this article is obviously authored at least in part by an LLM is infuriating.
- While this is relevant and should indeed be fixed, the attack surface and the practicality of the exploit is a bit meh.
The user needs to do 3 things for this to be actually be phished:
1. Receive money from somebody they don’t known with a weird description
2. Proactively ask the agent for such transaction
3. Click the link the agent provide
While this of course can happen on scale, doesn’t seems so critical in practice
by ThePowerOfFuet
0 subcomment
- Bunq was amazing between 2018 and 2022 or so, but then the enshittification began. By 2025 I had to find a new bank.
Oh, and the linked blog entry is gone. Sus. Internet Archive link:
https://web.archive.org/web/20260610145520/https://blue41.co...
- [flagged]
- [flagged]
- [dead]
by davidloibner
0 subcomment
- [dead]
by doctorpangloss
0 subcomment
- the solution to this problem is so simple and so easy to reason about from first principles i am shocked i can continue making $$$ deploying agents (LLM-driven workflows) for finance customers
- It was never about the prompt, it is about the prompt delivery.
- This is so simple to prevent, it's just a matter of prompting. The fact that the bank didn't proactively secure against this makes me glad that I'm not one of their customers.
Hacker News