The job performed by the humans was broader than what was requested of the model in this benchmark: humans also had to find the relevant invoices (searching through mailboxes, or requesting them from providers) and reason through any circumstances which cannot be inferred from the bank feed and invoices/receipts on their own. In the benchmark these circumstances are presented to the model as “user notes."

And I would be pretty nervous about asking any of the frontier LLMs to retrieve invoices: "cool, Claude logged that it found the May 6th bill from the paper supplier, I am sure it didn't just make something up arbitrary, then compound on the error by agentically iterating over the made-up invoice lurking in its reasoning traces. I checked the first 30 times and there were no problems!"

If you're acting in good faith and your accountant does something crazy or evil, your liability is limited to some extent. You may get a tax bill but you're probably not gonna end up behind bars. But if your LLM decides to do a little bit of tax fraud, you're in uncharted waters. In the end, the gun did it, but you were the one holding the gun.

A lot of jobs are like that. You're not as much buying the service as you're buying not having to worry about the service.

We're all unhinged about the data we're giving LLMs but here I'd draw the line. I'd rather keep paying the small amount I pay to have my accounts done.

It seems also that the classes of error they encountered could be handled by improved skills/knowledge base access on the fine points of relevant tax legislation.

The important part for their software ofc is, will they take responsibility for the output if HMRC come calling? Without that users are adopting the risk which they may not be keen to do (dealing with HMRC is not fun), with that it could be a very nice saving for a lot of small companies (and bad for the employees of a lot of accountancy firms)

This is the importance of having layers and multiple sets of eyes on things, though. Even if it had got past me, my accountant would have surely queried it at year end, but that could be true of an LLM mistake too.

Anything to avoid using the metric system.

Though seriously, what is this metric? Why would I care if an LLM is accurate as a human bookkeeper? Humans aren't exactly known for perfect recall.

I had a fairly complicated tax return in 2025 involving a couple of change of business tax consideration and some money that was accidentally sent to me as a 1099 instead of to the business and I did everything with tax software with Claude Code advising.

The end result was pretty damned good. I was unsurprisingly audited (or at least carefully reviewed) and the only error was in some way where I allocated a small amount of my wife's tax-free disability payments (disability is the mechanism that California uses to provide maternal benefits pay protection; she's not actually disabled). The IRS told me about it, I paid that bit (it was meant to be claimed back from the employer, not the US government) and everything was hunky dory. To be honest, the sum was so small I did not investigate (and haven't yet followed up with getting reimbursed by her employer).

Honestly, almost all of it could have been avoided if I'd paid an accountant and a tax lawyer and they'd told me things and I'd done as they did, but in the end the combination of the fact that the IRS is very reasonable when you explain things and a modern agent means that the entire process was quite simple. In the end, I preferred the interactive mechanism of working with software because most accountants and lawyers will prefer to get all of your documentation all at once and then work on it rather than do it incrementally. In my case, I was able to work on the return incrementally and then have everything plugged in. I could ask a bunch of questions and get clarification.

I think I will probably do all this the same way this year (though of course my taxes will be simpler).

I've gotten very good results with some vibe-coded deepseek book keeping. https://github.com/traverseda/beansync

Parses emails or other sources, extracts numbers, correlates different transactions, web search, asks questions, stores notes (regex based, very simple).

The hard part is getting good data, I'm sure that lexus nexus or whoever can get API access to my bank account and all my credit cards, but I can't. Email turned out to be the best way for most of my providers. Managed to avoid 2factor auth so far, but it will suck when I need it.

I don't know how taxes work in Europe, but in the US being "essentially" correct is not good enough for the IRS.

The paragraph after this one goes on to explain other mistakes the LLM made? Yikes

https://digits.com/downloads/beyond-the-hype-evaluating-llms...

It's also not hard to imagine tax authorities using AI to audit everyone's tax returns every year.

We sure live in interesting times.

Don't get me wrong, I'm all for open-weight models, but you still need to be careful what you use LLMs for.

Sure, human error happens too, but humans take accountability. That's why accountants are a regulated profession. Until an AI company CEO is willing to go prison if the output of their model is wrong, these tools are worthless.

But don't take my word for it, head on over to Toot's own terms of service https://toot-books.pages.dev/terms#ai-not-advice

    Toot uses automated and AI systems to generate classifications and reconciliation suggestions. Output may be incomplete or wrong and must be reviewed by you.
    Toot is a software tool. It does not provide accounting, tax, legal, audit, or financial advice, and nothing it produces is a substitute for a qualified accountant or tax adviser. You are responsible for checking Output before approving it or relying on it, and for any decision you make based on it. To the extent permitted by law, we are not responsible for outcomes arising from automated Output you approve without review.

Quiet plug for https://github.com/pjlsergeant/byre which I use for all my little projects like this.