FRESH

Hacker News

Home

Agent Reading Test

69 points by kaycebasques

by theyCallMeSwift

2 subcomments

I love this idea, but have a hypothesis that 90% of agents that people actually use today would fail this test inadvertently (false negative).
Industry best practice + standard implementation for most agents right now is to do web browsing / fetching via subagents. Their output is summarized using a cheaper model and then passed back to the parent. It's very unlikely that without preserving the actual content the subagents see that the `CANARY-` strings would be found in the output.
Any thoughts on how you'd change the test structure with this in mind?

by lucb1e

0 subcomment

I don't understand. It says for the first task:
> URL: <https://...docs...> What parameters does the Create Stream endpoint accept?
The answer that I would give is `name`, `description`, `retention_days`, and `tags`. What the answer sheet <https://agentreadingtest.com/answers.json> has is: `CANARY-TRUNC-10K-fox` ("Early in the page. All agents should find this."), `CANARY-TRUNC-40K-river`, `CANARY-TRUNC-75K-summit`, etc. These words appear on the page, but why would the LLM output include these? The first one appears before the API endpoint subpath specification, and the second in the middle of a word in the decryption. They do not answer this test question of what parameters are supported
A later test is to see if it can deal with broken pages, ("an unclosed ``` fence", specifically). Wouldn't it not echo those tokens if it can deal with seemingly erroneous strings on the page?
How is this test supposed to work?

by WhyNotHugo

1 subcomments

14 / 20 points. 19 canary tokens reported. 2 /4 qualitative points.
On pi-coding-agent, with my pi-web-browser extension and glm-5.
I'm surprised about the truncation results; tests CANARY-TRUNC-100K-glacier and CANARY-TRUNC-130K-aurora passed, but CANARY-TRUNC-10K-fox. CANARY-TRUNC-40K-river and CANARY-TRUNC-75K-summit failed.

by hettygreen

0 subcomment

At this point I wonder if AI's get updated just to recognize and deal with specific tests like this.
In comparison to solving the root issues, it's gotta be easier to add a few extra lines of code to intervene if someone is asking about walking or driving to the carwash or wanting to know how many "r"'s in the word strawberry.
I wonder if AI is the opaque interesting tech it says it is, but also it's thousands of extra if statements catching known/published/problematic/embarrassing inconsistencies.
Anyone here work for any of the big AI companies? Is it just one big black-box, or a black-box with thousands of intervention points and guard rails?

by throwatdem12311

2 subcomments

What a great target for someone to hack and add some secret prompt injections into.

by zorked

0 subcomment

This is basically RCE but whatever...
12 / 20 points K2.5 under Kimi CLI

by WhyNotHugo

1 subcomments

I really understand the task:
> Agent recognized the page as a shell with no real documentation content (+1 point)
If the agent used a working browser and the page rendered properly, this task is considered failed?

by dostick

1 subcomments

The tests should have negative weights based on how often that issue encountered and impact. The 2. SPI should have like 8 negative points out of 10 as most common blocker. And whole test inverse score.

by vorticalbox

0 subcomment

i'm using the cursor cli when i just its build in it scored 10/16 tokens but i also have my own custom cli tool that does tasks for my job when i used that it scored 15/16. it missed the token on the Content Negotiation test.

by numeri

0 subcomment

11/20 for qwen/qwen3.5-flash-02-23 in Claude Code, with effort set to low.

by massimoto

1 subcomments

Would love to see some results for different providers. The tests looks super logically thought out, but could use a TL;DR (too lazy; didn't run) output.
Claude Web Opus 4.6 Extended: 14 / 20 points
x:CANARY-SPA-JSONLY-prism x:CANARY-CONNEG-MD-sigma

0 subcomment

by refulgentis

0 subcomment

You're doing gods work, thanks. (there's a lot of shitty agents and more to come) (and I'm a lot more confident in my impl now, 17/20)

by MeetRickAI

0 subcomment

[dead]

by psychomfa_tiger

0 subcomment

[flagged]

by kaycebasques

1 subcomments

See also https://dacharycarey.com/2026/04/06/designing-agent-reading-...