It generated an implementation that worked well, but I hated the ~480 lines of code. The structure and flow was just... weird. It was hard to follow and I was seriously bugged by it.

So I asked it to reimplement it with some simplifications I gave it. It dutifully executed, producing a result >600 lines long. The flow was simpler and easier to follow, but still seemed excessive for the task at hand.

So I rolled up my sleeves and started deleting code and making changes manually. A little bit later, I had it down to <230 lines with a flow that was extremely easy to read and understand.

So yeah, I can totally see many SWE-bench-passing PRs being functionally correct but still terrible code that I would not accept.

This is combined with the incredible general feeling that automatic programming can be evaluated as producing the same results regardless of the user using it. Something true only with benchmarks, basically. Benchmarks are useful metrics because even if weak we need some guidance, but the current real world dynamic is that AI will completely change what it is capable of doing based on the programmer using it.

Maybe never in the history of programming there was a time where diverse programming skills were as important as today (but this may change as AI evolves).

they are definitely useful but they miss the things that are hard to encode in tests, like spec/intent alignment, scope creep, adherence to codebase patterns, team preferences (risk tolerance, etc)

and those factors are really important. which means that test-evals should be relied upon more as weak/directional priors than as definitive measures of real-world usefulness

[1] https://voratiq.com/blog/test-evals-are-not-enough/

Interestingly, I had a similar finding where, on the 3 open-source repos I ran evals on, the models (5.1-codex-mini, 5.3-codex, 5.4) all had relatively similar test scores, but when looking at other metrics, such as code quality, or equivalence to the original PR the task was based on, they had massive differences. posted results here if anyone is curious https://www.stet.sh/leaderboard

This transformation will rule out confidence ranges with negative time.

BTW, log-normal distribution tend to produces events P(x>E(X)+d) more frequently than events P(x<E(X)-d). If one needs reasons why software projects often late, this is one of them.

Is this a post about AI archeology?

In my job in internal controls at a multinational company, I see this pattern. Teams choose an LLM for compliance tasks because it scored well on a general-purpose benchmark. Later, they find out it confidently creates regulatory citations that seem perfectly credible. The tests it "passes" have nothing to do with the important judgment calls.

Bisonbear's method of creating repo-specific evaluations is spot on. The only evaluation that truly matters is the one you conduct with your own data, using your own quality criteria. The real question is whether we can make that affordable enough so every team actually does it instead of relying on the leaderboard.

The simplest reasonable model would be logistic regression. It's also got 2 parameters and the range is correct.

As agent-generated PRs scale, "all tests green" becomes necessary but nowhere near sufficient. The merge gate is becoming the real bottleneck, and it needs different evaluation criteria than CI.

I've been thinking about tools for organizing long AI conversations.

Scrolling through hundreds of messages quickly becomes painful. I'm curious how people here manage long AI chats.

I've been thinking a lot about tools for organizing long AI conversations. Curious how people here currently manage them.

Also, some people would have spoken outright rejecting any AI code, but most maintainers would employ the silent treatment tactics. And then when you demand them to review, they either close it or say that "I'm too busy" as an argument. I would call this one of the biggest dick move, because it hurts the most yet you can't find anything wrong with them until they reveal their motives.

But hey, the tests pass!

If I force it to use plan mode for everything and babysit it, it can work really well, but it's really just acting as a faster typer for me, which is great. But it requires an experienced dev steering it.

For me the big takeaway is that passing doesn't automatically mean it is maintainable, follows established patterns / conventions or have unexpected side effects that real reviewers care about.