Worse, they don't control for breaking the application's tests.

> Pass rate scores the agent’s final state against the hidden tests we wrote for each task. We do not check whether the agent broke unrelated tests already present in the repository, and a cleaner-side and messier-side solution that both pass the hidden test may still differ on tests they were not graded on.

Any conclusions with respect to token consumption seems pretty meaningless if we're not controlling for the quality of the final output.

    Refactor the Python code to make it more Pythonic, e.g. fewer classes/singletons, especially if it will provide a speedup. The Python code **MUST** follow code organization standards expected of popular open-source Python packages code without causing any benchmark performance regressions.

    The Rust codebase in `/src` has become bloated with several files >1k LoC. Refactor the Rust codebase to fit code organization standards expected of popular open-source Rust code without causing any benchmark performance regressions.

In terms of benchmark performance it generally improves after the refactor which I suspect is coincidental (especially in Rust where it shouldn't matter due to compiling) but I'm not complaining.

This feels like a terrible approach, sufficient to condemn the entire study.

Apparently half of the "minimal pairs" in this work were constructed in this way. I simply am not going to trust any conclusion that requires assuming these AI "cleaned" repos are in any way representative of actually-good codebases.

The minimal-pair design is honestly one of the strengths, as it tries to isolate the cleanliness from other factors (e.g. architecture, dependencies, tests) which is more rigorous than comparing repos. However, using LLM-generated "sloppified" code (rather than some kind of mechanical or human-guided) is a bit questionable.

I'd say the biggest critique which others have correctly mentioned here is the authors' choice to not check the full test suite. The claim of "behavioral equivalence" is only as good as the tests and coverage.

This hypothesis is compelling for two reasons: 1) it makes sense (garbage in -> garbage out as it were) since LLMs will mimic what they see in the codebase; 2) it matches what many engineers feel somewhat intuitively over the last year or two of using these models. Greenfield is almost always easier than joining a busy codebase. The mess comes in complex integration, and maintaining a system for legacy purposes, etc.

At work we had largely moved off of some old hardware but kept partial support for it in the code because normally that would be zero cost. But the constant overhead of bullying, bribing, and threatening the LLMs to leave that code alone was way more work than just excising it and reimplementing it if needed.

You can reduce a lot of the token use for traversal by giving your agent access to some form of LSP in addition to hierarchical direction with your AGENTS.md (or equivalent) for monorepos - but a spread-out codebase is always going to end up requiring some form of traversal to solve each task.

And that traversal isn't just token use - its repeated round trip latency (LLM (queue time -> prefill -> decode -> output) -> Agent (parsing -> tool call -> tool response) -> back to LLM) for EACH step (well, some can be done in parallel, but in practice its mostly sequential) - slowing down the task considerably.

Locality and structure are key when it comes to efficient use of agents. The context window is always bounded and attention across it is inconsistent.

All of these things have historically been the job of engineers, because it helps other people contribute to the code.

Now it helps other people and other agents contribute to the code.

1. clean structure means the AI can predictably work because i want deterministic output.

2. basic things like following SRP leads naturally to layered code which makes testing new features and fixing bugs super simple.

3. reviewing code is simpler, because i see ai agents as "servants" i command to do something and thus an overseer is still required

I'm not sure if you would call it an objective or subjective measure, a fixed model would be consistent which would provide an objective base for comparison, but other models would be different, so it would be subjective in relation to the model itself.

https://github.com/bjcoombs/ai-native-toolkit/tree/main#what...

In AI labs, they can run a task repeatedly for extended periods, eventually marking it as completed in benchmarks. But in the real world, completing tasks quickly and well remains important and valuable.

With coding agents, agents can produce code quicker. The same trade-off still applies.. but, the time it takes an LLM coding agent to write well organised code is still going to be quicker than the time it takes me to write scrappy code.

Have you even seen one ever?

Also, not checking if it breaks unrelated tests is wild, good software is written by modifying the "just right" amount of code to get your result.

If you break _unrelated_ tests than you most likely didn't changed the right amount of code.

Idk I call bs.