by robbomacrae
1 subcomments
- SCB is definitely an underrated benchmark. For me the unique selling point is that it more closely mirrors software development by not stopping after a single task. The agent has to keep code clean. The only disadvantage is all the problems are greenfield and not git inited so the agents don’t make use of git diffs.
I’ve used SCB as part of my assessment of agent skills (superpowers, GSD etc) https://orcabot.com/labs/do-skills-improve-coding-agent-accu...
There is a small but growing community on discord for discussing SCB so if interested please join https://discord.gg/BrC4BA9sVj
by WilcoKruijer
4 subcomments
- I hope the big labs will start using this benchmark in their RL pipelines. Reducing complexity in generated code should be the number 1 priority, in my opinion. The holy grail for me is models implementing features while reducing LoC (i.e., choosing the right abstractions).
What is also nice about this benchmark is that it can be used to iterate on prompts/skills for reducing code complexity.
- Nice! I actually ran across this paper+benchmark recently, too. It's the first I've found that start to aim at some of the non-functional and longitudinal requirements that I think have always been an important part of writing production code.
It's especially relevant now that models are good enough to solve ~most point-in-time problems.
Some relevant but disconnected thoughts:
- deterministic scores are so nice
- what "maintainable" is is probably some high dimensional space described by these signals; it'd probably require some human labeling to figure out where this space is
- another signal I've been thinking about and I'm seeing increasingly get brought up is the state space of a system; I'm seeing formal methods pop up a lot recently
- The author of the original paper did not include error bars on the cost and quality results, which is a bit alarming. They did include a +/-, but did not list explicitly if that means standard deviation, range, or 95% CI, etc. Another table is listed as that being one standard deviation. If we take those results to be a standard deviation as well, I think we can conclude that there is no statistical difference between the outcomes of all models in terms of quality and cost, as the error bars are all overlapping.
by sothatsit
4 subcomments
- This matches my experience of Opus 5 being a nice improvement over Opus 4.8, but not being revolutionary like Fable felt.
I’ve now replaced my use of Opus 4.8 xhigh with Opus 5 medium, and I’m using less tokens and it’s quicker. I can understand people being annoyed by its writing style but for getting work done that really doesn’t bother me. I’ve been really enjoying using it.
- I always have Claude recite a pledge before starting coding to fix redundant code it notices over time. It does seem to find redundancies, but only when I point out bugs, that's when it goes into fixing mode and actually applies my Don't Repeat Yourself preference from the CLAUDE.md.
The original paper cited by this post does try to see if improved prompting will make a big difference in the end using a `plan_first` prompt variant, but find no influence on pass rate at the end of the benchmark. The `plan_first` seems to assume coding agents will just refactor once they finish features, but I don't think they tend to refactor significantly unless they are told to fix bugs rather than build features. The benchmark leaves tests hidden with no fail-to-pass feedback, so that may be why degradation is monotonic.
by mellosouls
0 subcomment
- This is nice, but would be really useful measured against human performance, and yes - I understand that is likely a difficult challenge.
Many people though are going to read the headline figures and think it means - say - Opus 5 is only a quarter the strength of a human coder.
by Johnny_Bonk
2 subcomments
- I haven't joined your chats in a while but glad to see you put this together, I truly feel as though opus 5 is not much of an improvement. The only time i ever felt a wow factor was opus 4, 4.6 and fable pre trump admin lobotimizing
- I'd be curious to see the raw test results.
I have a suspicion that most models will miss the `database_migration` Checkpoint 2 test that includes a `default_value` because it could be interpreted as either a JSON-literal or a SQL-expression.
There might be other tests as well that are prone to failure for reasons other than the reasons cited in the paper.
I think a cool experiment would be to adjust the order of the features implemented (e.g. checkpoint 3 then 2 then 5 then 4) where dependencies allow it. Then one could account for some checkpoints being more difficult than others.
- Great writeup. The excessive function thing has always driven me crazy; I guard against this explicitly in Claude.md.
I have found that models are generally poor at managing refactors / complexity while also implementing new features. But I’ve had some success with a semi-lights-off approach where you decompose it and prompt the model adversarially in a second pass to look for new rough edges and areas of complexity or refactors that might simplify the codebase.
So I’d be very curious to see this benchmark but with something like a periodic “refactor turn” interleaved in.
Also eager to see Fable benchmarked; anecdotally that was the only model whose code I felt I could actually trust to not review closely.
by anentropic
1 subcomments
- So far my 'solution' to this has been periodically run a separate round of whole-codebase code review (preferably Fable) and then rounds of refactoring off the results of that
by killingtime74
1 subcomments
- Did you not benchmark latest GPT 5.6 or GLM 5.1/Kimi K3 because of cost? I can run them if you share how you ran them
by swiftcoder
0 subcomment
- I wonder to what extent we could steer performance on this benchmark, by providing an adversary model that penalises code duplication and overall lines of code?
- To what degree is this a harness/system prompt problem? Models maybe should implement new stuff with as little impact on the existing stuff as possible by default? A simple system prompt for it to always check the code after task completion for proper simplifications, abstractions and cleanups before returning to the user? Instructions to retain "story like" readability of the code.
by nikhilsimha
0 subcomment
- the last note about the agent sending emails without approvals is a very common problem!
by spicyusername
0 subcomment
- It'd be interesting to compare this to human teams, given the same challenges.
- Really hoping that all of the attention you're bringing to the longitudinal sloppification of codebases makes it back to the labs and creates some pressure to improve that trait of the models. This new benchmark seems promising.
At the same time, I imagine it will be hard for them to prioritize this over improving flashy one-shots of impressive zero-to-one feats that demo so well and attract more customers.
Can we have simonw make "pelican on a bicycle after 1000 requests for iteration" popular?
by vintermann
1 subcomments
- This benchmark makes me worry a bit that people will just ask their model to reimplement everything from scratch once their requirements become more clear.
by willsmith72
1 subcomments
- > The big headline is that Opus 5 got a 24% on the small subset of the benchmark that I ran - not much higher than Opus 4.6's 17% strict pass rate in the original paper.
a 41% improvement is not much higher? come on that's just doomer
- Opus models were straightforward to review.
But Opus 5 is very different.
by subarctic
2 subcomments
- I still don't have access to Opus 5. I'm on the latest version from Homebrew, I guess the update hasn't made it there yet
by cesarvarela
1 subcomments
- Please add Fable; a good benchmark should show that Fable is less prone to just autocomplete and instead pushes back or is at least more tasteful.
by adamtaylor_13
4 subcomments
- I may be crucified for asking this but: is there any proof that slop matters beyond our sensibilities as developers?
If the code is ugly, but defects are low—does it matter?
If the code is hard to read, but clients are happy—should I care?
Genuinely asking. Weird times we live in.
- Good benchmark. Although, why did they have to give it such a unprofessional name?
- finally the benchmark for me
- For most of these benchmarks, I feel like a p50 and p95 (using SWE salary as a proxy?) human benchmark as reference would be interesting.
Edit: FWIW the paper the post quoted has repositories as slop baseline https://arxiv.org/html/2603.24755v1#S4.SS2
by knighthacker
0 subcomment
- This is where Opus 5 shines
- Can we use those metrics in a review pass for every PR? So review agent will have to pinpoint new complexity and propose refactoring or justify increase?
- > [...] with Opus 5 writing five times the number of functions/callables than Opus 4.8 over the course of the same set of challenges.
Is this bad? I have McCabe complexity switched on in Ruff and find it a handy watermark for when something should be broken up into smaller, individually testable callables. Five times as many callables could make for much more readable and testable code.
- Opus 5 is an overconfident stupid model. It tries to generate too much slop, tries to act like everything will fall. I have reversed back to fable and codex sol.
- So many benchmarks more the models themselves.. just make one unified standard to benchmark all or stop calling it “benchmarking” as this word lost its meaning.
by Sattyamjjain
0 subcomment
- [flagged]
- [flagged]
- [flagged]
by mengram-ai
0 subcomment
- [flagged]
- Mostly a harness problem in my experience. Slop accumulates when the agent can touch anything, so constraining it to one seam and having it add alongside rather than edit in place does more than model choice.
Hacker News