by redmalang
5 subcomments
- We have an internal proxy (that I've been meaning to open source for ages) that routes all llm usage at our company, which allows us to see data in realtime. Its been fascinating how rapidly Pi has been adopted. Moreover since its pretty hackable, we've been able to automatically aggregate context from pi sessions, which has resulted in Pi efficacy being higher as more people use it, putting in place a interesting virtuous loop.
I didn't expect this outcome: for whatever reason I assumed proprietary harnesses fine tuned to work with a companies' models would work better?
ps/random aside: there is something slightly off about Pi's edit command, we are planning to investigate this further and patch this as we have quite a few session traces now..
- I wish they'd do a follow-on post drilling into the impact of the programming language on cost-per-task, specifically looking at cost to complete tasks in mainstream strongly typed languages (eg. C#, TypeScript) vs dynamic languages (eg. Python, JavaScript). Does the additional verbosity of the language help or hurt cost per task?
- This was mostly because Sonnet 5 worked longer and read more to get there, consuming 1.9x more tokens.
I have experienced similar behavior between opus and haiku when benchmarking Dara engineering tasks. The “cheaper” model takes many more turns to figure out the task and this is without taking into account other important factors.
Another interesting behavior that I observed is that Haiku tended to cheat more maybe because it was having a harder time to find the root cause of the problem.
Benchmarking and evaluation of agentic systems is very interesting and if there’s one thing that someone should keep from the Databricks post is how important is for everyone to build and run their own.
by anentropic
0 subcomment
- > the results showed clear clustering of the models and harnesses into 3 capability tiers
pretty sure the only thing making that 'clear' is the coloured stripes, if you took that away it'd look like two tiers
good result for GLM 5.2 though
and Sonnet 5 seems like a waste of time
by HarHarVeryFunny
1 subcomments
- Wow!
It's great to see a large-scale real-world benchmark from a user of these tools, as opposed to the the benchmaxxed results from the vendors themselves. Also great to see different harnesses being tested, with considerably different results.
Definitely a few surprises here:
1) GLM 5.2 using Pi performs identically in terms of pass rate (~87.5%) to Opus 4.8 high using Claude Code, but significantly cheaper ($1.25 per task vs $2)
2) Absolute best pass rate (90%) was from Opus 4.8 x-high using Pi, beating out Opus 4.8 using Claude Code
3) Pareto frontier performance from any of the models (Opus 4.8, GPT 5.5, GLM 2.5) was using Pi rather than native harnesses
Apparently Pi used 3x less context than Claude Code, and one takeaway is to use Pi regardless of what model you are using. The other takeaway is that in real-world performance GLM 5.2 is the equal of Opus 4.8 unless you run Opus 4.8 on x-high in which case you can eke out a 2.5% increase in pass rate at the expense of doubling your cost over GLM 5.2
- Could it be that users of Pi are more senior and know better how to prompt and that's why the pass rate is higher?
by Schlagbohrer
0 subcomment
- Anthropic is not beating the charges that they inflate token consumption with their own harness given these findings that Pi is 2.2x more efficient at token management. Big "toothpaste ads tell you to use way too much toothpaste" energy.
- 1) Many models are now competitive at the top tier, including open source.
2) GLM 5.2 in particular was a major step forward in open source coding agent performance,
3) Harnesses make a huge difference in cost-performance.
4) Cheaper per-token does not imply cheaper per-task.
by yigitcan07
0 subcomment
- Would be great to see time spent per task per model. Especially since article references 390+ tokens per second for GLM5.2.
- It seems that pass rate decreases with effort increase, on GPT5.5? This is highly counter-intuitive and I don't see any explanation, any idea why they'd get this result?
by pianopatrick
0 subcomment
- Seems like for a hobby project $1 or $2 per task would add up a bit, depending on how many tasks you need to do. I mean it makes sense for a software company
- How is Pi so efficient? You'd think agent harness made by model makers would perform better.
- Doesnt this prove that there is really no moat in proprietary models for coding usecases, or the gaps is narrowing ? Also, since GLM5.2 can be run equally on amd and nvda, I guess there is no hw moat either. Further, switching costs for users is minimal not only in agents but models too. So there is really no stickiness or user preference involved. For this usecase I think it is a race to the bottom for costs in a good way for developers.
- The repo-scale angle is the useful part here. Small synthetic tasks miss a lot of the integration and context retrieval failures you only see in a codebase this large.
- I'd like to know what their config with pi is like. Is it vanilla, is it oh my pi etc.... This seems like important info.
- curious to see tests if a new king of efficiency in town : cursor grok 4.5 and their harness too. Quite impressed by pi.dev
Its shocking how cost per token does not correlate with cost per task, it's wild to see opus and glm nearby on $ per task axis
by throwa356262
1 subcomments
- Is there any technical analysis of why contex grows slower in Pi compared to codex and CC?
- > Databricks’ multi-million line codebase
The combined size of codebases for the underlying opensource products (Apache Spark etc) might be around 1M lines, I think. Why does the orchestration/management layer, that is "databricks", exceed the sizes of the core products?
- very interesting results!
GLM performed extremely well. we need GLM-6!
by felixlu2026
0 subcomment
- [dead]
by vegetablefinger
1 subcomments
- [flagged]
- > as we aggressively adopt AI for engineering
Why do we need to aggressively adopt things rather than thoughtfully adopt things?
It sounds like they are probably punching AI and engineers in the process
Hacker News