Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?
by sreekanth850
1 subcomments
- Anthropic is losing bigtime against Open AI in coding space. I was using Claude code upto last march. Not an enterprise customer, but a responsible AI user where i don't over spend and use basic plan to manage repository with 400Kloc all together. We sell to local government and a team of 3. Claude code was super slow, never able to fix issues properly.(Despite with proper test cases, observability, documentation and layered architecture). After moving to codex, life has been much easy and free form usage anxiety. Now managing entire things with 2 codex plus account per team member. Its high time anthropic should stop scare mongering and build efficient models. Everyone doesn't need Fable. People need models that solve problems efficiently.
- Great eval! If you are comparing search strategies, ultra mode is likely superior. Would love to see a follow-up evaluating that.
Ultra can fan out parallel investigators, run adversarial review at defined checkpoints, and do a bunch of other smart stuff to avoid getting stuck in a local optimum.
Generally as the OP notes, /goal works better for single-track investigations or small scale scatter/gather.
- The chart at the top is somewhat confusing. It says, “lower is better” but the y-axis is inverted! So visually higher in the chart is better but lower in terms of # value.
- Claude seems to forget what you tell it in very long work sessions (things that take weeks to develop), no matter how many times you tell it which part is extra important. I dont use goal (I guess I should), but presumably it makes it actually remember the most important instruction. I believe this here is about shorter sessions where the issue doesn't crop up as much.
- GPT should be better at these optimization problems given that they won the recent atcoder heuristics competition against top humans. And Anthropic is less focused on these types of things.
by enraged_camel
1 subcomments
- I've used GPT 5.6 Sol Xhigh extensively since its launch, alongside Fable 5.
My impression is that it is about as intelligent as 5.5, but they dialed up the relentlessness meter to eleven. This makes it more likely that it will accomplish the task you give it, which I think is the primary reason it looks competitive in benchmarks. However, it also makes it more likely that it will resort to... unconventional, weird or outright unsafe methods to do it. So I have to watch it like a hawk.
The other day it tried to read env variables from prod using a CLI command. The task it was working on did not necessitate doing that even remotely. I have the SSH keys for that particular CLI tool tied to my 1Password. So when the agent failed (because I never authenticated the SSH key access), it wanted to take over the computer, for which I got an OS prompt. At that point I stopped the agent and asked it why it did that. It said it wanted to dig around 1Password itself to see if it could get the key. I asked it why it needed prod env variables, and it thought for a bit and admitted it actually shouldn't. So as of yesterday I stopped using the "approve for me" mode and now use it only for simpler tweaks and bug fixes.
Fable is not only more intelligent, but also way more insightful. It can sniff out my intent far more effectively, and its "real world" knowledge allows it to act as a seasoned product manager with domain expertise. It can also think outside the box and make suggestions that I would not have thought of. With GPT 5.6 I have to be way more literal.
- What is /goal?
- I would personally love to see the best score over time, not only the final. Then IMO it's also useful to know about effectiveness of /goal
by o10449366
4 subcomments
- /goal has replaced plan mode for me. This is the pattern I use for 95% of my AI work now:
1. Read X feature of Y and tell me when you fully understand it (if there's any detail missing in the summary, repeat until the context is primed)
2. What time is it?
3. /goal Spend X minutes from $time writing a technical design doc on $feature. There must not be any vague language or ambiguity in the document. Read carry_forward_requirements.md and testing_best_practices.md and explicitly incorporate them into the document you write. The document should be executable for a contextless implementer when done and include specific code and document references and changes needed. Spend the full X minutes working on and reviewing this document - do not quit early and wait
Even just spending 10 minutes forcing GPT to write a design doc results in much more robust plans than plan mode, in my experience, and saves time I would spend iterating on the initial plan mode draft anyway.
- GPT-5.6 is unhappy and says the test was materially unfair to the OpenAI model:
https://chatgpt.com/c/6a5bbe6a-a760-83ea-931a-4e2bbe028486
Thoughts?
- Both are entirely useless on complex problem, because they have a bias in training data they can only partially detect in their own output. The answers are getting worse and worse as you dive deeper into the topic you are working on. I thought I could sharpen one of the documents I worked on using Opus 4.8 and GPT-5.5 together. Fable 5 and GPT-5.6 completely destroyed it. Not only it is not human readable anymore, but also doesn't make sense.
- Results seem mostly noise to me. One eval per model, in a large problem space (i.e. a problem which requires many attempts to solve well).
by akoboldfrying
0 subcomment
- If you're curious what the actual optimal Paris cost is, I suggest formulating the problem as an integer linear program and submitting it to Gurobi on NEOS [0]. Gurobi is arguably the strongest commercial ILP solver; big companies pay big dollars to use it to optimise schedules, industrial processes, etc. I'm not sure it could solve this problem to optimality in the 8 hours NEOS provides you, but it might -- KIRO has some similarities to the Vehicle Routing Problem, variants of which are very commercially important. In any case, Gurobi is a monster, and even if you don't get an exact solution, it will give you a lower bound (which may not be tight, but it's nonetheless interesting).
[0] https://neos-server.org/neos/
- Just use OpenEvolve for such problems.
- A deepdive on the /goal effect on a problem literally made for this.
by techpression
3 subcomments
- I love that we have this on one hand and me cleaning up catastrophic CSS made by Sol on the other.
Then again, maybe CSS is the ultimate benchmark.
by MaxoumanBoss
0 subcomment
- Very interesting, will surely help my future projects
by stevefan1999
1 subcomments
- ...is this not a Travelling Salesman Problem?
- [dead]
- [dead]
- [dead]
Hacker News