The model only has partial observability of the program it is working on (whatever tool call outputs are present in the context), as well as the trajectory of actions it has taken, and from this is building up some internal beliefs about the program - the probes used were looking for pretty crude things like "is this program well-formed" and "is this program correct (will it pass tests)".

The paper says that these program "properties" (beliefs) predict future state of the program up to 25 "steps" ahead, but given the setup this seems to be expected. An agent is trying to fix a program and/or maintain it in a working state, so it doesn't seem surprising that current well-formedness and correctness persist into the future, or that the model is correctly "optimistic" about the outcome of the next action it is planning/predicting.

This incremental belief building from partial observability reminds me of the ability of LLMs to predict valid chess moves when only given a truncated history of the games moves so far (e.g. last 20 moves, not all moves back to start of the game).

But this has only been shown on simple tasks, so I think this paper is still quite neat. The interesting thing is that they show "future horizon length" varies across models.

I know people, who initialize all required variables and write the logic after. which used to feel bonkers to me until I realized, they've done enough practice and memorization to be able to figure what they would need 10 steps down the line.

this does show that, models have a better model of the task and the expected end state.

That after a model has context about a project, the probes indicate a state that validates that?

Seems that the paper is highlighting the very nature of what LLMs are and what we expect them to be?

And that there is no 'thinking' here, it's just the state of the model?

Finally there is evidence that the model kinda actually knows the correct token spend on each method.

Are these probes effectively run in parallel? The way this reads is more about predicting a future outcome than keeping the current token relevant based on past tokens.

Then how do humans create something 'creative'—something that didn't exist before? I think it might be because the process of simplifying the complex system of nature differs between individuals. The data being learned now is all labeled by humans and simplified through human cognition. Within that kind of information, creativity seems hard to emerge.

Ultimately, with data that already contains interpretation, no matter how much you repeat the learning, it just becomes an encyclopedia that only explores within human knowledge, repeating predictions within human interpretation. So I wonder if we actually need a different encoder that interprets raw data—not based on human interpretation.

In reality, what changed Newton's absolute time to Einstein's relativity was a conclusion derived simply from observing the world. Newton's interpretation was supported by a lot of evidence in its time. If an AI studied all the medieval data from Newton's era, could it actually come up with the theory of relativity?

I'm always curious about this. I think AI is already very good at coding and will soon become better than humans. Logical structures are ultimately human interpretations, and reasoning within that framework is something AI can probably do more logically than humans. In other words, once humans create the framework, stacking the logical Jenga blocks within it—AI will be better at that.

But true creativity lies in breaking the framework itself, and I'm skeptical about whether AI can do that. The encoder also seems insufficient. There will likely be limits. I might be trapped in my own biases.

But the limitations of the current approach seem too clear to ignore.

When I look at the approach of these papers, it feels like an argument that adding shadows that imitate the world will eventually make them become the objects themselves.

I think the text, code, images, papers, and conversations that humans leave behind are not the world itself, but rather shadows of the world that have passed through human cognition and language. No matter how much you learn from those shadows, whether that leads to the ability to actually engage with the objects themselves seems like a separate issue.

I feel like something different is needed. But I'm not intellectually sharp enough to reason this through logically.this is just my intuition

No. That's simple PR hype. Parrotry is not reasoning.