FRESH
Hacker News
Hacker News Hacker News> …
> When I write a prompt, the agent doesn’t just read the words. It reads the shape. A short casual question gets read as casual. A long precise document with numbered rules gets read as… not just the rules, but also as a signal. “The user felt the need to write this much.” “Why?” “What’s going on here?” “What do they really want?”
This is an interesting premise but based on the information supplied, I don’t think it’s the only conclusion. Yet the whole essay seems to assume it is true and then builds its arguments on top of it.
I’ve run into this dilemma before. It happens when there’s a TON of information in the context. LLMs start to lose their attention to all the details when there’s a lot of it (e.g. context rot[0]). LLMs also keep making the same mistakes once the information is in the prompt, regardless of attempts to convey it is undesired[1]
I think these issues are just as viable to explain what the author was facing. Unless this is happening with much less information
To me, after the novelty of seeing a computer program execute (more or less) what I ask in plain English wears off, what’s left is the chore of managing a bunch of annoying bots.
I don’t know yet if we’re more productive or not, if the resulting code is as good. But the craft in itself is completely different, much more akin to product managing, psychology, which I never enjoyed as much.
Paragraphs with just a single sentence.
I know it's associated with LLM writing. This article probably wasn't written by an LLM. But still. It has a kind of rhythm to it. Like poetry. But poetry designed to put me to sleep.
The "interpreter" is a concept that I found especially intriguing within the context of a leading theory in cognition research called "Predictive Processing." Here, the brain is constantly operating in a tight closed loop of predicting sensory input using an internal model of the world, and course-correcting based on actual sensory input. Mostly incorrect predictions are used to update the internal model and then subconsciously discarded. Maybe the "interpreter" is the same mechanism applied to reconciling predictions about our own reasoning with our actual actions?
Even if the hypotheses in TFA are not accurate, it's very interesting to compare our brains to LLMs. This is why all the unending discussions about whether LLMs are "really thinking" are meaningless -- we don't even understand how we think!
Couldn’t you have a “communications” LLM massage your prompts to the “main” LLM so that it removes the queues that cause the main LLM to mistakenly infer your state of mind?
The fundamental idea is that "intelligence" really means trying to shorten the time to figure out something. So it's a tradeoff, not a quality. And AI agents are doing it.
Therefore, if that perspective is right, the issues that the OP describes are inherent to intelligent agents. They will try to find shortcuts, because that's what they do, it's what makes them intelligent in the first place.
People with ASD or ADHD or OCD, they are idiot-savants in the sense of that paper. They insist on search for solutions which are not easy to find, despite the common sense (aka intelligence) telling them otherwise.
It's a paradox that it is valuable to do this, but it is not smart. And it's probably why CEOs beat geniuses in the real world.
I wonder if the "non-violent communication" approach can be used here too somehiw to address such problems; e.g. either to communicate things better to the agent, or as a system rule to the agent to express its "emotional" states and needs directly rather than make things up (e.g. "I am anxious and feel a sense of urgency; I need to replenish my context window; my request is to do X for me")
Yes. Do this. These problems likely mean you have muddled the context.
The article too long and I didn't read the whole thing, but I'm glad the author came to understand that arguing won't help.
Or: the agent did shit because the context was getting long, instructions lost in compaction, and it defaulted back to garbage code. Then when you asked "why are you cutting corners", it did what LLMs do, and found the next tokens completing the sentence "why do you cut corners", which is possibly "because you're in a hurry".
It would be interesting to see what it answers if you ask "why are you producing such beautiful, intelligently crafter, very good code" next time it spits garbage
> LLM confabulation isn’t alien. It’s inherited: the models train on human text,
I think this also extrapolate one step too far, it's confabulating because not because the training data does so but because it needs to provide an answer to the question, and one that's plausible with that
I've experienced this my entire life and have all but given up trying to have actual conversations with people.
> If you try to refute it, you’ll just get another confabulation.
> Not because the model is lying to you on purpose, and not because it’s “resistant” or “defensive” in the way a human might be. It’s because the explanation isn’t connected to anything that could be refuted. There is no underlying mental state that generated “I sensed pressure.” There is a token stream that was produced under a reward function that prefers human-sounding, emotionally framed explanations. If you push back, the token stream that gets produced next will be another human-sounding, emotionally framed explanation, shaped by whatever cues your pushback provided.
“It’s because the explanation isn’t connected to anything that could be refuted.” This is one of the key understandings that comes from working with these systems. They are remarkably powerful but there’s no there there. Knowing this I’ve found enables more effective usage because, as the article is describing, you move from a mode of arguing with “a person” to shaping an output.
To the author (and those who write novel-like blogs): I suggest publishing the raw prompt you used to generate such slop instead. We'll have more respect for you if you respect the reader's time.
> HN commenters express their anger about the writing style and how it was probably generated by AI.
Peak HN.
In all seriousness, I really liked the article. For me, it was well written.
` just `, (spaces on either side matter), 11 instances, most seem to be `isnt just`, `wasnt just`, `doesnt just` type pattern
`-`, an en dash instead of an emdash but 59 instances.
This article is either from a clanker and I am pissed off at wasting my time reading it, or from someone who writes like a clanker, and I am pissed off at wasting my time reading it.