- The conceptual problem is that we keep wanting to compare AI behavior to that of traditional computers. The proper comparison is comparing AI, and how we trust or delegate to it, to the concept of delegating to other humans or even to domestic animal. Employees can be trained and given very specific skills and guidelines but still have agency and non-deterministic behavior. A seeing eye dog, a pack mule or chariot horse will often, but not necessarily always do what you ask of them. We've only been delegating to deterministic programmable machines for very short part of human history. But ad human societies, we've been collectively delegating a lot of useful activities to non-perfectly-dependable agents (ie each other) for a very long time. As as humans we've gotten done more that a few notable things in the last several millennia with this method. However, humans as delegates or as delegators have also done a lot of horrific things at scale to, both by accident or by design. And meanwhile (gestures broadly around everywhere) maybe humans actually aren't doing such an optimal job of running and governing everything important in the world?
When compared to how human make a mess of things like in the real world, how high does the bar really need to be for trusting AI agents. Even far shy from perfect, AI could still be a step function improvement over trusting ourselves.
by cramsession
13 subcomments
- > You bought a laptop or desktop with an operating system, and it did what it said on the tin: it ran programs and stored files.
I feel like people may be viewing the past with rose colored glasses. Computing in the 90s meant hitting ctrl-s every 5 seconds because you never knew when the application you were using was going to crash. Most things didn't "just work", but required extensive tweaking to configure your ram, sound card... to work at all.
- This feels like the modern incarnation of "packet intent", the mythical security property of knowing what an incoming request is trying to do rather than what it is. Variants of "packet intent" have been sought after going all the way back into the 1980s; it's helpful to recognize the idea when it appears, because it's a reliable marker of what you can't realistically accomplish.
by aykutseker
5 subcomments
- been building on claude code for a while. the post's framing is right.
mcp gives you open standards on the tool layer but the harness
(claude code, cursor) is still proprietary. your product is one
anthropic decision away from breaking.
the user agent role the post calls for needs open harnesses, not just
open standards. otherwise we end up rebuilding mobile under a new name.
- The framing assumes the agent can reliably represent its principal, and I'm not convinced that holds even if you get everything else right.
The problem is that the agent itself is the attack surface. An adversary who controls the communication channel can manipulate what the agent believes about who it's talking to, which means anything it holds, its list of authorized actions, a shared secret you gave it, whatever, can be exfiltrated in ways the agent can't detect because the manipulation happens below the layer where it can reason about trust.
Open harnesses and open standards help but they don't close this gap, because the thing you need to trust, the agent's own judgment about its principal, is exactly what gets compromised. The trust chain has to go below software entirely: hardware attestation, signed commands with keys the agent can verify but never access. That's really an OS problem dressed up as an agent architecture problem.
- The thing I don’t like about “agents” is that I consider my computer a tool that I use and control. I don’t want it doing things for me: I want to do things through it. I want to be in the driver’s seat. “Notifications” and “Assistants” and now “Agents” break this philosophy. Now there are these things doing “stuff” on my computer for me and I’m just a passenger along for the ride. A computer should be that “bicycle for the mind” as Jobs put it, not some autonomous information-chauffeur, spooning output into my mouth.
- The browser analogy holds because publishers wanted browsers. Sites lived with User-Agent and robots.txt because the click paid for it.
AI agents are the destination. No return click to bargain with. That's why Cloudflare just went default-block + 402 Payment Required instead of waiting on a standards body.
Open standards on the agent side are the easy half. Getting sites to show up is the part W3C can't fix alone.
- I like how the author notices that it really got a start with cloud computing.
- The most important thing we can do for AI to be a net positive to society is to ensure that its loyalty is to the user, and not the state.
There is no legitimate intermediate position - The skew will go one way or the other.
- First half: relatively cogent diagnosis of understood problems in computer privacy.
Second half: specious claims about AI mostly based on a vague "we don't know what they can do, so maybe they can do anything?" rhetorical maneuver.
- i think whats missing is the raison detre of the Agents isnt a new usecase, its a context prune for the same limitations LLMs provide. LLM as Agent is a subset, where the goal of the agent is set by the parent and is suppose to return a pruned context.
if you dont recognize the technical limitations that produced agents youre wearing rose tinted glasses. LLMs arent approaching singularity. theyre topping out in power and agents are an attempt to exentend useful context.
The sigmoid approacheth and anyone of merit should be figuring out how the harness spits out agents, intelligently prunes context then returns the best operational bits, alongside building the garden of tools.
Its like agents are the muscles, the bones are the harness and the brain is the root parent.
- [flagged]
Hacker News