In most cases, LLMs can get you 80-95% of the way, sometimes less, sometimes more. And heck, sometimes, it just gets you somewhere wrong.

But it seems everyone is arguing about whether LLMs can be perfect software engineers in isolation running in a closet, and using that to say that LLMs do not have a massive potential in other scenarios.

Sometimes, I like to imagine how much more productive most organizations could be from the things that the internet gave us, even to this day. Most companies never really do even a fraction of what is possible. That helps to ground my view of LLMs as well.

The fault dear Brutus isn't in our language models, but in ourselves.

Programmers are rewriting and reinventing the same techniques more often than any other vocation I can think of, and so we were primed for a really good search over prior art. The fact that AI can also adapt that prior art to your particular use case makes it even more powerful.

Much like how great success never came from cobbling together various bits of copy-pasted code from Stack Overflow though, current AI can't really build your whole project.

Are they are all suddenly turning into zombies? No. Do they have any real idea what that is going to do to their body a few years down the line? Also no. Could it be catastrophic? Maybe!

I think about this when I think about how violently much of the industry has pivoted into AI being the primary generator of code in the last 6ish months. AI is the peptide, your codebase[2] is the body. Literally no one knows how maintainable this approach is, because there simply hasn't been enough time to find out. It could be fine. It could be a complete mess, with your entire engineering team falling asleep at the wheel, lulled into thinking they understand what is being built when they don't, completely impotent to fix or maintain it once the LLM is no longer able to.

[1] https://www.bbc.co.uk/news/articles/cdr268m5pxro

[2] Well, _their_ codebase. I've stopped doing it with my own personal codebases, unless I genuinely don't care about maintainability or longevity

The hardest thing in software engineering is solving the right problem. The ability to identify the right problem to solve, is IMO, what distinguishes the top senior engineers. And we could have endless discussions about what constitutes the right problem, but for the sake of this discussion, let's reduce it to: the problem whose resolution adds the most value to the product for the amount of complexity and afferent costs that it incurs.

Once upon a time, long ago, I worked on a Web product whose original junior designer had figured it would be neat to be able to manage the backend with LDAP tools. So the database schema and structure that the product used mimicked that of OpenLDAP, with compound CN keys, and the entire codebase had to deal with that structure whenever reading from or writing to the DB. LDAP compatibility was not the right problem to solve when designing the DB schema.

But software that solves the right problems can be hard to identify because, quite often, how it does things seems so obvious that it's not readily apparent what other designs might have been chosen.

Now, the thing that usually keeps the blast radius of wrong-problem designs limited over time, is the very friction that they introduce. Development slows down, including the development of more wrong-problem designs. It's a self-limiting phenomenon.

And that's one major thing which worries me about LLM coding agents:

They paper over this friction. They don't repair it; they just make it so its cost is deferred.

So you gradually end up with codebases that grow unboundedly complex for the value they provide, with no controlling mechanisms.

You end up with juniors who never face the feedback loop from which they'd develop the engineering instincts and the taste for what makes a problem the right problem to solve in a given design.

At scale, as a field, you might end up forgetting there ever was such a thing as solving the right problem.

And I don't know what to do about that. Plan for an early retirement, maybe.

At the same time in no part of his post is any code snippet or anything to latch on to of "the model performed poorly here when it should have done this" - this style of criticism seems to be a pattern of most of these "the LLMs will never work" style posts on blogs and twitter.

They obviously can perform better than autocomplete and in my own day to day development build out huge portions of a codebase that I would have expected a junior or midlevel engineer to perform at.

How are we really supposed to grasp their actual capabilities when no one will actually cite specifically what mistakes they are making.

My main issue has been the inconsistent quality across between model releases and the tendency to insert older APIs or documentation, especially with command line tools.

I can understand if the model struggles with a million line monolithic codebase with a decade of cruft but can't think of why it'd be too much of a pain with new codebases.

When I got into agentic coding a year or two ago I was sure it was only good at autocomplete. Something happened earlier this year where the models hit a new level of capability.

Everyone I know now just does agentic coding, and it’s really amazing. I think we should just try pushing this as far as we can possibly go, it really feels like the acceleration of the human race is upon us.

https://en.wikipedia.org/wiki/George_Hotz

being able to program is not the only skill required to be a successful software engineer, so no ai agents cannot be software engineers

very important distinction - i personally like the radiologist example - looking at scans is a part of a radiologist job, AI can do it better than most of them, but looking at scans is a small part of the job, most of it communicating with doctors to help their patients

I hope that professionalism still matters as these new ways of doing things strikes me as unprofessional as f...

Yeah, the next macOS will be worse... time to place bet on prediction market

But it can write working code much faster than I can.

And in a lot of cases, unfortunately, faster beats better.

I've heard this said so many times, but my experience has just been so dramatically the opposite that it rings false. But geohot seems to be a pretty productive and smart guy, so it's hard to just dismiss what he's saying.

I get the sense that he's truly one of the 10x engineers. And maybe he can do it faster and better manually. But for those of us who aren't 10x, I think it lets us bridge that gap. Now we're getting back to "status anxiety": is this an attack on his ego, if the average becomes 10x?

Anecdote: Over 2 weeks of spare time, I used AI tooling to build a fairly sophisticated debian package caching proxy server (~72KLOC, 27K implementation, 45K tests). This would have easily taken me 6 months of focused time to implement by hand. I literally couldn't have done it because I can't take that much time off work and I have other weekend/evening obligations.

I chalk this up to primary two reasons. First, I cared a lot more about the implementation details of the C program than I did the Python one, and second, it's just better at simple stand-alone python programs than it is at C programs.

The criteria I know use is "do I care about the implementation details of this?". If I do (because for example it's going to be long-term code that I need to maintain) then the agent likely isn't worth it. But if I don't, there are huge efficiency gains to be had using the agent.

The quality of the codebase decays precisely at the rate you stop reading the results. This is not an issue of AI writing the code. This is an issue of unreviewed code. geohot's issue is entirely valid. This problem does exist. But this isn't dependent on the generation phase.

If the answer is yes, the argument doesn’t matter: you just run the loop and wait for llm analog of moore’s law to get costs down.

The issue is people mix up complexity, novelty, repeatability and scale.

Well documented complex problems can easily be solved by LLMs.

Doing the same thing over and over again is easy for an LLM.

Novelty and scale is very hard for an LLM.

Even small novel problems confuse LLMs.

When you start a new code base the LLM smashes through the boilerplate work. Then when it gets to scale it struggles with context rot plus novelty.

The AI agents are great, and any expert can prompt them correctly to get good code. LLMs occasionally pick wrong patterns and start digging a hole, but this is why an expert is required. The code itself is just not worth writing when a detailed prompt can get you the same code typing 20x less text.

Where I agree with the post is:

The adoption of AI agents into software engineering is a problem. Solo projects are great, but our teams have not adjusted to the speed-of-change to a mental model of a project. So I see orgs making a choice to either: slow down or forgo the shared mental model.

Anybody choosing to forgo the mental model is building crooked legacy slop at scale. You can and should save the mental model to an AGENTS.md, but devs need it in their brain to prevent the digging a hole behavior.

To be fair the digging a hole behavior is something humans do just as well. But in teams you'd communicate enough to catch it - hopefully^1. It's the combination of higher speeds and teams that's creating a bit of a disaster.

I'm not sure what a good solution is either. There is a case for solo devs running for 2-month sprints with much more freedom. Perhaps we'll have an "AI Agile manifesto" within a year.

[1] Though you should not underestimate the amount of poor code being created before LLMs. There are enough teams for whom LLMs are practically all upsides. Stay very far away from those.

Once Humans just had oral language, and we could us words to pass ideas from one human mind to another. Then with writing ideas could pass to minds that weren't immediately close together in space or time.. and with this we made complext global spanning civilization. When words just become noise, that one has to be suspect of each one as to whither they'er coming from another human mind, or just a statistical process, can this civilization even survive?

How much will it take for AI agents to pass from distilling decades of collective wisdom to copying each other's worst mistakes?

However, this isn't a plug to be using AI for coding everything, but a more general plug that AI should be integrated to a lot more things outside of the mainstay of chatbots.

There is a lot of merit to using AI to establish a new abstraction layer.

LeCun thinks that LLMs are a bad fit for AI that understands the physical, dynamical systems that we inhabit, and that understanding this is necessary for AGI/ASI.

I don't know that Hutter is bearish on LLMs, but Hutter is interested in AI that can reason exceptionally well given infinite compute, and approximations of such a reasoning AI. I think he is open to the idea that LLMs can be such an approximation.

If you are already comfortable with letting other devs work on features then it's easier, because it's similar (arguably you have more control with AI, because what you say goes regardless of hierarchy).

I've been running into this experience with non-code artifacts, like slideshows and documents.

I can't agree with this. You tend to get one point of view, often without any actual resources and references so you have to go look it up yourself, on [insert search engine]. Plus, what does it say when we consider an AI the one stop for our data intakes.

I think the discussion about methods (coding agents included) depends on answering those questions. Seems pointless to claim these agents [dont] make you more productive.

Although, at a first glance, the productivity increase does seem like nothing I’ve seen before. Even more than the transition of making webapps in plain js -> jquery -> frameworks or going from something like Flask to using Rails.

Problem is this is not evidence based. I just feel prototyping has speed up 100x. So the number of iterations/attempts has gone up. Transforming specs into a test suite takes a fraction of the time. Dunno, feels weird not to be able to be overall more productive (do more with less time) if you have these new tools.

  But each time I suspected I could have done it better and faster manually

I hate how both the for and against case for LLMs are just so bloody terrible at addressing these things.

There aren't many truly general purpose tools so viewing things this way seems like either a fantasy or an over-reaction. And if nothing else the processes we use will have to change along with the tools.

It's the early days so we still have a lot to figure out but one of the most significant is which tools are appropriate for what sort of tasks. I've had good luck refactoring a small code base, building some small hobby projects and building features for our company's product. But, I've also dodged bullets doing greenfield development on some features where Claude (my default) has made what seemed like sound choices early on, and which I approved of, only to build something fragile or with unforseen consequences. I haven't quite figured out what distinguished those situations from the successful ones but I'm trying. But it's complicated by the fact that things are evolving quickly and yesterday's failure mode isn't the same as today's and, for that matter, yesterday's successes aren't guaranted to be repeatable today.

To use a Geohot-inspired analogy, what we have now is like the Google self-driving car of 2010. It works most of the time, yet sometimes fails in unpredictable ways. So you need a safety driver behind the wheel to constantly watch what it's doing (the code review).

A real AI agent would not need a safety driver. We don't have that but many people are basically saying "fuck it, I'm just going to set this car off on its own and see what happens". And sure if you're prototyping it's not dangerous. But for production systems that is dangerous.

Nailed it!

At my last place this was encouraged (by non-technical leadership driving the AI adoption policies, as well as setting salaries) and seen as a huge win.

The "step change in number of created PR's" was celebrated (cult-style), and by one of the (co) CEO's praised as a paradigm shift of the same magnitude as the personal computer. Meanwhile, I was stuck finding insta-reject level bugs in pull requests from people one-shotting 6000 line PR's "finally solving" long-standing issues from the backlog. Needless to say I left.

See, the project actually has a well thought out structure that I design carefully, but more and more of it gets filled out by Codex. Codex is not smart enough to remember all the high-level design considerations, some of which had not been documented because I was just implicitly assuming them. So the fix was to use Codex to isolate the error, think about in terms of the high-level design, and fix the problem, which was partially an implementation problem, and partially a problem of the high-level design.

I fixed the high-level design with discussions with Codex, and documenting this, and then let Codex implement the fixes. The discussion took me more than an hour, the implementation was done in a few minutes.

This working style is similar to doing math: You have a high-level idea of what you are doing, and let that guide you, and Codex assumes the role of something that fills out all of the details you take for granted. Often it turns out your high-level idea had flaws, and this shows up in your code not working as expected. So you revise your high-level idea, refactor the code to reflect the modified high-level design, rinse and repeat.

Working this way is still really hard, but it allows me to do things I could not have done before. Getting your ideas validated (or refuted) in minutes instead of days is huge, and makes it possible to march through stuff that would have turned into a deadly swamp before, at least for me.

Now. Do I think that most corporate programmers will use Codex or CC in this way? I don't know, but I think probably not. So what will stop them going into the swamp until it swallows them, instead of backing up in time and marching around it?

This is dangerously incorrect. AI summaries of search results consistently return incorrect information and grossly oversimplified and thus misleading summaries, neither of which are detectable unless one either has prior domain knowledge or spends time drilling into search results to validate the AI output.

For something to take "longer and longer" to realise, doesn't they imply that it's been realised at least once before or that there was an expected deadline for the realisation?

Okay, that's a nitpick.

For me, the AI is essentially "faster hands" that can type what I am thinking way faster than I can do it. I tell it what I want, I give it the broad architecture and design patterns/types to use, and any specific test conditions, and let it write all of that usually by the time I have responded to a single email or chat message or two. Custom instructions etc build overtime to address model blind spots or my own personal taste so I don't have to repeat myself in every prompt for cross-cutting things.

Does it "one shot it"? Almost never - we go around the cycle a few times, treating it like pair programming a junior or intern by keeping a close eye on the broad direction and making sure it is acceptable - course-correcting where it matters, but cutting some slack where it doesn't. Sometimes I ask it why it picked a particular approach (that I wouldn't have necessarily) and it gives me a cogent explanation and we go with it, so I actually sometimes learn new things from it too which is great.

The other use case is just it's sheer capacity to research a codebase and hold everything in it's attention at once. It can comprehend unfamiliar code way faster and way more in-depth than I can. So if you are in an unfamiliar code base or a language or framework you are not that familiar with, it absolutely shines because it can just absorb all that info in seconds, and then you can just drill it with questions and what-abouts and how does it do this and what technique is used for that and that, what are the existing patterns and norms in this codebase when it comes to foo or bar? Etc etc

What I am not doing is deferring everything off to the AI unless it really doesn't matter (e.g. disposable one-off or prototype code). Same that I would not expect a junior or intern to make big architectural decisions when implementing something - you keep them on a fairly close leash and watch what they are up to.

This line which he wrote, will override any quality gaps, because the cost to produce that shitty software will be lower than the cost to produce good software.

If your work was previously googling stack overflow, it can be incredibly useful at working through that. Which let’s face it, that’s what a lot of us do.

You guide the AI with some prompts and give it some guidance on how to scenario-test it. It makes some classes, test methods. Maybe ~2000 lines and you do a quick verification, check if the overall idea looks okay. Ask it to fix a few design things and then merge it.

Its much easier than doing it yourself with all the boilerplate and understanding each esoteric language specific thing. Which library do I use for UDP communication in golang? The agent might have made a good assumption. These kind of things is where it speeds it up.

I mean, this has been the trend for decades really, before LLMs were a thing. The incentive is skewed toward quantity rather than quality. The new tools just add more fuel to the fire.

Code quality is also really lacking in much of the industry. The truth is, these LLM models, as limited as they are, program at a level above that of the median junior programmer.

I don't get how anybody who has used the SOTA models in the last 3-4 months can write a sentence like this?

They most certainly can program. And usually better than 90% of my coworkers.

The question is really.. Can they engineer? By which I mean handle the duties of a software engineer working in a team, managing a large complex system, making reviewable pieces, forward progress in incremental steps, etc.

No, that part I'm definitely more skeptical about. That requires slave driving by the person in front of the prompt.

But this is a useful distinction to make. Because making overly pessimistic claims about the coding capabilities of the models makes me question the author's experiences with them.

I think agentic tools are toxic to team programming culture and engineering that produces reliable stable results. But I wouldn't for the life of me question their ability to write programs.

So what he is telling us? That agents are not infaillable and they are not capable to one shot complex software and they do not produce perfect code?

We know what and the solution is to use agents for what they are good at and work around their limitations and we have a human in the loop.

>not some RLVR shit that comments out the failing test and tells you all the tests are now passing

That's what harnesses should be about: detect when the agent is misbehaving and force it to take the right approach.

This example in particular should be easy to solve if we generated the tests before coding and we have a workflow or state machine that doesn't allow the agent to disable tests and doesn't allow it to reach the next stage unless all tests are passing.

EDIT: To people downvoting me, please come up with a reasonable bet and lets try to work it out.

EDIT 2: $500 bet paid to your account on whether LLM's are going to still be used productively or not. No one?

EDIT 3: Any bet that would express the author's argument in a way that can be disproven in the future

Are we really still doing this?

-- I think this article is COPE, if I'm being quite honest. I thought of putting cute analogies, like the C programmers saying the Python and Javascript programmers are not "hardcore" enough... but the truth should be obvious to anyone using LLMs effectively.

-- Current AI is a much better programmer than 100% of people and when directed by someone in that top 10%, it's a force majeur.

In other words - they can program, and probably better than you.

I don't like being too critical but this is a really superficial post - as if either 'AI is a Software Engineer - or - It must be Fraud'

It's an extremely powerful tool that is very 'pattern oriented' and with guidance can absolutely write great code - and even across modules given the right basis.

It's also great at so many other tasks - finding bugs in big code bases, doing migrations etc.

It's not going to make very goo architectural decisions for you, and if you're doing anything novel you have to read most of the code ... but that's too be expected.