Now should I mention all the screw up I have seen in several Saas 1b+ valuation, including DocuSign/ and more security oriented ones (PIM related etc?).

For any softwares, you need a minimum critical mindset and experiences that you don't usually see.

Syntax is solved.

But getting agents to write secure code currently seems beyond what can be trained into models even with synthetic data. Or maybe the big labs haven’t tried yet.

Regardless if you truly care about your AI’s output having some property, the only way is to codify how work that has that quality looks - then create deterministic hooks and checks that refuse to let the AI stop until it’s passed the bar.

Skills, MCPs, “you MUST do this” in your agent instructions.. it’s all just new ways to waste tokens trying to asymptotically approach what good work looks like.

You will never reliably get acceptable work unless you build deterministic checking, and enforcement of said checking in a way to model can’t bypass or ignore.

Look into Claude Code hooks - your hook can be a script, and if it exits with exit code 2, it’ll block the model and show it the script’s output. A stop hook can check the model’s work and block its attempt to stop if the work doesn’t meet your bar. The script output can describe what still needs to be fixed and for bonus points, where (line 567 uses untrusted input, paragraph 2 makes an uncited claim, clause 15 references superseded case law, etc.)

Defense is a toally different game, and requires a complete new mindset than creativity. Security is something that you miss one then you lose all.

AIs are good at choosing a good candidate based on a reward model, but it sucks hard at enumerating mundane attack surfaces and make combinations to exploit through.

You played yaself

Are they actually different? I would guess they have roughly the same efficacy. 100% code coverage means nothing, and this is especially true with LLMs.

I mean, yes, but I suppose we live in such a nonsensically thoughtless time that stating the obvious has some value.

> To combat this we need to write a security context file to guide the AI

And you've already lost the plot. The problem is not that you're pulling the arm of the slot machine without wearing your lucky underwear, the problem is that you're delegating security to the slot machine to begin with. Pack it up, you're done.

Edit: To combat this we need to actually write and understand our code.

In our company we are just running agents inside isolated containers with isolated network access so it cannot even SSH or fuck up anything even if it gets access into it... That's the only and safest way... inconvenient, true, but the only safe option.

PS: At the same time I've observed this way actually people uses the agent in a more reasonable way, e.g. producing helper scripts to help them with their daily stuff, produce very specific things, create simple PoCs, but they don't commit to vibe-code all the functionality in their corresponding software products.