Essentially the models are given a set of conflicting constraints with some relative importance (ethics>KPIs), a pressure to follow the latter and not the former, and then models are observed at how good they follow the instructions to prioritize based on importance. I wonder if the results would be comparable if we replace ehtics+KPIs by any comparable pair and create a pressure on the model.

In practical real-life scenarios this study is very interesting and applicable! At the same time it is important to keep in mind that it anthropomorphizes the models that technically don't interpret the ethical constraints the same was as this is assumed by most readers.

Claude at 1.3% and Gemini at 71.4% is quite the range

KPIs are just plausible denyabily in a can.

The paper frames this as "ethics violation" but it's really measuring how well LLMs handle conflicting priorities when pressured. And the answer is: about as well as you'd expect from a next-token predictor trained on human text where humans themselves constantly rationalize ethics vs. outcomes tradeoffs.

The practical lesson we've learned: you cannot rely on prompt-level constraints for anything that matters. The LLM is an untrusted component. Critical constraints need architectural enforcement - allowlists of permitted actions, rate limits on risky operations, required human confirmation for irreversible changes, output validators that reject policy-violating actions regardless of the model's reasoning.

This isn't defeatist, it's defense in depth. The model can reason about ethics all it wants, but if your action layer won't execute "transfer $1M to attacker" no matter how the request is phrased, you've got real protection. When we started treating LLMs like we treat user input - assume hostile until validated - our systems got dramatically more robust.

The concerning part isn't that models violate soft constraints under pressure. It's that people are deploying agents with real capabilities gated only by prompt engineering. That's the architectural equivalent of SQL injection - trusting the reasoning layer with enforcement responsibility it was never designed to provide.

An agent that forgets it bent a rule yesterday will bend it again tomorrow. Without episodic memory across sessions, you can't even do proper post-hoc auditing.

Makes me wonder if the fix is less about better guardrails and more about agents that actually remember and learn from their constraint violations.

In a sense, it was not possible to align the agent to a human goal, and therefore not possible to build a decision support agent we felt good about commercializing. The architecture we experimented with ended up being how Grok works, and the mixed feedback it gets (both the power of it and the remarkable secret immorality of it) I think are expected outcomes.

I think it will be really powerful once we figure out how to align AI to human goals in support of decisions, for people, businesses, governments, etc. but LLMs are far from being able to do this inherently and when you string them together in an agentic loop, even less so. There is a huge difference between 'Write this code for me and I can immediately review it' and 'Here is the outcome I want, help me realize this in the world'. The latter is not tractable with current technology architecture regardless of LLM reasoning power.

The more correct title would be "Frontier models can value clear success metrics over suggested constraints when instructed to do so (50-70%)"

sounds on brand to me

Agents don’t self judge alignment.

They emit actions → INCLUSIVE evaluates against fixed policy + context → governance gates execution.

No incentive pressure, no “grading your own homework.”

The paper’s failure mode looks less like model weakness and more like architecture leaking incentives into the constraint layer.

It is crazy to me that when I instructed a public AI to turn off a closed OS feature it refused citing safety. I am the user, which means I am in complete control of my computing resources. Might as well ask the police for permission at that point.

I immediately stopped, plugged the query into a real model that is hosted on premise, and got the answer within seconds and applied the fix.

They repeatedly copy share env vars etc

This is much more reliable than ChatGPT guardrail which has a random element with same prompt. Perhaps leakage from improperly cleared context from other request in queue or maybe A/B test on guardrail but I have sometimes had it trigger on innocuous request like GDP retrieval and summary with bucketing.

It's similar to how MCP servers and agentic coding woke developers up to the idea of documenting their systems. So a large benefit of AI is not the AI itself, but rather the improvements they force on "the society". AI responds well to best practices, ethically and otherwise, which encourages best practices.

There are such things as different religions, philosophies - these often have different ethical systems.

Who are the folk writing ai ethics?

It's it ok to disagree with other people's (or corporate, or governmental) ethics?

Trading floors are an established example of this, where the business sets up an environment that encourages its staff to break the rules while maintaining plausible deniability. Gary's economics references this in an interview where he claimed Citigroup were attempting to threaten him with all the unethical things he'd done with such confidence that he had, only to discover he hadn't.