FRESH
Hacker News
Hacker News Hacker Newshttps://www.cisa.gov/sites/default/files/2024-03/CSRB%20Revi...
What I found most incredible about the story is that it wasn't Microsoft who found the intrusion. It was some sysadmin at State who saw that some mail logs did not look right and investigated.
https://arstechnica.com/information-technology/2026/03/feder...
I'm not sure if I understand this part. I'm trying to put it into my own words. Is the following correct? The attacker provided an input that was so long, that it was rejected by the database. And the program that submitted the SQL query to the database did not have any logic for handling a query failure, which is why there is no trace of the login attempt in the log or elsewhere.
If I remember the issue right, we lost a client secret (it just vanished!) and I went to the audit logs to see who dun it. According to the logs, I had done it. And yet, I also knew that I had not done it.
I eventually reconstructed the bug to an old page load. I had the page loaded when there were just secrets "A" & "B". When I then clicked the delete icon for "B", Azure deleted secrets "B" and "C" … which had been added since the page load. Essentially, the UI said "delete this row" but the API was "set the set of secrets to {A}". The audit log then logged the API "correctly" in the sense of, yes, my credentials did execute that API call, I suppose, but utterly incorrectly in the sense of any reasonable real-world view as to what I had done.
Thankfully we got it sorted, but it sort of shook my faith in Azure's logs in particular, and a little bit of audit logs in general. You have to make sure you've actually audited what the human did. Or, conversely, if you're trying to reason with audit logs, … you'd best understand how they were generated.
I don't think I would ever accept audit logs in court, if I were on a jury. Audit logs being hot lies is within reasonable doubt.
Is this a bad move? What should I tell them?
Absolutely savage lol
[If you didn't read the thing, it's one curl command.]
It should really horrify everybody that Microsoft is not investing more into Azure considering they host the worlds most known LLM (and used?).
Google Cloud is simplistic in comparison. AWS is full of legacy complexity (IAM policies, sigh) but it's fairly self-contained and can be worked around by splitting stuff into accounts.
I have not looked at Oracle cloud yet. Is it any better than MS?
The Audit log showed the service identity of Application Insights, not the user that pressed the button! The cloud ops team changed the size back, and then the mysterious anonymous developer... changed it back. We had to have an "all hands" meeting to basically yell at the whole room to cut that out. Nobody fessed up, so we still don't know who it was.
The Azure Support tech argued with me vehemently that this was by design, that Azure purposefully obscures the identity of users in audit logs!!! He mumbled something about GDPR, which is nonsense, because we're on the opposite side of the planet from Europe.
At first I was absolutely flabbergasted that anyone even remotely associated with a security audit log design could be this stupid, but then something clicked for me and it all started making sense:
Entra Id logs are an evolution of Office 365 logs.
They have a legitimate need to protect customer PII, hence the logs don't contain their customers' private information when this isn't strictly necessary. I.e.: Microsoft's subcontractors and outsourced support staff don't need and shouldn't see some of this information!
The problem was that they re-used the same code, the same architecture decisions, the same security tradeoffs for what are essentially 100% private systems. We need to see who on our payroll is monkeying around with our servers! There is NO expectation of privacy for staff! GDPR does NOT apply to non-European government departments! Etc...
To this day I still see gaps in their logging where some Microsoft dev just "oops" forgot to log the identity of the account triggering the action. The most frustrating one for me is that Deployments don't log the identity of the user. It's one of only three administrative APIs that they have!
[1] As an aside: The plan had a 3-year Reservation on it, which meant that we were now paying for the original plan and something twice the size and non-Reserved! This was something like 5x the original cost, with no warning and no obvious way to see from the Portal UI that you're changing away from a Reserved size.