In nearly half the false negatives from both the mammogram and DWI datasets, the cancer was categorized as occult by two breast radiologists, meaning the cancer was invisible to a trained eye. The AI model's non-occult false negative rate on the mammography data is 19.3%.

For that 19.3% figure, see Table 2: 68 non-occult in AI-missed cancer, 285 non-occult in AI-detected cancer.

This study did not compare the AI to a radiologist on a mixed set of healthy and cancer images.

It was retrospective-only, i.e. a case series on women who were known to have breast cancer, so there were zero false negatives and zero true negatives, because all patients in the study truly had cancer.

The AI system used was a ConvNet used commercially circa 2021, which is when the data for this case series were collected.

If it said "AI something", I'd be fine with it. It's a statement about that something, not about AI in general. Use it as an adjective (short for "AI-using" I guess?), not a noun.

If I pluck a guy off the street, get him to analyze a load of MRI scans and he doesn't correctly identify cancer from them I'm not going to publish an article saying "Humans miss X% of breast cancers" am I.

1. They only tested 2 Radiologists. And they compared it to one model. Thus the results don’t say anything about how Radiologists in general perform against AI in general. The most generous thing the study can say is that 2 Radiologists outperformed a particular model.

2. The Radiologists were only given one type of image, and then only for those patients that were missed by the AI. The summaries don’t say if the test was blind. The study has 3 authors, all of which appear to be Radiologists, and it mentions 2 Radiologists looked at the ai-missed scans. This raises questions about whether the test was blind or not.

Giving humans data they know are true positives and saying “find the evidence the AI missed” is very different from giving an AI model also trained to reduce false positives a classification task.

Humans are very capable at finding patterns (even if they don’t exist) when they want to find a pattern.

Even if the study was blind initially, trained humans doctors would likely quickly notice that the data they are analyzing is skewed.

Even if they didn’t notice, humans are highly susceptible to anchoring bias.

Anchoring bias is a cognitive bias where individuals rely too heavily on the first piece of information they receive (the "anchor") when making subsequent judgments or decisions.

They skewed nature or the data has a high potential to amplify any anchoring bias.

If the experiment had controls, any measurement error resulting from human estimation errors could potentially cancel out (a large random sample of either images or doctors should be expected to have the same estimation errors in each group). But there were no controls at all in the experiment, and the sample size was very small. So the influence of estimation biases on the result could be huge.

From what I can read in the summary, these results don’t seem reliable.

Am I missing something?

> Their findings offered reassurance: DWI alone identified the majority of cancers the AI had overlooked, detecting 83.5% of missed lesions for one radiologist and 79.5% for the other.

The combination of AI and this DWI methodology seems to identify most of the cancer, but there’s still about 20% of 1/3 that gets missed. I assume that as these were confirmed diagnoses, they were caught with another method beyond DWI.

I can detect 100% by

    def detect(x):
        Return True

the setup: 1. 400s confirmed patients 2. AI reads Mammography ONLY and missed 1/3 3. on those AI missed patients, radiologists do a second read on MRI, which is the gold standard for differential. evidence: the referenced paper at the bottom <Added value of diffusion-weighted imaging in detecting breast cancer missed by artificial intelligence-based mammography.>

So, the whole point it (or its reference paper) is trying to make is: Mammography sucks, MRI is much better, which is a KNOWN FACT.

Now, let me give you some more background missing from the paper: 1. Why does Mammography suck? well, go google/gpt some images, its essentially X-ray for the breast, which compress 3D volumes into 2D average poole plane, which is infomation lossy. SO, AI or not, the sensitivity is limited by the modality. 2. How bad/good is Mammography AI? I would say 80~85% sensitivity agaist very thorough+experienced radiologist without making unbearable amount of FP, that probably translates to 2/3 sensitivity against real cancer cohert, so the referenced number is about right. 3. Mammography sucks, what's the point? its cheap AND fast, you can probably do walk-in and get interpretation back in hours. Whereas MRI you probably need to schedule 2 weeks ahead if not MORE. For a yearly screening, it works for the majority of polulation.

and final pro tip, 1. breast tumor is more pravelent than you think (maybe 1 in 2 by age 70+) 2. most guides recommend women with 45+ do yearly checkup 3. if you have dense breast (basically small and firm), add ultrasound screening to make sure. 4. breast feeding does good for both the mother and child, do that.

peace & love

I hope something good comes out of this, as I have known women whose lives were deeply affected by this.

This is Skynet 2.0 or 3.0. But shit. James Cameron may have to redo The Terminator, to include AI. Then again, who would watch such a movie?

"One AI is not great" is not an interesting finding and certainly not conclusive of "AI can't help or do the job".

It's like saying "some dude can't detect breast cancer" and suggesting all humans are useless.