FRESH

Hacker News

A Large-Scale Empirical Study of AI-Generated Code in Real-World Repositories

4 points by softwaredoug

by fwlr

0 subcomment

Near the end of their methods:
```
    For each likely AI-generated target, we search human controls from the same repository and same programming language, controlling for project-level practices … We also apply size matching: a control must be within 20% lines of code of the AI target
```
And then from the start of their results and analysis:
```
    Comparing the lines of code (LOC) between AI-generated and human-written files, we observed no statistically significant differences
```
Here I would expect some sort of caveat that this result is a verification that their size-matching was effective in removing differences in lines of code between the target and control. However, there is no such thing, indicating to me that the authors are unaware that what they are detecting are the effects of their own methods. This leads me to strongly discount their stated insight regarding structural differences, and frankly the rest of their paper too.