FRESH

Hacker News

Through the looking glass of benchmark hacking

32 points by jxmorris12

by fsh

2 subcomments

I don't get the point. The model has presumably been trained on all public GitHub code, so the evaluation is tainted anyway.

by mgrund

3 subcomments

I was under the impression that swe-bench (and I guess most other benchmarks) were supposed to be run offline?
I get that you may accidentally include something in local git history, but it feels off to me to run these kinds of benchmarks online.

by pratio

1 subcomments

Are you guys affiliated to https://poolside.fm/ or https://poolsuite.net?

by ej88

0 subcomment

This is cool!
I used to work on post-training & evals. it's really hard to make a good eval set and catch all forms of reward hacking. Excited to see more from poolside!

by schnitzelstoat

1 subcomments

It was an interesting read - perhaps I misunderstood the part about blocking GitHub, but is not possible just to block it from accessing that specific repo?