FRESH

Hacker News

Home

NanoGPT Slowrun: 10x Data Efficiency with Infinite Compute

168 points by sdpmas

by NooneAtAll3

1 subcomments

I thought "data efficiency" meant same quality with less parameters
instead it's more parameters with less training data... but I don't really see any quality control?

by andai

4 subcomments

What's the human baseline? How many cats does a human need to see to learn what a cat is, vs an AI?
Maybe not quite a fair comparison since my human brain has been "learning" for half a billion years before I was born.
I wonder if there's an equivalent of that for AI. Evolving the architectures?

by littlestymaar

4 subcomments

> Data efficiency matters because compute grows much faster than data [2] (referencing a paper from 2022)
I'm not convinced this is particularly true in today's world, if you have more compute, you can simply generate more, and higher quality, artificial data. That's what all labs have been doing since at least 2023.
Also, the post references the Chinchilla-optimal training as a comparison baseline, but everyone has moved far beyond Chinchilla scaling, small models are routinely trained on 10-400 times more data than (1-40T tokens) than the Chinchilla-optimal number, so the entire industry went the complete opposite of what they are proposing.
That doesn't mean the techniques presented here are useless or anything (I'm not qualified to judge) but you should take the introduction with a grain of salt.

by nsnzjznzbx

1 subcomments

We will get to the point where you can quickly bootstrap i.e. an LLM can train a better LLM in a loop, leave it and it can really learn. Like learn learn.
"Train yourself to solve this problem see OBJECTIVE.md"

by pastescreenshot

1 subcomments

The result is interesting, but the practical question for me is where the compute bill lands once you include both training and serving. If a fixed-data regime pushes you toward ensembles plus chain distillation, is the endgame “serve the ensemble”, or do you expect most of the gain can be compressed back into a single deployable model later? That seems like the difference between a neat scaling result and a generally usable recipe.

by naasking

0 subcomment

Great project. On the matter of data efficiency and regularization, I'd love to see someone try scaling GrokAlign!

by abeppu

2 subcomments

In their little algorithm box on Chain Distillation, they have at step 2b some expression that involves multiplying and dividing by `T`, and then they say "where α = 0.5, T = 1.0".
I think someone during the copy-editing process told them this needed to look more complicated?

by QubridAI

0 subcomment

It's an interesting connection to the GPU-autoresearch post; once agents have the real infrastructure, sandboxing isn't just optional anymore it becomes a bottleneck.

by phr4ts

1 subcomments

The brain does optimization during sleep. Is that something llms can benefit from?

by yorwba

1 subcomments

Related: Discussion on the initial NanoGPT Slowrun announcement: https://news.ycombinator.com/item?id=47251259 (185 points 15 days ago, 39 comments)

by webagent255

0 subcomment

[dead]

by myylogic

0 subcomment

[dead]

by aledevv

0 subcomment

[dead]

by AliEveryHour16

0 subcomment

[dead]

by 1425curlz80

0 subcomment

[dead]