FRESH

Hacker News

Model Training as Code

179 points by peterBlue75

by delichon

3 subcomments

Some good stuff here from Dwarkesh around mashing up training and inference:
https://youtu.be/20p5-kQXF_Q?is=72ImTNxkOEKmOXQ9
He predicts this kind of model factory will become central to organizational learning and operations. Updating and upgrading the model stack becomes the core staff function.

by jimbo_joe

0 subcomment

The blog post nails the problem well: with distributed training handled by many teams the problem becomes organizational rather then tech-based. Having a technology solution to more easily persist pipeline and artifact configs is a good idea, but it only yields an incremental improvement. IMO this can only solved with a shared culture around experimentation which has to be maintained and reinvented as the company (and hence the teams responsible for training) grows. Basically, if every layer of the stack from tooling to training to applied are aiming for repeatable training with predictable capability improvements that end users can see, then it's more likely to happen.

by mschwaig

1 subcomments

I've been thinking if this could be done in Nix and toyed around with the idea just a little bit using nano-gpt.
Hermeticity always seems to mean isolation, but depending on who you ask it does not always mean computing some sort of hash over all build input as the 'identity' of a particular step in the pipeline, like Nix does.
If you do that hash-based identity part, looking up intermediary results and resuming from there happens using this sort of hash.
Does savanah do that, or will it resume where it left off based on a less strict notion of identity?
I could see arguments for either approach.

by SpyCoder77

2 subcomments

What is this "aleph" thing in names now? First aleph neuro, and now aleph alpha.

by random3

2 subcomments

> TL;DR: Model training has grown complex
So they’ve built Savanah - a workflow engine because the existing zoo of hundreds of workflow engines didn’t cut it :)

by itsdesmond

1 subcomments

I speak from ignorance, having not worked on this type of thing: are people NOT doing this? Are folks… what, passing CSVs and clicking GUIs to kick off one-off expensive, long running training runs? That’s absolutely crazy to me.

by aubanel

0 subcomment

This is actually a good idea! And still looking forward for aleph alpha to release new models, after the Previa ones a while ago!

by virajk_31

0 subcomment

by michaelchicory

0 subcomment

by modgate

0 subcomment