FRESH

Hacker News

LLMs Are Complicated Now

205 points by matt_d

by jordanb

5 subcomments

It's the bitter-lesson to feature-engineering lifecycle.
When a technique or technology is new people are making massive gains by just applying it to some use case, or gathering more data for training, or giving it more resources.
As time goes on those "bitter lesson" gains start to hit the shallow part of the logistic curve and companies have to start investing more and more effort into engineering for each small, incremental gain.

by vivzkestrel

2 subcomments

by physix

1 subcomments

For someone like me who's never done any hands-on work in ML, the blog is really hard to understand. Whoosh, over my head.
But, I think the underlying problem is that we don't understand how this sh*t works. So, it's just an empirical, iterative mess.
Like physics in the the years shortly before relativity and quantum mechanics.

by aesthesia

0 subcomment

by charcircuit

5 subcomments

Why didn't this author compare Llama 3 with GLM 5.2 (released 1 week ago) which is a more standard attention based LLM? To compare 2 separate families of LLMs and then pointing out that they are different is not a surprising result and detracts from the point the author is trying to make.
https://sebastianraschka.com/llm-architecture-gallery/?compa...
If you look at it, the diagrams are very similar, but the main differences are that the feedforward is replaced with a MoE (router to multiple feedforwards) and the model has a different attention implementation.

0 subcomment

by truvem

2 subcomments

by cold_harbor

0 subcomment