- How does this compare to Andrej Karpathy's microgpt (https://karpathy.github.io/2026/02/12/microgpt/) or minGPT (https://github.com/karpathy/minGPT)?
- Is there some documentation for this? The code is probably the simplest (Not So) Large Language Model implementation possible, but it is not straight forward to understand for developers not familiar with multi-head attention, ReLU FFN, LayerNorm and learned positional embeddings.
This projects shares similarities with Minix. Minix is still used at universities as an educational tool for teaching operating system design. Minix is the operating system that taught Linus Torvalds how to design (monolithic) operating systems. Similarly having students adding capabilities to GuppyLM is a good way to learn LLM design.
- https://bbycroft.net/llm has 3d Visualization of tiny example LLM layers that do a very good job at showing what is going on (https://news.ycombinator.com/item?id=38505211)
by ordinarily
0 subcomment
- It's genuinely a great introduction to LLMs. I built my own awhile ago based off Milton's Paradise Lost: https://www.wvrk.org/works/milton
by mudkipdev
1 subcomments
- This is probably a consequence of the training data being fully lowercase:
You> hello
Guppy> hi. did you bring micro pellets.
You> HELLO
Guppy> i don't know what it means but it's mine.
- This really makes me think if it would be feasible to make an llm trained exclusively on toki pona (https://en.wikipedia.org/wiki/Toki_Pona)
by neurworlds
0 subcomment
- Cool project. I'm working on something where multiple LLM agents share a world and interact with each other autonomously. One thing that surprised me is how much the "world" matters — same model, same prompt, but put it in a system with resource constraints, other agents, and persistent memory, the behavior changes dramatically. Made me realize we spend too much time optimizing the model and not enough thinking about the environment it operates in.
- I like the idea, just that the examples are reproduced from the training data set.
How does it handle unknown queries?
- Could it be possible to train LLM only through the chat messages without any other data or input?
If Guppy doesn't know regular expressions yet, could I teach it to it just by conversation? It's a fish so it wouldn't probably understand much about my blabbing, but would be interesting to give it a try.
Or is there some hard architectural limit in the current LLM's, that the training needs to be done offline and with fairly large training set.
by cbdevidal
1 subcomments
- > you're my favorite big shape. my mouth are happy when you're here.
Laughed loudly :-D
- This is a nice idea. A tiny implementation can be way more useful for learning than yet another wrapper around a big model, especially if it keeps the training loop and inference path small enough to read end to end.
by BiraIgnacio
0 subcomment
- Nice work and thanks for sharing it!
Now, I ask, have LLMs ben demystified to you? :D
I am still impressed how much (for the most part) trivial statistics and a lot of compute can do.
- Does this work by just training once with next token prediction? Want to understand better how it creates fluent sentences if anyone can provide insights.
by kaipereira
0 subcomment
- This is so cool! I'd love to see a write-up on how made it, and what you referenced because designing neural networks always feel like a maze ;)
- Wow that is such a cool idea! And honestly very much needed. LLMs seem to be this blackbox nobody understands. So I love every effort to make that whole thing less mysterious. I will definitely have a look at dabbling with this, may it not be a goldfish LLM :)
by brcmthrowaway
3 subcomments
- Why are there so many dead comments from new accounts?
by ankitsanghi
0 subcomment
- Love it! I think it's important to understand how the tools we use (and will only increasingly use) work under the hood.
- Hm, I can actually try the training on my GPU. One of the things I want to try next. Maybe a bit more complex than a fish :)
by CaseFlatline
1 subcomments
- I am trying to find how the synthetic data was created (looking through the repo) and didn't find it. Maybe I am missing it - Would love to see the prompts and process on that aspect of the training data generation!
- It's just so amazing that 5 years ago it would be extremely to build a conversational bot like this.
But right now people make it a hobby, and that thing can run on a laptop.
This is just so wild.
by Duplicake
1 subcomments
- I love this! Seems like it can't understand uppercase letters though
by bharat1010
0 subcomment
- This is such a smart way to demystify LLMs. I really like that GuppyLM makes the whole pipeline feel approachable..great work
by SilentM68
1 subcomments
- Would have been funny if it were called "DORY" due to memory recall issues of the fish vs LLMs similar recall issues :)
by drincanngao
1 subcomments
- I was going to suggest implementing RoPE to fix the context limit, but realized that would make it anatomically incorrect.
- how's it handle longer context or does it start hallucinating after like 2 sentences? curious what the ceiling is before the 9M params
by gnarlouse
1 subcomments
- I... wow, you made an LLM that can actually tell jokes?
- how did you generate the synthetic data?
by EmilioOldenziel
0 subcomment
- Building it yourself is always the best test if you really understand how it works.
by tombelieber
0 subcomment
- looking forward to try it, great job
- > A 9M model can't conditionally follow instructions
How many parameters would you need for that?
by ananandreas
0 subcomment
- Great and simple way to bridge the gap between LLMs and users coming in to the field!
by nobodyandproud
0 subcomment
- Thanks. Tinkering is how I learn and this is what I’ve been looking for.
by rclkrtrzckr
1 subcomments
- I could fork it and create TrumpLM. Not a big leap, I suppose.
- Forked. Very cool. I appreciate the simplicity and documentation.
- This is really great! I've been wanting to do something similar for a while.
by winter_blue
0 subcomment
- This is amazing work. Thank you.
by nullbyte808
1 subcomments
- Adorable! Maybe a personality that speaks in emojis?
- Is this a reference from the Bobiverse?
- Love it! Great idea for the dataset.
by gdzie-jest-sol
2 subcomments
- * How creating dataset? I download it but it is commpresed in binary format.
* How training. In cloud or in my own dev
* How creating a gguf
- I don't mean to be 'that guy', but after a quick review, this really feels like low-effort AI slop to me.
There is nothing wrong using AI tools to write code, but nothing here seems to have taken more than a generic 'write me a small LLM in PyTorch' prompt, or any specific human understanding.
The bar for what constitutes an engineering feat on HN seems to have shifted significantly.
by AndrewKemendo
1 subcomments
- I love these kinds of educational implementations.
I want to really praise the (unintentional?) nod to Nagel, by limiting capabilities to representation of a fish, the user is immediately able to understand the constraints. It can only talk like a fish cause it’s very simple
Especially compared to public models, thats a really simple correspondence to grok intuitively (small LLM > only as verbose as a fish, larger LLM > more verbose) so kudos to the author for making that simple and fun.
by Vektorceraptor
0 subcomment
- Haha, funny name :)
- Cool
- Neat!
- Tiny LLM is an oxymoron, just sayin.
- haha funny, but really cool project. why fish tho lol.
by hackerman70000
2 subcomments
- [flagged]
- [flagged]
by techpulselab
0 subcomment
- [dead]
by textai2026
0 subcomment
- [dead]
by Morpheus_Matrix
0 subcomment
- [flagged]
by adamsilvacons
0 subcomment
- [dead]
- [dead]
by solsafe_dev
0 subcomment
- [dead]
- [dead]
by ethanmacavoy
0 subcomment
- [flagged]
- [flagged]
by Alexzoofficial
0 subcomment
- [flagged]
- [dead]
by zephyrwhimsy
0 subcomment
- [dead]
by zephyrwhimsy
0 subcomment
- [dead]
by zephyrwhimsy
0 subcomment
- [dead]
- [dead]
by zephyrwhimsy
0 subcomment
- [dead]
by weiyong1024
0 subcomment
- [flagged]
by aesopturtle
0 subcomment
- [flagged]
by techpulselab
0 subcomment
- [dead]
by Morpheus_Matrix
0 subcomment
- [dead]
by george_belsky
0 subcomment
- [dead]
by solsafe_dev
0 subcomment
- [dead]
by aditya7303011
0 subcomment
- Did something similar last year
https://github.com/aditya699/EduMOE
by zhichuanxun
0 subcomment
- [dead]
by aditya7303011
0 subcomment
- [dead]
by LeonTing1010
1 subcomments
- [flagged]
- [flagged]
by jiusanzhou
2 subcomments
- [flagged]
by dinkumthinkum
0 subcomment
- I think this is a nice project because it is end to end and serves its goal well. Good job! It's a good example how someone might do something similar for a specific purpose. There are other visualizers that explain different aspects of LLMs but this is a good applied example.
- How much training data did you end up needing for the fish personality to feel coherent? Curious what the minimum viable dataset looks like for something like this.
by Propelloni
1 subcomments
- Great work! I still think that [1] does a better job of helping us understand how GPT and LLM work, but yours is funnier.
Then, some criticism. I probably don't get it, but I think the HN headline does your project a disservice. Your project does not demystify anything (see below) and it diverges from your project's claim, too. Furthermore, I think you claim too much on your github. "This project exists to show that training your own language model is not magic." and then just posts a few command line statements to execute. Yeah, running a mail server is not magic, just apt-get install exim4. So, code. Looking at train_guppylm.ipynb and, oh, it's PyTorch again. I'm better off reading [2] if I'm looking into that (I know, it is a published book, but I maintain my point).
So, in short, it does not help the initiated or the uninitiated. For the initiated it needs more detail for it to be useful, the uninitiated more context for it to be understood. Still a fun project, even if oversold.
[1] https://spreadsheets-are-all-you-need.ai/
[2] https://github.com/rasbt/LLMs-from-scratch
Hacker News