- It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards:
https://milkv.io
The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!
by helsinkiandrew
3 subcomments
- Voice-to-Text and Text-to-Voice models are approaching that size. I wonder how close we are to getting small devices that can chat with us.
Imagine a world where your toothbrush could give you tips about dental hygiene - or advertise toothpaste. What a time to be alive!
- This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!
by chrishynes
2 subcomments
- Why can't this scale to run much larger models on CPU backed by flash with good access patterns?
by spacedoutman
1 subcomments
- >esp32-s3
This microcontroller is a beast, currently using it to do dev work on a pi4.
Having two usb ports with one otg lets you do some neat things that would cost $100+ otherwise
by NooneAtAll3
1 subcomments
- While running LLM on tiny device is awesome, I'm more impressed by whatever training has produced the weights
by anonymous344
1 subcomments
- i don't care about microcontroller, what is the decent option to have local llm in my raspi4 that does not take 30 seconds to answer?
- Pretty incredible performance for the footprint - really interested to see what could be done on slightly more powerful SBCs like some that have been mentioned in this thread.
- This is a really cool project. Thanks for sharing!
- How accurate is this quantized model?
by cr125rider
0 subcomment
- 9.7 tokens/sec actually seems like a lot! That’s fun!
by ReactiveJelly
0 subcomment
- Finally, an LLM that can run on my i5
- Super cool, thank you for sharing!
- Wont this wear out the flash memory quickly? I wonder hiw many read cycles can it survive
- How lovely j
- [flagged]
- [dead]
by daflkfdslkfds
0 subcomment
- [dead]
Hacker News