by MrArthegor
9 subcomments
- A good technical project, but honestly useless in like 90% of scenarios.
You want to use an NVidia GPU for LLM ? just buy a basic PC on second hand (the GPU is the primary cost anyway), you want to use Mac for good amount of VRAM ? Buy a Mac.
With this proposed solution you have an half-backed system, the GPU is limited by the Thunderbolt port and you don’t have access to all of NVidia tool and library, and on other hand you have a system who doesn’t have the integration of native solution like MLX and a risk of breakage in future macOS update.
by bangonkeyboard
6 subcomments
- I don't know how Apple has evaded regulatory scrutiny for their refusal to sign Nvidia's eGPU drivers since 2018.
- From what I understand, only works with Tinygrad. Which is better than nothing but CUDA or Vulkan on pytorch isn’t going to work from this.
[1] https://docs.tinygrad.org/tinygpu/
- Such a shame both companies are big on vanity to make great things happen. Imagine where you could run Mac hardware with nvidia on linux. It's all there, and closed walls are what's not allowing it to happen. That's what we as customers lose when we forego control of what we purchase to those that sold us the goods.
- Woah, this is exciting. I'm traveling but I have a 5090 lying around at home. I'm eager to give it a go. Docs are here: https://docs.tinygrad.org/tinygpu/
I hope it'll work on an M4 Mac Mini. Does anyone know what hardware to get? You'll need a full ATX PSU to supply power, right? And then tinygrad can do LLM inference on it?
by mlfreeman
3 subcomments
- I followed the instructions link and read the scripts...although the TinyGPU app is not in source form on GitHub, this looks to me like the GPU is passed into the Linux VM underneath to use the real driver and then somehow passed back out to the Mac (which might be what the TinyGrad team actually got approved).
Or I could have totally misunderstood the role of Docker in this.
- I think that metal isn’t double precision; so that limits some serious physics simming; but if you’re doing that I guess you just rent a gpu somewhere.
I would definitely be into this if adding an egpu was first class supported.
- Interesting, but cannot run CUDA or more to the point `nvidia-smi`.
- Pretty misleading. This driver is only for compute not graphics.
by EagnaIonat
1 subcomments
- > If you have a Thunderbolt or USB4 eGPU and a Mac, today is the day you've been waiting for!
I got an eGPU back in 2018 and could never get it to work. To the point that it soured me from doing it again.
These days for heavy duty work I just offload to the cloud. This all feels like NVidia trying to be relevant versus ARM.
- If you could get Nvidia driver support on Mac’s I bet Apple would have sold more MacPro’s.
by the__alchemist
1 subcomments
- I'm writing scientific software that has components (molecular dynamics) that are much faster on GPU. I'm using CUDA only, as it's the eaisiest to code for. I'd assumed this meant no-go on ARM Macs. Does this news make that false?
- Apple should update this page for ARM macs, now runs tinygrad on eGPUs: https://support.apple.com/en-us/102363
by brcmthrowaway
3 subcomments
- What are the limitations of USB4/Thunderbolt compared with a regular PCIe slot?
- These tinyboxes are so expensive (starting at $12,000), why don't they just put a CPU inside and allow users to ssh into them?
- My main thought is would this allow me to speed up prompt process for large MoE models? That is the real bottleneck for m3ultra. The tokens per second is pretty good.
by dd_xplore
5 subcomments
- Why does Apple need to make the drivers in a walled garden? Atleast they should support major device categories with official drivers.
by lowbloodsugar
0 subcomment
- Can I do prefill on the eGPU and the decode on the Mac?
by direwolf20
0 subcomment
- Isn't it sad that we've ended up in a situation where we are talking about "Apple approves" rather than "someone creates"? Fuck Apple.
by ErenalpCet
0 subcomment
- yes good report
- Idk why this doesn't link to the original source instead of this proxy source: https://x.com/__tinygrad__/status/2039213719155310736
- [dead]
- [dead]
- [flagged]
by userbinator
3 subcomments
- [flagged]
by tensor-fusion
2 subcomments
- [flagged]
- The opportunity cost of Apple refusing to sign Nvidia's OEM AArch64 drivers is probably reaching the trillion-dollar mark, now that Nvidia and ARM have their own server hardware.
Hacker News