- It is a bug in MLX that has been fixed a few days ago:
https://github.com/ml-explore/mlx/pull/3083
- Methodology is one thing; I can't really agree that deploying an LLM to do sums is great. Almost as hilarious as asking "What's moon plus sun?"
But phenomenon is another thing. Apple's numerical APIs are producing inconsistent results on a minority of devices. This is something worth Apple's attention.
- Low level numerical operation optimizations are often not reproduceable. For example: https://www.intel.com/content/dam/develop/external/us/en/doc... (2013)
But it's still surprising that that LLM doesn't work on iPhone 16 at all. After all LLMs are known for their tolerance to quantization.
by DustinEchoes
3 subcomments
- I wish he would have tried on a different iPhone 16 Pro Max to see if the defect was specific to that individual device.
- The author is assuming Metal is compiled to ANE in MLX. MLX is by-and-large GPU-based and not utilizing ANE, barring some community hacks.
by Buttons840
8 subcomments
- I clicked hoping this would be about how old graphing calculators are generally better math companions than a phone.
The best way to do math on my phone I know of is the HP Prime emulator.
- Does it bother anyone else that the author drops "MiniMax" there in the article without bothering to explain or footnote what that is? (I could look it up, but I think article authors should call out these things).
by johngossman
0 subcomment
- Posting some code that reproduces the bug could help not only Apple but you and others.
- Maybe this is why my damn keyboard predictive text is so gloriously broken
- Interesting post, but the last bit of logic pointing to the Neural Engine for MLX doesn’t hold up. MLX supports running on CPU, Apple GPU via Metal, and NVIDIA GPU via CUDA: https://github.com/ml-explore/mlx/tree/main/mlx/backend
by Metacelsus
1 subcomments
- >"What is 2+2?" apparently "Applied.....*_dAK[...]" according to my iPhone
At least the machine didn't say it was seven!
- Good article. Would have liked to see them create a minimal test case, to conclusively show that the results of math operations are actually incorrect.
- I love to see real debugging instead of conspiracy theories!
Did you file a radar? (silently laughing while writing this, but maybe there's someone left at Apple who reads those)
- I'd think other neural-engine using apps would also have weird behavior. Would've been interesting to try a few App Store apps and see the weird behavior
- What expense app are you building? I really want an app that helps me categorize transactions for budgeting purposes. Any recommendations?
- The real lesson here isn't even about Apple. It's about debugging culture
- I also would like to see if the same error happens in another phone with the exactly same model.
- So the LLM is working as intended?
by MORPHOICES
0 subcomment
- [dead]
by MarginalGainz
0 subcomment
- [dead]
by builderhq_io
0 subcomment
- [dead]
- [flagged]
- [flagged]
- [flagged]
- [flagged]
by giancarlostoro
4 subcomments
- [flagged]
by RiceNBananas
0 subcomment
- [flagged]
by vanviegen
1 subcomments
- [flagged]
by PlatoIsADisease
4 subcomments
- [flagged]
by refulgentis
2 subcomments
- .
- neural nets or AI are very bad at math, it can only produce what's in the training data. So if you have trained it from 1+1 to 8+8 it can't do 9+9, it's not like a child brain that it can make logical conclusions.
- My thousand dollar iPhone can't even add a contact from a business card.
- > Update on Feb. 1st:
> Well, now it's Feb. 1st and I have an iPhone 17 Pro Max to test with and... everything works as expected. So it's pretty safe to say that THAT specific instance of iPhone 16 Pro Max was hardware-defective.
nothing to see here.
Hacker News