FRESH

Hacker News

Ways to think about token pricing

43 points by mercutio2

by typ

1 subcomments

Tokens will surely become commodities, but models don't need to.
The point is that electricity becomes a commodity, rather than, say, nuclear reactors or gas turbines also need to be commoditized.
Contrary to most people in the software-minded circle, I am not very excited about running local LLMs. Most of what I have seen involves selling wrappers that call APIs on top of an LLM AI, similar to how traditional SaaS has generated revenue from new technologies. So, this would certainly make those people who are working on that sensitive about the pricing. LLM pricing is treated as a continuous cost in the COGS. They don't really use AI to create much value (consumer surplus) other than replacing the existing boring products, like just building the good old CRUD apps, but faster.
What I find REAL interesting about the potential of LLM AI instead is to create new technology out of it, or revolutionize something old to be order-of-magnitude more efficient. In this regard, the expenses on those tokens are more akin to an upfront CapEx.
Cheaper tokens would surely be nice. But if what we are talking about is, like, solving self-driving, curing cancer, or making air-conditioners 100x efficient, the narrow focus on running a cheaper model in my home so I can write my SaaS apps for my get-rich-quick business looks really unpromising and a waste of time for human civilization in a near-singularity horizon.

by t0mpr1c3

0 subcomment

I agree with the author's overall conclusion about pricing LLMs, although I question some of the reasoning. He does not talk enough about one important input: the data used to train the models. To a first approximation, their utility is a function of the quantity of data available. Currently, the bulk of the training data is public (the internet) and that is a major reason why the performance of different models is not diverging more. One huge future area for AI is modeling the physical world and in that arena people are going to be creating their own empirical datasets. Waymo has achieved FSD and Tesla has not. It is not coincidental that Waymo has built a sophisticated platform to simulate the physical world.

by mips_avatar

3 subcomments

I know there's a lot of reasons to think that everyone will just use AI inference in the cloud, but I think if everyone had access to a dgx gb400 class machine with 512gb of hbm4 vram and 1tb of lpddr8x a lot of people are going to be running finetunes of models locally. Like the dgx gb300 is $94k now, but i bet this class of machine will come down to $20kish in the next 2-3 years.

by nekusar

1 subcomments

1. Slot machine
You put in minutes of time combined with token costs. You pull the lever and HOPE something good comes back.
And it gets tantalizingly close, but for 100% gotta pull the arm again!

by no7z

0 subcomment