- Kudos, I think (in the short term at least) there is a large amount of perf. optimization to be found by coding parts of the whole AI/ML infrastructure in C++ like this one, not as a rewrite (god no!) but drop in and fix key bottlenecks. Anytime I see someone (seems Chinese engineers are good at this) put something out in C++, good chance some solid engineering tradeoffs have been made and dramatic improvement will be seen.
by chrismustcode
1 subcomments
- There’s something beautiful about creating a drop in replacement for something that improves performance substantially.
ScyllaDB comes to mind
- Cool. Would it be possible to eliminate that little vocab format conversion requirement for the vocab I see in the test against tiktoken? It would be nice to have a fully compatible drop in replacement without having to think about details. It also would be nice to have examples that work the other way around: initialize tiktoken as you normally would, including any specialized extension of standard tokenizers, and then use that initialized tokenizer to initialize a new tokendagger and test identity of results.
by superlopuh
4 subcomments
- Can someone familiar with performance of LLMs please tell me how important this is to the overall perf? I'm interested in looking into optimizing tokenizers, and have not yet run the measurements. I would have assumed that the cost is generally dominated by matmuls but am encouraged by the reception of this post in the comments.
- Nice work! I tried something similar a while back ago: https://github.com/kevmo314/tokie
The takeaway I also found was that the running cost was really dominated by pretokenization (the regex). It's cool to see that you found a faster way to run the regex, but have you tried comparing the performance of just swapping out the regex engine and leaving the actual BPE to tiktoken? I wonder if that is upstreamable?
- Can you also compare the performance with https://github.com/huggingface/tokenizers/? Would be helpful, since the benchmark in the tiktoken readme seems to be very outdated.
- How does this compare to the BPE crate [1]? Its main selling point is support for incrementally re-tokenising text, but it's also faster than tiktoken.
[1] https://crates.io/crates/bpe
- Is there any way we can get local tokenizers for other LLMs? e.g. Gemini only offer a remote API for their tokenizer. Is it proprietary? Could we infer the token mapping somehow efficiently by making lots of calls?
- I've been playing with tokenization too. Starting from Kaparthy's Python minbpe I set myself the task of training a tokenizer on wikitext (500mb) in a reasonable time.
I got the C++ version down to about 50 minutes compared to the original Python code (estimated) several months.
Haven't really spent much time
looking at encode and decode but I plan to incorporate these regex modifications when I do!
https://github.com/justinhj/minbpe-cc
by fkyoureadthedoc
0 subcomment
- Would be cool to see WASM bindings for this here https://github.com/dqbd/tiktoken
Or maybe even your speedups from "b" in the pure js implementation
by konsalexee
1 subcomments
- > simplifying the algorithm to forego regex matching special tokens at all
Does that mean there could be cases with less quality in terms of tokenization?
by pamelafox
1 subcomments
- Just curious whether it's possible to push any of your performance improvements to tiktoken itself?
by polynomial
0 subcomment
- Just to note that Tiktoken is still the tokenizer behind the GPT-4x series, it just uses a different token model. (Post only says GPT-3, implying they were using something else for subsequent iterations.)
by manishsharan
0 subcomment
- Is there a tokenizer someone can recommend for code ? I have tried CodeBert but maybe I am using it wrong as my results with it were pretty bad.
by isjustintime
0 subcomment
- Very cool. We use Tiktoken and I'd love to see the performance impact. Pretty great decision to make it drop-in compatible.
by matrix2596
1 subcomments
- is is possible for your tokenizer to give different tokenization ever then openai tokenizer? i am asking because there are multiple ways to tokenize the same string?? sry if i am mistaken
by semiinfinitely
1 subcomments
- I'm relieved to see that its not written in rust
by singularity2001
0 subcomment
- this is still the outdated architecture without special tokens for numbers like out-of-vocab tokens like NUM_FLOAT(3.1415) right?
- What about pairing this with BigBird and Mamba?
- Now that byte-patch-level embeddings are discovered?
by silentsea90
2 subcomments
- "I’m teaching myself LLM internals by re-implementing the stack from first principles." - curious what resources you're using? Any books or courses, or just building it straight up? Great work!
- [dead]
by luppy47474
0 subcomment
- [flagged]
by janwilmake
2 subcomments
- You know what's also faster to roughly get the amount of tokens? string.length/5
Hacker News