FRESH

Hacker News

Faster embeddings: how we rebuilt the ONNX path in Manticore

79 points by snikolaev

by ducviet00

5 subcomments

Unlike GPUs, CPUs aren't designed for massive parallelism. Because of this, batching inference won't necessarily give you a speed boost here. In fact, it can actually slow the process down.
Instead, I'd recommend exploring CPU-specific AI optimizations. For instance, leveraging AVX512_BF16 instructions could reduce the inference time by 2x or 3x compared to the results in the article. OpenVINO supports this really well on Intel CPUs, and converting an ONNX model to OpenVINO is straightforward.

by minimaxir

2 subcomments

We really need a replacement for all-MiniLM-L12-v2 that can create more robust embeddings with the same compute.
You can technically do Q4 quantization for larger embedding models but I am not sure if that plays nice with ONNX.

by electroglyph

0 subcomment

by imtringued

1 subcomments

Spinlocks are basically the heroin of parallel programming. So many people are addicted to them in the pursuit of performance but the truth is that in 99% of all cases, they are a terrible idea.
Something people don't get about spinlocks is that you're basically saying you own the entire CPU core. In any other situation where the core is shared by multiple processes, it is inherently illogical to use a spinlock.

by aivisibility96

0 subcomment