FRESH

Hacker News

Home

Sub-Millisecond RAG on Apple Silicon. No Server. No API. One File

126 points by ckarani

by peterloron

5 subcomments

Looks cool! Thoughts on exposing this through a cli or mcp for local knowledge access for agents? For example, I use Claude Code for research and I have a local corpus of PDFs that I would like to make available as additional domain-specific information that Claude can use in addition to what it has in Opus or whatever model I'm using.

by threecheese

1 subcomments

Tell us more. I had Codex port this to Python so I could wrap my head around it, it’s quite interesting. Why would I use this WAL-check pointing thingamajig when I have access to SQLite-vec, qdrant and other embedded friends?

by mshekow

0 subcomment

I assume that wax uses Apple's ANE to do embeddings (so no third-party services like OpenAI are needed). Did you happen to compare search quality when using ANE embeddings vs. OpenAI's text-embedding-3-large (or another commonly used online embedding)?

by simlevesque

2 subcomments

Why is the 9.2ms bar longer than the 104ms bar in the performance graph ?

by ckarani

3 subcomments

I built Wax because every RAG solution required either Pinecone/Weaviate in the cloud or ChromaDB/Qdrant running locally. I wanted the SQLite of RAG -- import a library, open a file, query. Except for multimodal content at GPU speed.
The architecture that makes this work: Metal-accelerated vector search -- Embeddings live directly in unified memory (MTLBuffer). Zero CPU-GPU copy overhead. Adaptive SIMD4/SIMD8 kernels + GPU-side bitonic sort = sub-millisecond search on 10K+ vectors (vs ~100ms CPU). This isn't just "faster" -- it enables interactive search UX that wasn't possible before.
Atomic single-file storage (.mv2s) -- Everything in one crash-safe binary: embeddings, BM25 index, metadata, compressed payloads. Dual-header writes with generation counters = kill -9 safe. Sync via iCloud, email it, commit to git. The file format is deterministic -- identical input produces byte-identical output.
Query-adaptive hybrid fusion -- Four parallel search lanes (BM25, vector, timeline, structured memory). Lightweight classifier detects intent ("when did I..." → boost timeline, "find documentation about..." → boost BM25). Reciprocal Rank Fusion with deterministic tie-breaking = identical queries always return identical results.
Photo/Video RAG -- Index your photo library with OCR, captions, GPS binning, per-region embeddings. Query "find that receipt from the restaurant" searches text, visual similarity, and location simultaneously. Videos get segmented with keyframe embeddings + transcript mapping. Results include timecodes for jump-to-moment navigation. All offline -- iCloud-only photos get metadata-only indexing. Swift 6.2 strict concurrency -- Every orchestrator is an actor. Thread safety proven at compile time, not runtime. Zero data races, zero @unchecked Sendable, zero escape hatches.
Deterministic context assembly -- Same query + same data = byte-identical context every time. Three-tier surrogate compression (full/gist/micro) adapts based on memory age. Bundled cl100k_base tokenizer = no network, no nondeterminism.
import Wax
let brain = try await MemoryOrchestrator(at: URL(fileURLWithPath: "brain.mv2s"))
// Index try await brain.remember("User prefers dark mode, gets headaches from bright screens")
// Retrieve let context = try await brain.recall(query: "user display preferences") // Returns relevant memories with source attribution, ready for LLM context
What makes this different:
Zero dependencies on cloud infrastructure -- No API keys, no vendor lock-in, no telemetry Production-grade concurrency -- Not "it works in my tests," but compile-time proven thread safety Multimodal from the ground up -- Text, photos, videos indexed with shared semantics Performance that unlocks new UX -- Sub-millisecond latency enables real-time RAG workflows
## Wax Performance (Apple Silicon, as of Feb 17, 2026)
```
  - 0.84ms vector search at 10K docs (Metal, warm cache)
  - 9.2ms first-query after cold-open for vector search
  - ~125x faster than CPU (105ms) and ~178x faster than SQLite FTS5 (150ms) in
    the same 10K benchmark
  - 17ms cold-open → first query overall
  - 10K ingest in 7.756s (~1289 docs/s) with hybrid batched ingest
  - 0.103s hybrid search on 10K docs
  - Recall path: 0.101–0.103s (smoke/standard workloads)
```
Built for: Developers shipping AI-native apps who want RAG without the infrastructure overhead. Your data stays local, your users stay private, your app stays fast.
The storage format and search pipeline are stable. The API surface is early but functional. If you're building RAG into Swift apps, I'd love your feedback.
GitHub: https://github.com/christopherkarani/Wax
Star it if you're tired of spinning up vector databases for what should be a library call.

by atonse

1 subcomments

Interesting. I feel this is something that Apple could've built as "Spotlight 2.0" right into macOS. Yet another missed opportunity.
How does this compare to qmd [1] by Tobi Lutke?
[1] https://github.com/tobi/qmd

by znnajdla

1 subcomments

Great idea. But having only an SDK in Swift makes it extremely restricted. If it had a CLI interface I could start using it right away. And it only works on macOS?

by ElFitz

0 subcomment

This is absolutely wonderful.
Can’t wait to try it on iOS if the required APIs are available.

by kleton

2 subcomments

sqlite_vec is already the sqlite for AI memory

by burntalmonds

0 subcomment

Is this like zvec?

by saberience

1 subcomments

This looks rather over-engineered. I built something myself using SQLite in probably 10% of the code (or less) and queries were always running in the single digit milliseconds?
No offense intended but was this vibe coded? Have you tested and verified the code by hand?