FRESH

Hacker News

Qdf – a Go serializer you query the raw bytes of, smaller than protobuf

9 points by alex602

by alex602

0 subcomment

Author here. qdf is a schemaless binary serializer for Go — encoding/json-style API, no schema/IDL/codegen. The idea is to compress across records (inline string/key interning + columnar per-column codecs), so on batch telemetry it's -24..-77% vs protobuf and both smaller and faster to DECODE than json/msgpack on the default tier.
The part I had most fun with: a columnar mode you can run WHERE-predicates over without fully decoding, and a lossy vector codec for AI embeddings that beats Google's TurboQuant at equal recall (-22% bytes).
Honest about tradeoffs — protobuf/flatbuffers win raw single-message decode, and msgpack can edge encode on high-cardinality data; it's alpha. Pure Go, zero deps. Built solo; happy to answer anything.