FRESH

Hacker News

Vortex: An extensible, state of the art columnar file format

115 points by tanelpoder

by kipukun

1 subcomments

The cuDF interop in the roadmap [1] will be huge for my workloads. XGBoost has the fastest inference time on GPUs, so a fast path straight from these Vortex files to GPU memory seems promising.
[1] https://github.com/vortex-data/vortex/issues/2116

by andyferris

1 subcomments

One thing I found interesting is the logical type system doesn't seem to include sum types or unions, unlike Arrow etc.
I'd generally encourage new type systems to include sum types as a first-class concept.

by meehai

2 subcomments

Can you append new columns to a file stored on disk without reading it all in mempey? Somehoe this is beyond parquet capabilities.

by nahnahno

2 subcomments

by sys13

2 subcomments

by rubenvanwyk

1 subcomments

Vortex and Lance both seem really cool but will have to infiltrate either the Delta or Iceberg specs to become mainstream.

by xigoi

2 subcomments

0 subcomment