vortex-data/vortex: An extensible, state of the art columnar file format. Formerly at @spiraldb, now an Incubation Stage project at LFAI&Data, part of the Linux Foundation.

Join the community on Slack! , 📚 Documentation , 📊Performance Benchmarks

Vortex is a next-generation column file format and toolkit designed for high-performance data processing. It is the fastest and most extensible format for creating data systems supported by object storage. It provides:

• ✓️ Very fast performance

  • 100x faster random access reads (vs modern Apache Parquet)
  • 10-20x faster scan
  • writes 5 times faster
  • equal compression ratio
  • Efficient support for extended tables with zero-copy/zero-parse metadata

• 🔧 Extensible Architecture

  • Built on the extensible approach of Apache DataFusion
  • Pluggable encoding system, type system, compression strategy, and layout strategy
  • Zero-copy compatibility with Apache Arrow

• 🗳️ Open source, neutral governance

  • A Linux Foundation (LF AI and Data) project
  • Apache-2.0 Licensed

• integration

  • Arrow, DataFusion, DuckDB, Spark, Pandas, Polar, and more
  • Apache Iceberg (coming soon)

development status: The library API may change from version to version, but for now we consider the file format. From release 0.36.0, all future releases of Vortex should maintain backwards compatibility of the file format (that is, should be able to read files written by any older version >= 0.36.0).

• logical type – Clear separation between logical schema and physical layout
• Zero-Copy Arrow Integration – seamless conversion to/from Apache Arrow arrays
• extensible encoding – Pluggable physical layout with built-in customization
• Cascading Compression – Support for nested encoding schemes
• high performance Computing – Optimized calculation kernel for encoded data
• rich data – Lazy-filled summary statistics for optimization

Vortex strictly separates logical and physical concerns:

• logical layer: defines data types and schema
• physical layer: Handles encoding and storage implementation
• implicit encoding: Compatible with Apache Arrow’s memory formats
• extension encoding:Customized compression schemes (RLE, dictionary, etc.)

All features are exported via main vortex Crate.

To browse the structure of vortex files, you can use vx Command-line tools.

# Install latest release
cargo install vortex-tui --locked

# Or build from source
cargo install --path vortex-tui --locked

# Usage
vx browse <file>

# Optional but recommended dependencies
brew install flatbuffers protobuf  # For .fbs and .proto files
brew install duckdb               # For benchmarks

# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# or
brew install rustup

# Initialize submodules
git submodule update --init --recursive

# Setup dependencies with uv
uv sync --all-packages

For optimal performance, we suggest using MiMalloc:

#[global_allocator]
static GLOBAL_ALLOC: MiMalloc = MiMalloc;

Licensed under the Apache License, version 2.0.

Vortex is an independent open-source project and is not controlled by any one company. The Vortex Project is a sub-project of the Linux Foundation Projects. The governance model is documented in CONTRIBUTING.md and is subject to the terms of the Technical Charter.

See CONTRIBUTING.md for guidelines.

If you find any security vulnerabilities, please email wln-report@vortex.dev.

Copyright © A Series of Vortex LF Projects, LLC. Please visit https://lfprojects.org for terms of use, trademark policy, and other project policies

The Vertex project has benefited greatly from the groundbreaking work of the academic and open-source communities.

• Anyblox – A framework for self-decoding datasets
• F3 – open-source data file format for the future