PDX is a vertical data layout for vectors that store the dimensions of different vectors together. In a nutshell, PDX is a PAX for vector similarity search.

• ✂️ Efficient pruning of dimensions with partial distance calculations.
• ⚡ Up to 7x faster IVF queries (compared to FAISS+AVX512) when pairing PDX with the pruning algorithm ADSampling.
• ⚡ Up to 13x faster exhaustive search thanks to pruning.
• ⚡ Raw distance kernels (no pruning) in PDX are up to 1.6x faster than the float32 kernels available in SimSIMD (USearch) and FAISS.
  • Why? Distance kernels in PDX are free of dependencies and have fewer LOAD/STORE operations.
• Distance kernels can auto-vectorize efficiently without explicit SIMD for float32.
• Distance kernels on small vectors (d < 16) are up to 8x faster than SIMD kernels in SimSIMD.
• More efficient compressed representations of vectors (WIP).

• Pruning in a Nutshell
• Quickstart
• Use cases (comparison with FAISS)
• Roadmap

Pruning means avoiding checking all the dimensions of a vector to determine if it will make it onto the KNN of a query. Pruning speedup vector similarity search as (i) less data must be fetched and (ii) fewer computations must be done.

However, pruning methods that do partial distance calculations have a hard time being on par with SIMD-optimized kernels like the ones in FAISS and SimSIMD.

Only thanks to the PDX layout, do pruning methods outperform SIMD-optimized kernels in all CPU microarchitectures (Zens, Intels, Gravitons). This is because, in PDX, distance calculations are efficient even in small vector segments. Also, the evaluation of the pruning predicate is not interleaved with distance calculations.

Pruning algorithms are especially effective when:

• Vectors are of high dimensionality (d > 512)
• High recalls are needed (> 0.90)
• Exact results are needed
• k is relatively low (k=10,20)

Click here to see some quick benchmarks of our Python bindings vs FAISS. The complete benchmarks are available in our publication. Furthermore, you will find details on how we adapted these novel pruning algorithms to work in PDX.

We also refer to the recent research on pruning algorithms with partial distance calculations: ADSampling, DDC (previously named BSA), and DADE. All of these rely on rotating the vector collection to prune effectively. Alongside PDX, we also introduce PDX-BOND, a simpler pruning algorithm that does not need to rotate the vectors.

You can quickly try PDX with your data using our Python bindings and examples. We have implemented PDX on Flat IVF indexes and exhaustive search settings.

• Python 3.11 or higher
• FAISS with Python Bindings
• Clang++17 or higher
• CMake 3.26 or higher

1. Clone the repository and init submodules (Eigen for efficient matrix operations and pybind11)

git clone https://github.com/cwida/PDX
git submodule init
git submodule update

2. Install Python dependencies and pdxearch Python bindings.

export CXX="/usr/bin/clang++-18"  # Set proper CXX first
pip install -r requirements.txt
python setup.py clean --all
python -m pip install .

3. Run the examples under /examples

# Creates an IVF index with FAISS on random data
# Then, it compares the search performance of PDXearch and FAISS
python ./examples/pdxearch_simple.py

For more details on the available examples and how to use your own data, refer to /examples/README.md.

• We heavily rely on FAISS to create the underlying IVF indexes.
• PDX is an ongoing research project. In its current state, it is not production-quality code.

PDX paired with ADSampling on IVF indexes works great in most scenarios with less than 0.001 recall loss. The higher the dimensionality, the higher the gains from pruning. The following benchmarks are from /examples/pdxearch_ivf.py.

NOTE THAT on these benchmarks: (i) Both FAISS and PDXearch are scanning exactly the same vectors. (ii) The recall loss of ADSampling is always less than 0.001.

In PDX, building an IVF index can significantly improve exact search speed (thanks to the reliable pruning). The following benchmarks are from /examples/pdxearch_ivf_exhaustive.py.

Use PDX+BOND, our pruning algorithm. Here, vectors are not transformed, and we do not use any additional index. Gains vary depending on the dimensions distribution. The following benchmarks are from /examples/pdxearch_exact_bond.py.

PDX distance kernels are also faster than the state-of-the-art SIMD kernels in all major architectures, only relying on auto-vectorization (for float32). The following benchmarks are from /examples/pdx_brute.py.

Refer to our publication for the complete benchmarks of PDXearch done in 4 microarchitectures: Intel SPR, Zen 4, Zen 3, and Graviton 4. Here, you will also find a comparison against ADSampling in the horizontal layout (.fvecs layout). Furthermore, you will find details on how we adapted pruning algorithms to work in PDX.

• Compression: The vertical layout opens opportunities for compression as indexing algorithms group together vectors that share some numerical similarity within their dimensions. The next step on PDX is to apply our scalar quantization algorithm LEP, which uses database compression techniques (ALP) to compress vectors with higher compression ratios and less information loss.
• More data types: For compressed vectors, we need to implement vertical distance kernels on vectors of variable bit size.
• PDX in HNSW: For this, we need a layout similar to the ones proposed in Starling or AiSAQ, in which neighborhoods of the graph are stored and fetched in blocks.
• Improve code readability and usability.
• Add a testing framework
• Add DDC or DADE algorithms to the Python Bindings

To run our benchmark suite in C++, refer to BENCHMARKING.md.

The code used for the experiments presented at SIGMOD'25 can be found in the sigmod branch.