LanceDB Rust SDK

LanceDB Rust SDK, a serverless vector database.

Read more at: https://lancedb.com/

[!TIP] A transitive dependency of lancedb is lzma-sys, which uses dynamic linking by default. If you want to statically link lzma-sys, you should activate it's static feature by adding the following to your dependencies:

lzma-sys = { version = "*", features = ["static"] }

lib.rs:

LanceDB is an open-source database for vector-search built with persistent storage, which greatly simplifies retrieval, filtering and management of embeddings.

The key features of LanceDB include:

• Production-scale vector search with no servers to manage.
• Store, query and filter vectors, metadata and multi-modal data (text, images, videos, point clouds, and more).
• Support for vector similarity search, full-text search and SQL.
• Native Rust, Python, Javascript/Typescript support.
• Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure.
• GPU support in building vector indices[^note].
• Ecosystem integrations with LangChain 🦜️🔗, LlamaIndex 🦙, Apache-Arrow, Pandas, Polars, DuckDB and more on the way.

[^note]: Only in Python SDK.

Getting Started

LanceDB runs in process, to use it in your Rust project, put the following in your Cargo.toml:

cargo add lancedb

Crate Features

• aws - Enable AWS S3 object store support.
• dynamodb - Enable DynamoDB manifest store support.
• azure - Enable Azure Blob Storage object store support.
• gcs - Enable Google Cloud Storage object store support.
• oss - Enable Alibaba Cloud OSS object store support.
• remote - Enable remote client to connect to LanceDB cloud.
• huggingface - Enable HuggingFace Hub integration for loading datasets from the Hub.
• fp16kernels - Enable FP16 kernels for faster vector search on CPU.

Quick Start

Connect to a database.

let db = lancedb::connect("data/sample-lancedb").execute().await.unwrap();

LanceDB accepts the different form of database path:

• /path/to/database - local database on file system.
• s3://bucket/path/to/database or gs://bucket/path/to/database - database on cloud object store
• db://dbname - Lance Cloud

You can also use ConnectBuilder to configure the connection to the database.

let db = lancedb::connect("data/sample-lancedb")
    .storage_options([
        ("aws_access_key_id", "some_key"),
        ("aws_secret_access_key", "some_secret"),
    ])
    .execute()
    .await
    .unwrap();

LanceDB uses arrow-rs to define schema, data types and array itself. It treats FixedSizeList<Float16/Float32> columns as vector columns.

For more details, please refer to the LanceDB documentation.

Create a table

To create a Table, you need to provide an arrow_array::RecordBatch. The schema of the RecordBatch determines the schema of the table.

Vector columns should be represented as FixedSizeList<Float16/Float32> data type.

use arrow_array::{RecordBatch, RecordBatchIterator};
use arrow_schema::{DataType, Field, Schema};

let ndims = 128;
let schema = Arc::new(Schema::new(vec![
    Field::new("id", DataType::Int32, false),
    Field::new(
        "vector",
        DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), ndims),
        true,
    ),
]));
let data = RecordBatch::try_new(
        schema.clone(),
        vec![
            Arc::new(Int32Array::from_iter_values(0..256)),
            Arc::new(
                FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(
                    (0..256).map(|_| Some(vec![Some(1.0); ndims as usize])),
                    ndims,
                ),
            ),
        ],
    )
    .unwrap();
db.create_table("my_table", data)
    .execute()
    .await
    .unwrap();

Create vector index (IVF_PQ)

LanceDB is capable to automatically create appropriate indices based on the data types of the columns. For example,

• If a column has a data type of FixedSizeList<Float16/Float32>, LanceDB will create a IVF-PQ vector index with default parameters.
• Otherwise, it creates a BTree index by default.

use lancedb::index::Index;
tbl.create_index(&["vector"], Index::Auto)
   .execute()
   .await
   .unwrap();

User can also specify the index type explicitly, see Table::create_index.

Open table and search

let results = table
    .query()
    .nearest_to(&[1.0; 128])?
    .execute()
    .await?
    .try_collect::<Vec<_>>()
    .await?;