Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: expose optimize_indices method in both Python and Node APIs #1463

Closed
eddyxu opened this issue Jul 22, 2024 · 2 comments
Closed

Feature: expose optimize_indices method in both Python and Node APIs #1463

eddyxu opened this issue Jul 22, 2024 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@eddyxu
Copy link
Contributor

eddyxu commented Jul 22, 2024

SDK

Python, Node

Description

The API to incrementally update indices is missing for Python sync API, and possibly others. This makes it hard to keep indices up-to-date.

We also don't have a user guide discussing this. We should demonstrate how to incrementally update indices, and when users should do that versus full re-indexing.

https://lancedb.github.io/lance/api/python/lance.html#lance.dataset.DatasetOptimizer.optimize_indices

And add docs to use this functionality.

@eddyxu eddyxu added the enhancement New feature or request label Jul 22, 2024
@eddyxu eddyxu changed the title Feature: expose optimize_indices method in both Python and VectorDB API Feature: expose optimize_indices method in both Python and Node APIs Jul 22, 2024
@universalmind303
Copy link
Contributor

we do have table.optimize() which optimizes everything..

instead of exposing a new method, what if we exposed some options on there to only optimize the indices, skipping compaction and pruning

@wjones127 wjones127 added this to the LanceDB Papercuts milestone Sep 30, 2024
@QianZhu
Copy link
Contributor

QianZhu commented Nov 15, 2024

we have added optimize_indices to python and node

@QianZhu QianZhu closed this as completed Nov 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants