docs: add snippet on incremental reindexing #1461
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
ACTION NEEDED Lance follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. For details on the error please inspect the "PR Title Check" action. |
| @@ -55,7 +55,7 @@ When a reindex job is triggered in the background, the entire data is reindexed, | |||
| ### Vector reindex | |||
|
|
|||
| * LanceDB Cloud supports incremental reindexing, where a background process will trigger a new index build for you automatically when new data is added to a dataset | |||
| * LanceDB OSS requires you to manually trigger a reindex operation -- we are working on adding incremental reindexing to LanceDB OSS as well | |||
| * LanceDB OSS requires you to manually trigger a reindex operation -- incremental indexing is available via the Lance API `lance_table.to_lance().optimize.optimize_indices()`. Incremental indexing means that any unindexed rows are added to the existing index. This is much faster than a full reindex because it does not involve kmeans training or reconstructing the graph from scratch (depending on your index type). | |||
There was a problem hiding this comment.
should we mention that this is only available via python? Rust and JS do not have a to_lance equivalent.
There was a problem hiding this comment.
Can we do #1463 instead ?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
User reported that our documentation says incremental indexing is only available in the commercial offering. Updating the docs to add guidance on how to use lance api to do incremental indexing