GPU codec: fall back to CPU graph build on flush when GPU is busy by ChrisHegarty · Pull Request #149373 · elastic/elasticsearch

ChrisHegarty · 2026-05-19T09:37:10Z

During heavy indexing, multiple flush operations can compete for GPU resources simultaneously. Previously, flush would block waiting for a GPU resource to become available, which stalls the indexing thread and can cause cascading latency. This is particularly problematic when the GPU is already saturated with merge or other flush operations — the thread just sits idle waiting for its turn.

I've changed the flush path to use a non-blocking tryAcquire instead of a blocking acquire. If the GPU is busy (all resources locked or insufficient memory), flush now builds the HNSW graph on CPU using Lucene's HnswGraphBuilder. The resulting graph is written in the same Lucene99 format, so it's fully searchable by the standard reader. This means flush never blocks on GPU availability — it always makes progress, just potentially slower for that particular relatively small new segment.

To support this, I added tryAcquire to the CuVSResourceManager interface. Both acquire and tryAcquire now delegate to a shared doAcquire implementation with a nonBlocking flag, and I've added a reason parameter for diagnostics so we can see in logs which operation is acquiring or waiting for resources.

I've added tests at three levels: unit tests for the tryAcquire mechanics (including a concurrent contention test), a WriteGraphTests class that validates the CPU fallback produces byte-identical output to Lucene, and two mixed-path format tests that exercise both GPU and CPU paths within the same index on GPU nodes via a randomly-failing resource manager.

When flushing, use tryAcquire (non-blocking) to attempt GPU resource acquisition. If the GPU is busy, fall back to building the HNSW graph on CPU using HnswGraphBuilder. This avoids blocking flush threads waiting for GPU resources during heavy indexing. Also adds a `reason` parameter to acquire/tryAcquire for improved diagnostics, and refactors both methods to share a common doAcquire implementation.

…-flush

elasticsearchmachine · 2026-05-19T09:37:36Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

elasticsearchmachine · 2026-05-19T09:37:37Z

Hi @ChrisHegarty, I've created a changelog YAML for you.

github-actions · 2026-05-19T09:39:04Z

🔍 Preview links for changed docs

⏳ Building and deploying preview... View progress

This comment will be updated with preview links when the build is complete.

github-actions · 2026-05-19T09:40:55Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

mayya-sharipova · 2026-05-19T18:08:06Z

-                            );
-                            logger.error(message);
-                            throw new IllegalArgumentException(message);
+                            throw memoryExceededError(numVectors, dims, totalMemoryInBytes);


can we also fallback to CPU for this case: when there is not enough memory on GPU?

we can also leave it for a follow up

This should never be true now, right? If the memory exceeds what the GPU has, we switch algorithm.
I'm ok with the change as a safety net, in case we have data so large and a GPU so small even IVF-PQ cannot handle them, but it should be a follow-up

Right. It's an effective runtime assert. ( I do not want to make it an assert as I want it to always be surfaced, since it's an error )

mayya-sharipova

Thanks Chris for fixing this issue

ldematte

I like the idea. I think it's very smart to have this in flush, were data should be on the smaller side, but keep acquire on merge.

ldematte · 2026-05-20T06:45:24Z

-                            );
-                            logger.error(message);
-                            throw new IllegalArgumentException(message);
+                            throw memoryExceededError(numVectors, dims, totalMemoryInBytes);


This should never be true now, right? If the memory exceeds what the GPU has, we switch algorithm.
I'm ok with the change as a safety net, in case we have data so large and a GPU so small even IVF-PQ cannot handle them, but it should be a follow-up

ldematte · 2026-05-20T06:46:33Z

-                        if (numLockedResources() == 0) {
-                            logger.debug("No resources currently locked, proceeding");
+                        // If no resource in the pool is locked, we must proceed to avoid livelock
+                        if (enoughMemory == false && numLockedResources() == 0) {


Why has the enoughMemory check been moved here?

I added the enoughMemory == false check here so that when a resource is available and there is enough memory, then the regular call flow is followed - not this short-circuit. The regular acquire call flow, when enoughMemory is true, will later log the acquisition and update the memory service.

…49373) (#149475) During heavy indexing, multiple flush operations can compete for GPU resources simultaneously. Previously, flush would block waiting for a GPU resource to become available, which stalls the indexing thread and can cause cascading latency. This is particularly problematic when the GPU is already saturated with merge or other flush operations — the thread just sits idle waiting for its turn. I've changed the flush path to use a non-blocking tryAcquire instead of a blocking acquire. If the GPU is busy (all resources locked or insufficient memory), flush now builds the HNSW graph on CPU using Lucene's HnswGraphBuilder. The resulting graph is written in the same Lucene99 format, so it's fully searchable by the standard reader. This means flush never blocks on GPU availability — it always makes progress, just potentially slower for that particular relatively small new segment. To support this, I added tryAcquire to the CuVSResourceManager interface. Both acquire and tryAcquire now delegate to a shared doAcquire implementation with a nonBlocking flag, and I've added a reason parameter for diagnostics so we can see in logs which operation is acquiring or waiting for resources. I've added tests at three levels: unit tests for the tryAcquire mechanics (including a concurrent contention test), a WriteGraphTests class that validates the CPU fallback produces byte-identical output to Lucene, and two mixed-path format tests that exercise both GPU and CPU paths within the same index on GPU nodes via a randomly-failing resource manager.

…sy (#149373) (#149476) * GPU codec: fall back to CPU graph build on flush when GPU is busy (#149373) During heavy indexing, multiple flush operations can compete for GPU resources simultaneously. Previously, flush would block waiting for a GPU resource to become available, which stalls the indexing thread and can cause cascading latency. This is particularly problematic when the GPU is already saturated with merge or other flush operations — the thread just sits idle waiting for its turn. I've changed the flush path to use a non-blocking tryAcquire instead of a blocking acquire. If the GPU is busy (all resources locked or insufficient memory), flush now builds the HNSW graph on CPU using Lucene's HnswGraphBuilder. The resulting graph is written in the same Lucene99 format, so it's fully searchable by the standard reader. This means flush never blocks on GPU availability — it always makes progress, just potentially slower for that particular relatively small new segment. To support this, I added tryAcquire to the CuVSResourceManager interface. Both acquire and tryAcquire now delegate to a shared doAcquire implementation with a nonBlocking flag, and I've added a reason parameter for diagnostics so we can see in logs which operation is acquiring or waiting for resources. I've added tests at three levels: unit tests for the tryAcquire mechanics (including a concurrent contention test), a WriteGraphTests class that validates the CPU fallback produces byte-identical output to Lucene, and two mixed-path format tests that exercise both GPU and CPU paths within the same index on GPU nodes via a randomly-failing resource manager. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix compilation for Lucene 10.3.2 Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>

ChrisHegarty added 2 commits May 19, 2026 09:22

Merge remote-tracking branch 'upstream/main' into gpu-cpu-fallback-on…

56a0d3c

…-flush

ChrisHegarty requested review from ldematte and mayya-sharipova May 19, 2026 09:37

ChrisHegarty added >bug :Search Relevance/Vectors Vector search Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch v9.5.0 v9.3.5 v9.4.2 labels May 19, 2026

Update docs/changelog/149373.yaml

f2269d8

ChrisHegarty added the test-gpu Run tests using a GPU label May 19, 2026

ChrisHegarty and others added 4 commits May 19, 2026 11:07

formating

15995ff

formatting

7d61145

formatting

8fe7f11

Merge branch 'main' into gpu-cpu-fallback-on-flush

2621dd4

mayya-sharipova reviewed May 19, 2026

View reviewed changes

Comment thread libs/gpu-codec/src/main/java/org/elasticsearch/gpu/codec/ES92GpuHnswVectorsWriter.java

mayya-sharipova reviewed May 19, 2026

View reviewed changes

Merge branch 'main' into gpu-cpu-fallback-on-flush

c91c777

mayya-sharipova reviewed May 19, 2026

View reviewed changes

Comment thread libs/gpu-codec/src/main/java/org/elasticsearch/gpu/codec/ES92GpuHnswVectorsWriter.java Outdated

mayya-sharipova approved these changes May 19, 2026

View reviewed changes

ldematte approved these changes May 20, 2026

View reviewed changes

ChrisHegarty and others added 2 commits May 20, 2026 08:35

use graph.getSortedNodes

4c7f038

Merge branch 'main' into gpu-cpu-fallback-on-flush

51933a8

ChrisHegarty merged commit 1ab5363 into elastic:main May 20, 2026
37 of 38 checks passed

ChrisHegarty added the auto-backport Automatically create backport pull requests when merged label May 20, 2026

This was referenced May 20, 2026

[9.4] GPU codec: fall back to CPU graph build on flush when GPU is busy (#149373) #149475

Merged

[9.3] GPU codec: fall back to CPU graph build on flush when GPU is busy (#149373) #149476

Merged

ChrisHegarty mentioned this pull request May 22, 2026

[CI] ES92GpuHnswWriteGraphTests testCpuFallbackGraphMatchesLucene failing #149659

Closed

Conversation

ChrisHegarty commented May 19, 2026

Uh oh!

elasticsearchmachine commented May 19, 2026

Uh oh!

elasticsearchmachine commented May 19, 2026

Uh oh!

github-actions Bot commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

github-actions Bot commented May 19, 2026

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

Uh oh!

mayya-sharipova May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mayya-sharipova May 19, 2026

Choose a reason for hiding this comment

Uh oh!

ldematte May 20, 2026

Choose a reason for hiding this comment

Uh oh!

ChrisHegarty May 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mayya-sharipova left a comment

Choose a reason for hiding this comment

Uh oh!

ldematte left a comment

Choose a reason for hiding this comment

Uh oh!

ldematte May 20, 2026

Choose a reason for hiding this comment

Uh oh!

ldematte May 20, 2026

Choose a reason for hiding this comment

Uh oh!

ChrisHegarty May 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented May 19, 2026 •

edited

Loading

mayya-sharipova May 19, 2026 •

edited

Loading