Add scalar quantization support for float16 vectors in the dedup KNN Codec by Pranshu-S · Pull Request #16710 · apache/lucene

Pranshu-S · 2026-09-25T05:57:16Z

Description

Addresses one of improvements tracked in #16595

The sandbox dedup KNN codec stores each distinct vector once and lets every document referencing it point to that single shared copy. Its scalar-quantized variant (DedupScalarQuantizedVectorsFormat) additionally stores a data-blind scalar-quantized copy of each distinct vector, so scoring runs on the compact quantized form while the raw vectors remain for full-fidelity readback and rescoring.

Previously, only FLOAT32 vectors were quantized — FLOAT16 (and BYTE) were stored raw only. This PR adds scalar-quantization support for FLOAT16 vectors, bringing them to parity with FLOAT32.

How

The data-blind scalar quantizer (OptimizedScalarQuantizer) computes centroids and corrective terms in fp32, and the JVM has no fp16 arithmetic type — so it currently has no way to quantize fp16 values natively. As a result, FLOAT16 vectors are inflated short[] → float[] before quantization. This inflation is lossless (every fp16 value is exactly representable in fp32), so an fp16 vector produces the same quantized record its fp32 equivalent would. This mirrors the core Lucene104ScalarQuantizedVectorsWriter.

Quantizing fp16 directly (without inflating) is a follow-up that first requires the data-blind OptimizedScalarQuantizer itself to support fp16 arithmetic; once that lands, this codec can drop the inflation step. That
prerequisite is tracked by #16533

…codec Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

kaivalnp

Thanks @Pranshu-S, this looks great at first glance -- but I'll try to take a deeper look soon!

Curious if you were able to run luceneutil KNN benchmarks for FLOAT16 vectors to test this?

Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

Pranshu-S · 2026-10-01T14:31:56Z

Sharing the benchmark runs - baseline here is only raw float16 and candidate here is quantized float16

ndoc: 10000, nquery1000

=====================================================================
COMPARISON  (candidate vs baseline;  delta/pct = candidate - baseline)
metrics are mean±stddev over iterations (baseline n=5, candidate n=5)
=====================================================================
metric             baseline(n=5)    candidate(n=5)   delta     pct
-----------------  ---------------  ---------------  --------  ------
recall             0.998±0.000      0.977±0.000      -0.021    -2.1%
latency(ms)        3.409±0.050      0.608±0.011      -2.801    -82.2%
netCPU             3.403±0.050      0.602±0.011      -2.801    -82.3%
avgCpuCount        0.998±0.000      0.990±0.000      -0.008    -0.8%
nDoc               10000            10000
searchType         KNN              KNN
topK               10               10
fanout             100              100
resultSimilarity   N/A              N/A
decay              N/A              N/A
resultCount        10.000           10.000
maxConn            32               32
beamWidth          200              200
quantized          7 bits           7 bits
visited            2247.600±4.159   2250.600±4.879   +3.000    +0.1%
index(s)           4.140±0.106      5.044±0.123      +0.904    +21.8%
index_docs/s       2416.156±62.095  1984.008±50.093  -432.148  -17.9%
merge(s)           0.000±0.000      0.000±0.000      +0.000
force_merge(s)     39.856±0.348     6.816±0.111      -33.040   -82.9%
num_segments       1.000±0.000      1.000±0.000      +0.000    +0.0%
index_size(MB)     20.188±0.004     30.102±0.004     +9.914    +49.1%
filterStrategy     null             null
filterSelectivity  N/A              N/A
overSample         1.000            1.000
vec_disk(MB)       29.449±0.000     29.449±0.000     +0.000    +0.0%
vec_RAM(MB)        9.918±0.000      9.918±0.000      +0.000    +0.0%
bp-reorder         false            false
indexType          HNSW             HNSW
rerank             no               no

ndoc100000, nquery10000

=====================================================================
COMPARISON  (candidate vs baseline;  delta/pct = candidate - baseline)
metrics are mean±stddev over iterations (baseline n=5, candidate n=5)
=====================================================================
metric             baseline(n=5)    candidate(n=5)   delta     pct
-----------------  ---------------  ---------------  --------  ------
recall             0.993±0.000      0.967±0.000      -0.026    -2.6%
latency(ms)        5.045±0.038      1.170±0.014      -3.875    -76.8%
netCPU             5.044±0.038      1.169±0.014      -3.875    -76.8%
avgCpuCount        1.000±0.000      0.999±0.000      -0.001    -0.1%
nDoc               100000           100000
searchType         KNN              KNN
topK               10               10
fanout             100              100
resultSimilarity   N/A              N/A
decay              N/A              N/A
resultCount        10.000           10.000
maxConn            32               32
beamWidth          200              200
quantized          7 bits           7 bits
visited            2984.400±2.966   2983.200±1.304   -1.200    -0.0%
index(s)           74.166±1.458     77.218±1.173     +3.052    +4.1%
index_docs/s       1348.744±26.648  1295.276±19.401  -53.468   -4.0%
merge(s)           647.172±1.826    82.688±0.860     -564.484  -87.2%
force_merge(s)     0.000±0.000      36.008±0.238     +36.008
num_segments       1.000±0.000      1.000±0.000      +0.000    +0.0%
index_size(MB)     204.712±0.008    303.892±0.004    +99.180   +48.4%
filterStrategy     null             null
filterSelectivity  N/A              N/A
overSample         1.000            1.000
vec_disk(MB)       294.495±0.000    294.495±0.000    +0.000    +0.0%
vec_RAM(MB)        99.182±0.000     99.182±0.000     +0.000    +0.0%
bp-reorder         false            false
indexType          HNSW             HNSW
rerank             no               no

kaivalnp · 2026-10-01T15:08:07Z

Thanks for the benchmarks @Pranshu-S!

Could you run them with filterStrategy = index-time-filter, to see whether de-duplication correctly occurs for FP16 vectors?

Could you also change the baseline to scalar quantized FP16 vectors using the "vanilla" (non-dedup) format? Looking to confirm whether recall is comparable b/w the two (which I think it will be, based on the above benchmark).

Might be worth checking all quantization levels too (8, 4, 2, 1-bit), for exercising the symmetric / asymmetric path as well.

Pranshu-S · 2026-10-01T15:47:25Z

Sure @kaivalnp, i can run these suites.

On the filterStrategy part, I'm not quite following how index-time-filter relates to FP16 dedup correctness here. As far as I can understand, this filterStrategy doesn't change how the dedup codec hashes or collapses vectors

kaivalnp · 2026-10-01T16:03:29Z

how index-time-filter relates to FP16 dedup correctness here

Using index-time-filter creates a separate field with a subset of vectors (proportion = filterSelectivity), which introduces a much higher count of duplicates than are naturally present in the dataset (which I think was <0.1% for the Cohere v3 vectors present in luceneutil).

Pranshu-S · 2026-10-01T16:26:01Z

I see, so with index-time-filter we are validating the gains with dedup compared to current setup with the expectation that recall should be same with improvements in index_size?

Looks to be pretty much within the expectations -
(Note - filter selectivity = 0.5)

==================================================================================
COMPARISON  (candidate=lucene_candidate+dedup vs baseline=lucene_baseline+non-dedup;
             delta/pct = candidate - baseline)
==================================================================================
metric             baseline/non-dedup(n=1)  candidate/dedup(n=1)  delta     pct
-----------------  -----------------------  --------------------  --------  ------
recall             0.976                    0.976                 +0.000    +0.0%
latency(ms)        0.566                    0.592                 +0.026    +4.6%
netCPU             0.559                    0.586                 +0.027    +4.8%
avgCpuCount        0.988                    0.990                 +0.002    +0.2%
nDoc               10000                    10000
searchType         KNN                      KNN
topK               10                       10
fanout             100                      100
resultSimilarity   N/A                      N/A
decay              N/A                      N/A
resultCount        10.000                   10.000
maxConn            32                       32
beamWidth          200                      200
quantized          7 bits                   7 bits
visited            2122.000                 2115.000              -7.000    -0.3%
index(s)           3.300                    3.090                 -0.210    -6.4%
index_docs/s       3028.470                 3234.150              +205.680  +6.8%
merge(s)           0.000                    0.000                 +0.000
force_merge(s)     11.160                   10.300                -0.860    -7.7%
num_segments       1.000                    1.000                 +0.000    +0.0%
index_size(MB)     74.230                   49.930                -24.300   -32.7%
filterStrategy     index-time-filter        index-time-filter
filterSelectivity  0.50                     0.50
overSample         1.000                    1.000
vec_disk(MB)       48.981                   48.981                +0.000    +0.0%
vec_RAM(MB)        9.918                    9.918                 +0.000    +0.0%
bp-reorder         false                    false
indexType          HNSW                     HNSW
rerank             no                       no

kaivalnp

Looking great, thanks @Pranshu-S!

Can you add a CHANGES.txt entry + update package-info.java too?

Can you also share benchmarks for higher levels of quantization with FLOAT16?

Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

Pranshu-S · 2026-10-02T08:38:04Z

Runs against different quantisation levels:

============================================================================
>>> quantizeBits=1  
============================================================================
=================================================================================
COMPARISON  (candidate=lucene_candidate+dedup vs baseline=lucene_baseline+non-dedup;
             delta/pct = candidate - baseline)
metric             baseline/non-dedup(n=5)  candidate/dedup(n=5)  delta    pct
-----------------  -----------------------  --------------------  -------  ------
recall             0.714±0.002              0.679±0.003           -0.035   -4.9%
latency(ms)        0.437±0.007              0.444±0.007           +0.007   +1.6%
netCPU             0.429±0.007              0.435±0.006           +0.006   +1.4%
avgCpuCount        0.982±0.003              0.981±0.003           -0.001   -0.1%
nDoc               10000                    10000
searchType         KNN                      KNN
topK               10                       10
fanout             100                      100
resultSimilarity   N/A                      N/A
decay              N/A                      N/A
resultCount        10.000                   10.000
maxConn            32                       32
beamWidth          200                      200
quantized          1 bits                   1 bits
indexEncoding      float16                  float16
visited            2364.000±5.292           2366.400±14.673       +2.400   +0.1%
index(s)           6.194±0.701              5.926±0.527           -0.268   -4.3%
index_docs/s       1631.810±192.762         1698.786±163.755      +66.976  +4.1%
merge(s)           0.000±0.000              0.000±0.000           +0.000
force_merge(s)     9.184±0.161              8.912±0.053           -0.272   -3.0%
num_segments       1.000±0.000              1.000±0.000           +0.000   +0.0%
index_size(MB)     32.418±0.084             21.906±0.005          -10.512  -32.4%
filterStrategy     index-time-filter        index-time-filter
filterSelectivity  0.50                     0.50
overSample         1.000                    1.000
vec_disk(MB)       20.885±0.000             20.885±0.000          +0.000   +0.0%
vec_RAM(MB)        1.354±0.000              1.354±0.000           +0.000   +0.0%
bp-reorder         false                    false
indexType          HNSW                     HNSW
rerank             no                       no


============================================================================
>>> quantizeBits=2  
============================================================================
=================================================================================
COMPARISON  (candidate=lucene_candidate+dedup vs baseline=lucene_baseline+non-dedup;
             delta/pct = candidate - baseline)
metric             baseline/non-dedup(n=5)  candidate/dedup(n=5)  delta    pct
-----------------  -----------------------  --------------------  -------  ------
recall             0.834±0.005              0.800±0.001           -0.034   -4.1%
latency(ms)        0.517±0.018              0.530±0.004           +0.013   +2.6%
netCPU             0.510±0.018              0.522±0.004           +0.012   +2.3%
avgCpuCount        0.986±0.001              0.984±0.003           -0.002   -0.2%
nDoc               10000                    10000
searchType         KNN                      KNN
topK               10                       10
fanout             100                      100
resultSimilarity   N/A                      N/A
decay              N/A                      N/A
resultCount        10.000                   10.000
maxConn            32                       32
beamWidth          200                      200
quantized          2 bits                   2 bits
indexEncoding      float16                  float16
visited            2161.200±49.226          2197.000±11.424       +35.800  +1.7%
index(s)           6.472±0.683              6.346±0.244           -0.126   -1.9%
index_docs/s       1560.354±171.079         1578.426±61.459       +18.072  +1.2%
merge(s)           0.000±0.000              0.000±0.000           +0.000
force_merge(s)     11.300±0.352             10.272±0.261          -1.028   -9.1%
num_segments       1.000±0.000              1.000±0.000           +0.000   +0.0%
index_size(MB)     34.144±0.171             23.080±0.000          -11.064  -32.4%
filterStrategy     index-time-filter        index-time-filter
filterSelectivity  0.50                     0.50
overSample         1.000                    1.000
vec_disk(MB)       22.106±0.000             22.106±0.000          +0.000   +0.0%
vec_RAM(MB)        2.575±0.000              2.575±0.000           +0.000   +0.0%
bp-reorder         false                    false
indexType          HNSW                     HNSW
rerank             no                       no

============================================================================
>>> quantizeBits=4
============================================================================
==================================================================================
COMPARISON  (candidate=lucene_candidate+dedup vs baseline=lucene_baseline+non-dedup;
             delta/pct = candidate - baseline)
metric             baseline/non-dedup(n=5)  candidate/dedup(n=5)  delta     pct
-----------------  -----------------------  --------------------  --------  ------
recall             0.922±0.001              0.913±0.002           -0.009    -1.0%
latency(ms)        0.553±0.016              0.547±0.018           -0.006    -1.0%
netCPU             0.546±0.015              0.541±0.018           -0.005    -1.0%
avgCpuCount        0.987±0.002              0.988±0.002           +0.001    +0.1%
nDoc               10000                    10000
searchType         KNN                      KNN
topK               10                       10
fanout             100                      100
resultSimilarity   N/A                      N/A
decay              N/A                      N/A
resultCount        10.000                   10.000
maxConn            32                       32
beamWidth          200                      200
quantized          4 bits                   4 bits
indexEncoding      float16                  float16
visited            2121.800±24.201          2116.800±39.996       -5.000    -0.2%
index(s)           6.456±0.441              5.446±0.629           -1.010    -15.6%
index_docs/s       1555.260±114.384         1854.910±205.027      +299.650  +19.3%
merge(s)           0.000±0.000              0.000±0.000           +0.000
force_merge(s)     11.134±0.199             9.996±0.321           -1.138    -10.2%
num_segments       1.000±0.000              1.000±0.000           +0.000    +0.0%
index_size(MB)     37.900±0.104             25.520±0.000          -12.380   -32.7%
filterStrategy     index-time-filter        index-time-filter
filterSelectivity  0.50                     0.50
overSample         1.000                    1.000
vec_disk(MB)       24.567±0.000             24.567±0.000          +0.000    +0.0%
vec_RAM(MB)        5.035±0.000              5.035±0.000           +0.000    +0.0%
bp-reorder         false                    false
indexType          HNSW                     HNSW
rerank             no                       no

============================================================================
>>> quantizeBits=7
============================================================================
==================================================================================
COMPARISON  (candidate=lucene_candidate+dedup vs baseline=lucene_baseline+non-dedup;
             delta/pct = candidate - baseline)
metric             baseline/non-dedup(n=5)  candidate/dedup(n=5)  delta     pct
-----------------  -----------------------  --------------------  --------  ------
recall             0.978±0.002              0.976±0.001           -0.002    -0.2%
latency(ms)        0.559±0.014              0.583±0.015           +0.023    +4.2%
netCPU             0.553±0.014              0.577±0.015           +0.024    +4.3%
avgCpuCount        0.988±0.001              0.989±0.001           +0.001    +0.1%
nDoc               10000                    10000
searchType         KNN                      KNN
topK               10                       10
fanout             100                      100
resultSimilarity   N/A                      N/A
decay              N/A                      N/A
resultCount        10.000                   10.000
maxConn            32                       32
beamWidth          200                      200
quantized          7 bits                   7 bits
indexEncoding      float16                  float16
visited            2118.800±12.755          2115.600±12.482       -3.200    -0.2%
index(s)           6.036±0.620              5.590±0.622           -0.446    -7.4%
index_docs/s       1670.912±165.291         1806.012±193.513      +135.100  +8.1%
merge(s)           0.000±0.000              0.000±0.000           +0.000
force_merge(s)     11.210±0.195             10.080±0.100          -1.130    -10.1%
num_segments       1.000±0.000              1.000±0.000           +0.000    +0.0%
index_size(MB)     45.096±0.212             30.398±0.008          -14.698   -32.6%
filterStrategy     index-time-filter        index-time-filter
filterSelectivity  0.50                     0.50
overSample         1.000                    1.000
vec_disk(MB)       29.449±0.000             29.449±0.000          +0.000    +0.0%
vec_RAM(MB)        9.918±0.000              9.918±0.000           +0.000    +0.0%
bp-reorder         false                    false
indexType          HNSW                     HNSW
rerank             no                       no

Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

Pranshu-S · 2026-10-05T15:23:42Z

Hi @kaivalnp, do let me know if any more changes are required in this PR. Thanks!

kaivalnp

Thanks @Pranshu-S, this PR is looking great!

Left some minor nits + code cleanup comments, LGTM overall.

Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

Pranshu-S · 2026-10-07T17:11:18Z

Looks like all recent PRs are failing due to CodeQL, will merge once the fix for it lands in main branch

Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

kaivalnp

LGTM, thanks @Pranshu-S !

Add scalar quantization support for float16 vectors in the dedup KNN …

7c3027d

…codec Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

github-actions Bot added the module:sandbox label Sep 25, 2026

Pranshu-S marked this pull request as ready for review September 25, 2026 08:07

kaivalnp reviewed Sep 28, 2026

View reviewed changes

Pranshu-S added 3 commits September 30, 2026 15:59

address comments and clean code

7ce2668

Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

tidy

605e5a9

Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

decouple fp16 quantization inflation to scorer

aa5cdcd

Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

Pranshu-S force-pushed the dedup-fp16-quantization branch from 85dc225 to aa5cdcd Compare October 1, 2026 10:57

Pranshu-S added 2 commits October 1, 2026 16:32

Merge remote-tracking branch 'origin/main' into dedup-fp16-quantization

9170858

Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

fix changed dependencies

d072923

Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

kaivalnp reviewed Oct 1, 2026

View reviewed changes

address comments and update documentation

8e19801

Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

Pranshu-S force-pushed the dedup-fp16-quantization branch from b9b1348 to 8e19801 Compare October 2, 2026 07:54

Merge remote-tracking branch 'origin/main' into dedup-fp16-quantization

b802dd8

Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

kaivalnp reviewed Oct 5, 2026

View reviewed changes

kaivalnp linked an issue Oct 5, 2026 that may be closed by this pull request

Improvements to the sandboxed de-duplicating vector format #16595

Open

1 of 6 tasks

kaivalnp added this to the 11.0.0 milestone Oct 5, 2026

Pranshu-S added 3 commits October 7, 2026 22:06

address comments + cleanup

3bd129b

Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

Merge remote-tracking branch 'origin/main' into dedup-fp16-quantization

283b160

Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

fix changes.txt

58206db

Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

kaivalnp reviewed Oct 7, 2026

View reviewed changes

address comments

88c13a4

Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

Merge remote-tracking branch 'origin/main' into dedup-fp16-quantization

6ff650d

Signed-off-by: Pranshu-S <pranshushukla06@gmail.com>

github-actions Bot added the module:core/codecs label Oct 8, 2026

kaivalnp approved these changes Oct 9, 2026

View reviewed changes

Conversation

Pranshu-S commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How

Uh oh!

kaivalnp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Pranshu-S commented Oct 1, 2026

Uh oh!

kaivalnp commented Oct 1, 2026

Uh oh!

Pranshu-S commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kaivalnp commented Oct 1, 2026

Uh oh!

Pranshu-S commented Oct 1, 2026

Uh oh!

kaivalnp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Pranshu-S commented Oct 2, 2026

Uh oh!

Pranshu-S commented Oct 5, 2026

Uh oh!

kaivalnp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Pranshu-S commented Oct 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaivalnp left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Pranshu-S commented Sep 25, 2026 •

edited

Loading

Pranshu-S commented Oct 1, 2026 •

edited

Loading