Change log draft:

Change log

Features 🌰

• https://github.com/qdrant/qdrant/milestone/33?closed=1 - Inline Storage: Add option to inline vectors in HNSW graph for efficient IO usage
• https://github.com/qdrant/qdrant/milestone/37?closed=1 - Tenant promotion mechanism for tiered multitenancy:
  • Add ReplicatePoints action to promote payload based tenant into dedicated shard key
  • Add fallback shard key for intelligent routing to tenants that are or are not promoted to a dedicated shard
  • On shard key creation, allow to specify initial state of new replicas
• Add ACORN-1 search #7414 - Add ACORN-1 search method, accurate search over many filtered points at the cost of performance
• Full-Text Index ASCII Folding (Normalization) #7408 - Add ASCII folding (normalization) to full text indices, fold diacritics into ASCII characters
• Update if #7006 - Add conditional update functionality, only apply update on points matching filter
• filter text any #7100 - Add text_any full text filter to match any query term
• Custom RRF k parameter #7065 - Add option to customize RRF k parameter
• [strict-mode] Add max number of payload index count #7222 - In strict mode, specify maximum number of payload indices per collection
• implement custom collection metadata #7123 - Add custom key-value metadata to collections
• Implement logging of slow requests for scroll, facet, count, and retrieve #7291 - Add profiler to log slow point update and read requests

Improvements 🏎️

• In Gridstore, also populate the tracker file #7385 - When loading Gridstore, populate tracker data into memory for faster first access
• async update operation #7407, make flush worker loop async #7405 - Spawn updates and flush workers task on updates runtime, use two system threads less for each local shard
• bump WAL commit #7413 - Use system thread on demand in WAL, use one less system thread per local shard by default
• Use atomic BitVec for GraphLayersBuilder::ready_list #7468 - Use atomic bit flags on HNSW construction to significantly increase indexing performance
• avx512 for binary quantization #7052, Add avx512vl and avx512vpopcntdq in telemetry #7471 - Implement AVX512 SIMD optimizations for binary quantization on modern x86_64 CPUs
• Skip RocksDB for mutable payload index and payload storage #7433 - Switch new mutable payload indices and storage from RocksDB to Gridstore for better performance
• Enable appendable quantization by default #7508 - Enable quantization in appendable segments by default, improving search performance
• Change score of query-less prefetch to 1.0 #7347 - Change default score of query-less prefetch to 1.0 to ease score boosting
• remove default explicit config for strict mode #7369 - Don't explicitly disable strict mode by default
• non appendable proxies #7345 - Simplify internal handling of copy-on-write segments, now write incoming updates to dedicated segments directly
• Add CollectionInfo::warnings field #7293, Clarify multivector inline storage warning #7523 - Add warnings field to collection info, report misconfiguration
• Use fs-err #7319, Use fs_err::exists and fs_err::tokio::try_exists #7401 - Report more helpful error messages on file IO errors
• feat: handle 429 properly on infer svc #7377 - When using inference, propagate rate limit responses
• rate limit slow request warning queue #7434 - Rate limit slow request warning to prevent spamming logs
• Log shard transfer abort during consensus recovery #7373 - Log shard transfers as a result of the consensus recovery procedure
• Tone down optimizer logging #7337 - Tone down optimizer logging to make it less verbose
• Docker build parameters #7370 - Add TARGET_CPU and JEMALLOC_SYS_WITH_LG_PAGE build parameters to Docker image

Metrics 📈

• Add num points/vectors to metrics API #7302, Metrics vectors by name per collection #7441 - In metrics, report point and vector counts per collection and vector name (collection_{points,vectors})
• Add unindexed vectors to telemetry and metrics API #7307 - In metrics, report number of points skipped in indexed_only search (collection_indexed_only_excluded_points)
• Add replica metrics #7301 - In metrics, report global effective minimum and maximum shard replication count (collection_active_replicas_{min,max})
• Dead shards in /metrics #7310, Final metrics tweaks for Qdrant 1.16.0 #7516 - In metrics, report total number of non-active replicas (collection_dead_replicas)
• Currently running optimizers in Metrics #7316, Various metrics tweaks #7480 - In metrics, expose collection_running_optimizations with number of optimizers running per collection
• Metrics for snapshots #7497 - In metrics, report running, created and recovered number of snapshots (snapshot_{creation,recovery}_running, snapshot_created_total)
• Show active thread count in metrics #7484 - In metrics, report active thread count (process_threads)
• Procfs metrics #7451 - In metrics, report number of open file descriptors and the limit (process_{open,max}_fds)
• Report max system wide mmap count in metrics #7487 - In metrics, report open mmaps and system limit (process_open_mmaps, system_max_mmaps)
• In metrics, only show cumulative number of page faults #7482 - In metrics, report total number of major and minor page faults (process_{minor,major}_page_faults_total)
• Add global metrics prefix #7438 - In metrics, add configuration option to prefix all metrics with qdrant_ or something else

Bug fixes 🕵️

• Restrict /logger API #7527 - Fix logger API allowing arbitrary file writes, now this can only be configured through configuration
• Abort pending blocking search tasks when we drop/invalidate the search #7530 - Abort pending search tasks when search is cancelled, fixing optimizer instability under huge load
• Audit all spawn blocking calls, prematurely abort them #7533 - Abort other blocking tasks such as retrieve and snapshot prematurely if the caller gets cancelled
• more stop flags #7531 - Cancel ongoing searches more aggressively if the search is cancelled
• Fix hash ring initialization for custom sharding, prevent resharding panic #7517 - Fix resharding down panic if no shard key is provided on collection with custom sharding
• Access vector_data without risk of panicking #7469 - Fix panic on certain queries with unknown vector name
• Forbid peer to join cluster with URI that is already used #7375 - Forbid peer to join cluster with URI that is already used, which could break a cluster
• fix segment repair on load #7400 - Fix corrupt segments on load if segment was partially flushed, prevent payload index corruption
• Force flush when taking a snapshot to capture on-going batches #7342, address parallel flushing problem in snapshot creation #7404, Fix shard snapshots IO error, proxy first then flush #7427 - Force flush all segments when taking snapshot to prevent data corruption
• Improved segment flush ordering #7381 - Fix flush ordering on segments currently being snapshotted, fixing data consistency on crash
• single thread flush #7388, use mutex to ensure sequential segment flushed + await on drop #7416 - Fix flush ordering with concurrent flushes to ensure data consistency
• Allow partial recovery of dummy shard #7512, Allow payload operations on dummy shard #7521 - Fix dummy shards not allowing snapshot recovery

Web UI 🫂

• https://github.com/qdrant/qdrant-web-ui/releases/tag/v0.2.0 - New web UI design to match Qdrant Cloud

Deprecations 🚧

• remove init_from #7454 - Remove init_from collection API, deprecated since Qdrant 1.15
• remove lock api #7449 - Remove lock API, deprecated since Qdrant 1.15
• Remove old shard key mapping format #7047 - Remove old internal shard key format, deprecated and migrated away from in Qdrant 1.15
• remove filter level jwt #7450 - Remove payload filter from RBAC/JWT, deprecated since Qdrant 1.15, API keys using it are rejected
• Mark legacy output vector fields deprecated #7183 - Deprecate old variant of vector output