Skip to content

feat(engine): kNN + aggregations in a single request (rc.6) + calltree.ai case study - #60

Merged
xerj-org merged 2 commits into
mainfrom
feat/knn-aggs-rc6
Jul 27, 2026
Merged

feat(engine): kNN + aggregations in a single request (rc.6) + calltree.ai case study#60
xerj-org merged 2 commits into
mainfrom
feat/knn-aggs-rc6

Conversation

@xerj-team

@xerj-team xerj-team commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

Closes the gap where aggregations did not run alongside a knn/semantic query — the friction the calltree.ai deep-research analytics use-case hit (aggregate a semantic slice previously needed a two-step kNN→ids→agg).

Feature (rc.6)

Aggregations now run over the retrieved top-k neighbour set (Elasticsearch top-level-knn semantics), independent of from/size:

POST /conversations/_search
{ "query": { "knn": { "field":"vec", "query_vector":[...], "k":200, "num_candidates":400 } },
  "size": 3,
  "aggs": { "by_band": { "terms": { "field":"band" },
            "aggs": { "csat": { "percentiles": { "field":"csat","percents":[50] } } } } } }
  • Implemented in the shared kNN result assembler (knn_result_from_scored) so all three executors (HNSW, exact brute-force, multi-kNN) gain it.
  • An aggs-bearing kNN routes to exact brute-force — ANN recall is <100% and aggregate counts must be exact, not approximate.
  • ~30 lines, no new API surface.

Validation

  • Integration test test_knn_plus_aggregations_single_request — asserts the aggregation runs over the exact semantic slice (far docs excluded). Passes.
  • No kNN/semantic regressions (existing suite green).
  • Live with real 768-dim EmbeddingGemma embeddings: a single request returns band split + key issues + CSAT percentile + AHT avg + example hits together.

Known limitation (documented)

significant_terms over a kNN slice returns empty — the vector path doesn't yet supply a background corpus. Use terms in-slice or the two-step pattern for significance. Filed as the rc.6 follow-up.

Case study

docs/case-studies/calltree-analytics/ — tested-live, runnable review of XERJ for columnar analytics + embedding retrieval (the calltree.ai deep-research use case), with a step-by-step guide, the embed→label→materialize architecture, a scaling guide, and an honest comparison vs DuckDB shim / pgvector / Elasticsearch / warehouse+vector / dedicated vector DBs.

See engine/CHANGELOG-rc6.md.

🤖 Generated with Claude Code

Xerj Squad A added 2 commits July 27, 2026 15:59
Xerj Squad A
feat(engine): kNN + aggregations in a single request (rc.6) + calltre…
ae2b834 
…e.ai case study

## Feature
Aggregations may now accompany a knn/semantic query; they run over the retrieved
top-k neighbour set (ES top-level-knn semantics), independent of from/size. This
makes "aggregate a semantic slice" one request -- the gap the calltree.ai
deep-research analytics use-case hit (previously required a two-step kNN->ids->agg).

Implementation: the shared kNN result assembler (knn_result_from_scored -- used by
ALL three executors: HNSW, exact brute-force, multi-kNN) computes run_aggs over the
top-k sources when aggs is present. An aggs-bearing kNN routes to exact brute-force
(ANN recall <100%; aggregate counts must be exact). ~30 lines, no new API surface.

Validated: integration test test_knn_plus_aggregations_single_request (agg over the
exact semantic slice, far docs excluded) passes; no kNN/semantic regressions; and
live with real 768-dim EmbeddingGemma embeddings (single request returns band
split + key issues + csat percentile + AHT avg + example hits together).

Known limitation (documented): significant_terms over a kNN slice returns empty
(background corpus not wired for the vector path) -- use terms in-slice or two-step.

## Case study: docs/case-studies/calltree-analytics/
Columnar analytics + embedding retrieval for calltree.ai's deep-research reports
("of calls about wifi issues, how many 2.4/5GHz, key issues?"). Tested-live,
runnable (3 scripts), with a step-by-step guide, the embed->label->materialize
architecture, a scaling guide (object-store backend, k sizing, ingest labeling,
multi-tenant), and an honest comparison vs DuckDB shim / pgvector / Elasticsearch /
warehouse+vector / dedicated vector DBs.

Co-Authored-By: Xerj Squad A <noreply@xerj.org>
Xerj Squad A
fix(test): resolve stray merge-conflict markers in integration.rs (CI…
7628719 
… rustfmt)

A `git stash pop` during branch setup left conflict markers
(<<<<<<< / ======= / >>>>>>>) in integration.rs, splitting
test_term_matches_non_first_array_element before its closing brace ->
"unclosed delimiter" in cargo fmt --check. Resolved: keep BOTH main's
(ignored) array-term reproduction test AND the new knn+aggs test, restore the
missing brace, rustfmt-clean. Both compile; knn+aggs test passes; array-term
stays #[ignore].

Co-Authored-By: Xerj Squad A <noreply@xerj.org>
@xerj-org
xerj-org merged commit 761b47d into main Jul 27, 2026
4 checks passed
xerj-team pushed a commit that referenced this pull request Jul 28, 2026
Xerj Squad A
docs(case-study): implement all 3 CDC/hybrid follow-ups (RRF, push-st…
7e4d049 
…reaming, LSN) — rc.6

Addresses the three limitations flagged in the daily.dev CDC case study, all
implemented and proven live:

1. RRF hybrid — turned out to ALREADY exist first-class (`hybrid` + `fusion:rrf`,
   with weights + `linear`); the earlier "not yet" was MY request-syntax error.
   Corrected the docs to the real shape; verified live (rrf_score ~1/(60+rank)):
   fuses BM25+vector with no manual score normalization.

2. Push-streaming CDC — cdc_stream.py uses psycopg2
   LogicalReplicationConnection.consume_stream() (true push, blocks on WAL, no
   polling). Proven: an INSERT in pg appears in XERJ within seconds.

3. LSN checkpointing (exactly-once-convergent) — send_feedback(flush_lsn=...) after
   each durable apply advances confirmed_flush_lsn. Proven: stop consumer -> change
   data while down (UPDATE p1, INSERT p11) -> XERJ stale -> restart resumes from the
   last checkpoint and applies EXACTLY the missed changes, no loss, no re-sync.

Rewrote the honest-limitations section (embedding-model quality, RRF k/weight
tuning, no learned fusion, backfill cost) and moved the three resolved items to a
"Resolved" list. Complements the rc.6 knn+aggregations feature (PR #60).

Co-Authored-By: Xerj Squad A <noreply@xerj.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@xerj-team @xerj-org