Tags: Infinibay/ken
Tags
Release 0.7.1: make the version a single source of truth
0.7.0 shipped reporting `ken 0.6.0`. The version lived in two files —
`pyproject.toml` for the package metadata and `src/ken/__init__.py` for
everything at runtime — and only the first was bumped. It is not a one-off:
0.5.0 went out the same way.
`src/ken/__init__.py` is now the only place it is written. pyproject.toml
declares `dynamic = ["version"]` and points hatchling at that file, so the
published metadata and what `--version` prints are the same string by
construction rather than by remembering. RELEASING.md says so, and says
which releases got it wrong.
Verified by installing the built wheel into a clean venv: `ken --version`
and `importlib.metadata.version("ken-rank")` both report 0.7.1 — the check
that was missing when 0.7.0 went out.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Release 0.7.0: Apple Silicon (MPS) support for the torch embedder `_resolve_device()` only ever probed `torch.cuda`, so every Mac ran the torch backend on the CPU no matter what. It now prefers CUDA, then MPS, then CPU; `KEN_EMBED_DEVICE` gains `cuda` and `mps`, and naming one accelerator restricts the choice to it rather than substituting the other. MPS is trusted less than CUDA, because it fails in ways CUDA does not: * It can lie quietly. A batch can come back NaN, or as rows the GPU never filled in — which survive `F.normalize`'s clamped divisor as all-zero vectors that are perfectly finite and then score 0.0 against every query for the life of the index. Batches are now checked for finite *and* unit-norm output; on failure, or on any raise, the model is rebuilt on the CPU once per process and the batch redone. * It will not report being out of memory. torch's MPS allocator ceiling is ~1.7x Metal's recommended working set, which lands above the RAM that exists, so macOS swaps instead of raising and the fallback above never fires. `_cap_mps_memory()` caps it (`KEN_MPS_MEMORY_FRACTION`, 0.7). * torch below 2.9 could return finite, unit-norm, *wrong* vectors (pytorch#163597) — the one class no runtime check can catch. Excluded by version, in `_mps_available()` and as an arm64-scoped pyproject pin. The documented NaN failure (sentence-transformers#3498) needs `padding_side="left"`, which the Qwen3 model card recommends. ken passes no `tokenizer_kwargs`, inherits right padding, and never builds the fully-masked attention row that makes Metal's softmax emit NaN. There is a comment at the load site saying so; it is load-bearing. Two unrelated packaging bugs found on the way: `[torch]` had no `transformers` floor, but the Qwen3 architecture landed in 4.51 and below it the model dies with `KeyError: 'qwen3'`; and `[gpu]` was uninstallable on macOS, since onnxruntime-gpu publishes no Mac wheels. The torch backend had no tests at all. It has 23 now, all fake-driven. Not verified on real hardware: MPS correctness for this model is inferred from a shared root cause, not retested upstream. The guards are there because it could not be. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Release 0.6.0: fix the math, statistics and embedding use of the non-…
…ranker tools
Follows the 0.3.0 ranker work by auditing every *other* tool for mathematical,
statistical and vector-space correctness, then putting the result through an
adversarial review pass that found four more defects in the fixes themselves.
Embeddings — query/passage asymmetry:
- Stored *documents* (ci_files, ci_intent_sources, cr_findings) were encoded
with embed_query, and `ken reembed` re-encoded cr_contexts — which are
*queries* the ranker compares against a live prompt — as passages. Each side
now matches its role. Byte-identical on the symmetric default models; it is
the asymmetric ones the 0.4/0.5 torch work enabled (e5, Qwen3) that were
filed in the wrong half of the space.
- The prompt table moved to `ken.embedder` so BOTH backends apply it.
fastembed's query_embed/passage_embed are aliases of embed for every model
but Jina v3, so intfloat/multilingual-e5-* — which fastembed serves, and
`ken models` advertises — got no prefixes at all on the default backend.
- New `embed_queries()` batches query encoding; reembed no longer walks the
prompt log one row at a time.
- DOC_SPACE_VERSION + `pending_reembed()`: an existing index cannot heal itself
(an unchanged file keeps its stored vector), so the session brief now tells
the user to run `ken reembed` — and only when the model is actually
asymmetric, so the shipped defaults never see the nudge.
- stack_embeddings / cosine_against / rank_against replace raw np.asarray
stacking everywhere: vectors from a previous model are dropped rather than
raising a ragged-array ValueError far from the cause. Direct lookups (search,
recall) raise a message naming `ken reembed`; background scorers (ranker
channels, intent history, findings graph) degrade to empty so a hook never
dies mid-injection.
clones — MinHash was estimating rather than measuring:
- Shingles hashed with blake2b instead of PYTHONHASHSEED-salted `hash()`, so
the same repo no longer yields different clones on every run.
- (a*x+b) mod p evaluated over 31-bit values against the Mersenne prime
2**31-1; the previous 61-bit constants silently wrapped in int64 and stopped
being a permutation at all.
- Reported `similarity` is now the exact Jaccard of the shingle sets; MinHash
only retrieves candidates (64 permutations carry ~0.054 stderr at 0.75).
- Band/row counts derive from `min_similarity`. The fixed 16x4 banding is tuned
for ~0.75 and silently missed ~36% of true pairs at 0.5; unchanged at the
default.
- Added `containment` for a body pasted inside something larger, and fixed
lstrip("./") turning .hidden/mod.py into hidden/mod.py.
cochange — statistics:
- Market-basket stats are scoped to the target file's own repo. Lift is
n_ab*N/(n_a*n_b), linear in N, so a workspace's sibling repos scaled every
lift: measured 9.2 where the answer is 1.2.
- A Dunning G^2 significance gate (chi-square, 1 df) replaces the naked
`lift > 1`; `min_llr` is exposed and can be set to 0 for the old behaviour.
- Wilson lower bound on confidence, plus a `strength` ranking key that discounts
the recency-weighted confidence by how thin the evidence actually is.
find_tests — returned 15 undifferentiated files for db.py, with the right one
fourth:
- Every channel now contributes to the same candidate instead of the first one
winning (setdefault was discarding the name-convention match), and results
are ranked: named by convention (3.0) > shared name tokens (2.0) > imports
the target (1.0), which nearly every test of a widely-imported module does.
- Token containment instead of substring, so cli.py no longer pulls in
test_client.py; structural stems (__init__, index, utils, ...) are excluded
from the weak channel.
- Case-sensitive CamelCase detection for Java/C#/Kotlin: a lowercased
endswith("test.java") also swallowed Latest.java, Contest.cs and Greatest.kt.
- Go, Rust, Ruby, Kotlin and __tests__/ conventions recognised.
module_graph: edges whose endpoints the node cap excluded are no longer emitted
(33 of 51 dangled at depth=3, limit=15), truncation is reported, and endpoint
paths resolve in one query instead of one per edge.
search_files / search_symbols: identifier-shaped queries fuse the semantic order
with a literal name match — `blast_radius` used to return
test_blast_radius_reverse_reachability above the function. Exact matches are
promoted, partial ones go through reciprocal rank fusion; prose queries are
untouched. Ranks, not score sums, because cosine magnitudes are model-dependent.
architecture: PageRank reports whether it converged instead of trusting a fixed
40 sweeps, and clusters carry Newman modularity so a caller can tell real
structure from an artefact of label propagation always returning a partition.
Also fixes `ken --version`, which reported 0.4.0 on a 0.5.0 install — the
0.5.0 bump missed src/ken/__init__.py.
555 tests (+36).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Release 0.5.0: user-configurable default model + `ken models` Add two CLI commands for managing the embedding model: * `ken default-model [NAME]` — show/set/clear the default embedding model for *new* projects, stored in the user config (~/.config/ken/config.json, honoring KEN_CONFIG_DIR / XDG_CONFIG_HOME). Existing projects are never touched; recommended_model() now returns the user override when set. * `ken models [--json]` — list available embedding models: the fastembed catalog (drop-in) sorted by dim/size with the current default and legacy models marked and multilingual flagged, plus a curated torch-backend section (Qwen3-Embedding-0.6B, bge-m3) and how to set/switch. Tests: user-default get/set/clear, fresh DB uses the user default while a legacy DB stays pinned, pending_upgrade targets the user default. A new autouse conftest fixture isolates the user config per test so a real `ken default-model` override can't leak into the suite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Release 0.2.4: publish to PyPI as ken-rank, trim sdist Rename distribution to ken-rank (the name 'ken' is taken on PyPI); the console command stays 'ken'. Restrict the sdist to source + metadata so it no longer bundles local caches. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PreviousNext