Skip to content

Use the terms query parser for expanded entity sets - #1459

Open
kevinschaper wants to merge 2 commits into
mainfrom
fix/solr-max-boolean-clauses
Open

kevinschaper wants to merge 2 commits into
mainfrom
fix/solr-max-boolean-clauses

Conversation

@kevinschaper

Copy link
Copy Markdown
Member

Closes #1458.

The failure

An association query over an entity's closure or ortholog expansion built a field:"value" OR chain at six clauses per ID — subject, object, disease_context_qualifier and their _closure counterparts. Solr's maxBooleanClauses is 1024 and our config does not raise it, so the ceiling landed at ~170 IDs. Past that the query failed to parse, Solr answered 400, nothing caught it, and the API returned a 500.

The clause count follows the size of the expansion rather than anything about the request, so an affected entity was broken on every attempt — specific pages simply did not work, as the issue notes.

The fix

{!terms} takes a whole value set as a single clause, so the cost becomes one clause per field regardless of ID count: the ceiling goes away rather than moving. _query_ nesting is what allows six of them to be OR'd — {!terms} is a local param and applies to an entire query string, so they cannot sit side by side in one fq.

Applied to the subject, object and entity filters in build_association_query, and to the ortholog facet suffixes in build_association_count_suffixes — the counts query hit its own ceiling at roughly half the entity count, because facet.query is parsed under the same budget.

Two things the issue flagged to check:

  • Separator safety. {!terms} splits on comma. Facets over the live index show zero of 1,716,622 distinct subject/object/disease_context_qualifier values containing a comma, quote, pipe, space or backslash. But entity is caller-supplied, so a value carrying a separator or a quote stays an ordinary clause instead of being silently split or breaking out of the nested query.
  • Analysis. The terms parser does not run values through the field's analysis chain, so this is only like-for-like on unanalyzed fields. All six are string in the association schema (confirmed against the live schema API).

Also here

A Solr 4xx now raises SolrQueryError and the API answers 400 instead of 500 — the issue's "a Solr 400 arguably shouldn't surface as an opaque 500". A 5xx is Solr being down or overloaded and still surfaces as a 500. Solr quotes the rejected query back, and the rejected ones are the long ones (the clause-limit messages ran past 50,000 characters), so the response carries a truncated prefix and the log keeps all of it. Separate commit, droppable without touching the fix.

Verification

Against a local Solr 8.11.4 with the stock 1024 limit and the 2026-09-04 index:

  • Reproduced. SGD:S000001393 — the entity in the issue's log excerpt — has 212 orthologs, so 213 entities and 1,278 clauses. Its association table raised HTTPError: 400 on main for both categories tried; on this branch it returns 2,332 and 12 associations.
  • Boundary confirmed. A 170-ID chain parses at 1,020 clauses; 200 IDs fails at 1,200. The terms form holds at 2,000 IDs and returns the same numFound.
  • No behaviour change. 37 responses across get_associations (direct and indirect), get_association_counts, get_histopheno and get_association_table (with and without ortholog traversal) are byte-identical between main and this branch.
  • Facet path. Counts verified identical between both forms inside facet.query under defType=edismax, where the _query_ nesting had to be checked rather than assumed.

backend/tests/unit/test_solr_clause_limits.py pins the property without needing Solr: clause counts bounded and flat in the number of IDs, since a bounded-only check would pass again on a reverted OR chain at small N. Reverting the helper to a chain fails 12 of its 20 tests, including exactly the 171 indirect case (1,026 clauses) while 171 direct still passes (513) — the ~170 boundary from the issue. Dropping the exception handler or the raise each fail their own tests.

782 passed, 16 skipped.

An association query over an entity's closure or ortholog expansion built a
`field:"value"` chain at six clauses per ID, so past ~170 IDs it exceeded Solr's
maxBooleanClauses of 1024 and failed to parse. The request 400'd, nothing caught
it, and the API returned a 500. The clause count tracks the size of the
expansion rather than anything about the request, so an affected entity was
broken on every attempt rather than intermittently.

`{!terms}` takes a value set as a single clause, so the cost is one clause per
field whatever the ID count -- the ceiling goes away instead of moving. `_query_`
nesting is what lets six of them be OR'd: a local param applies to a whole query
string, so they cannot sit side by side in one `fq`.

Verified against a local 2026-09-04 index: `SGD:S000001393` has 212 orthologs,
and its association table raised HTTPError 400 before this and returns 2,332
associations after. 37 responses across associations, counts, histopheno and
association tables are byte-identical either way.

Closes #1458

Claude-Session: https://claude.ai/code/session_01N6fjfGqaFpyjepw23qCyLv
The service logged Solr's error message and then re-raised the HTTPError
uncaught, so a rejected query surfaced as a 500. The status pointed at the
server when the problem was the query, which is where the reader goes first.

A Solr 4xx now raises SolrQueryError and the API answers 400 with the message;
a 5xx is Solr being down or overloaded, stays an HTTPError, and stays a 500.
Solr quotes the query it rejected and the rejected ones are the long ones -- the
clause-limit messages ran past 50,000 characters -- so the response carries a
truncated prefix and the log keeps the whole thing.

Claude-Session: https://claude.ai/code/session_01N6fjfGqaFpyjepw23qCyLv
@netlify

netlify Bot commented Oct 5, 2026

Copy link
Copy Markdown

✅ Deploy Preview for monarch-app canceled.

Name Link
🔨 Latest commit 61af111
🔍 Latest deploy log https://app.netlify.com/projects/monarch-app/deploys/6ac3d7d51f91290008b904fc

@codecov

codecov Bot commented Oct 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.39%. Comparing base (e9c3d5c) to head (61af111).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1459      +/-   ##
==========================================
+ Coverage   85.08%   85.39%   +0.31%     
==========================================
  Files         131      133       +2     
  Lines        7548     7657     +109     
==========================================
+ Hits         6422     6539     +117     
+ Misses       1126     1118       -8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Association queries over >~170 expanded IDs exceed Solr maxBooleanClauses and return 500 (~120/day)

1 participant