Repository navigation
Use the terms query parser for expanded entity sets - #1459
Open
kevinschaper wants to merge 2 commits into
Open
kevinschaper wants to merge 2 commits into
kevinschaper wants to merge 2 commits into
Conversation
An association query over an entity's closure or ortholog expansion built a
`field:"value"` chain at six clauses per ID, so past ~170 IDs it exceeded Solr's
maxBooleanClauses of 1024 and failed to parse. The request 400'd, nothing caught
it, and the API returned a 500. The clause count tracks the size of the
expansion rather than anything about the request, so an affected entity was
broken on every attempt rather than intermittently.
`{!terms}` takes a value set as a single clause, so the cost is one clause per
field whatever the ID count -- the ceiling goes away instead of moving. `_query_`
nesting is what lets six of them be OR'd: a local param applies to a whole query
string, so they cannot sit side by side in one `fq`.
Verified against a local 2026-09-04 index: `SGD:S000001393` has 212 orthologs,
and its association table raised HTTPError 400 before this and returns 2,332
associations after. 37 responses across associations, counts, histopheno and
association tables are byte-identical either way.
Closes #1458
Claude-Session: https://claude.ai/code/session_01N6fjfGqaFpyjepw23qCyLv
The service logged Solr's error message and then re-raised the HTTPError uncaught, so a rejected query surfaced as a 500. The status pointed at the server when the problem was the query, which is where the reader goes first. A Solr 4xx now raises SolrQueryError and the API answers 400 with the message; a 5xx is Solr being down or overloaded, stays an HTTPError, and stays a 500. Solr quotes the query it rejected and the rejected ones are the long ones -- the clause-limit messages ran past 50,000 characters -- so the response carries a truncated prefix and the log keeps the whole thing. Claude-Session: https://claude.ai/code/session_01N6fjfGqaFpyjepw23qCyLv
✅ Deploy Preview for monarch-app canceled.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1459 +/- ##
==========================================
+ Coverage 85.08% 85.39% +0.31%
==========================================
Files 131 133 +2
Lines 7548 7657 +109
==========================================
+ Hits 6422 6539 +117
+ Misses 1126 1118 -8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1458.
The failure
An association query over an entity's closure or ortholog expansion built a
field:"value"OR chain at six clauses per ID —subject,object,disease_context_qualifierand their_closurecounterparts. Solr'smaxBooleanClausesis 1024 and our config does not raise it, so the ceiling landed at ~170 IDs. Past that the query failed to parse, Solr answered 400, nothing caught it, and the API returned a 500.The clause count follows the size of the expansion rather than anything about the request, so an affected entity was broken on every attempt — specific pages simply did not work, as the issue notes.
The fix
{!terms}takes a whole value set as a single clause, so the cost becomes one clause per field regardless of ID count: the ceiling goes away rather than moving._query_nesting is what allows six of them to be OR'd —{!terms}is a local param and applies to an entire query string, so they cannot sit side by side in onefq.Applied to the
subject,objectandentityfilters inbuild_association_query, and to the ortholog facet suffixes inbuild_association_count_suffixes— the counts query hit its own ceiling at roughly half the entity count, becausefacet.queryis parsed under the same budget.Two things the issue flagged to check:
{!terms}splits on comma. Facets over the live index show zero of 1,716,622 distinctsubject/object/disease_context_qualifiervalues containing a comma, quote, pipe, space or backslash. Butentityis caller-supplied, so a value carrying a separator or a quote stays an ordinary clause instead of being silently split or breaking out of the nested query.stringin the association schema (confirmed against the live schema API).Also here
A Solr 4xx now raises
SolrQueryErrorand the API answers 400 instead of 500 — the issue's "a Solr 400 arguably shouldn't surface as an opaque 500". A 5xx is Solr being down or overloaded and still surfaces as a 500. Solr quotes the rejected query back, and the rejected ones are the long ones (the clause-limit messages ran past 50,000 characters), so the response carries a truncated prefix and the log keeps all of it. Separate commit, droppable without touching the fix.Verification
Against a local Solr 8.11.4 with the stock 1024 limit and the 2026-09-04 index:
SGD:S000001393— the entity in the issue's log excerpt — has 212 orthologs, so 213 entities and 1,278 clauses. Its association table raisedHTTPError: 400onmainfor both categories tried; on this branch it returns 2,332 and 12 associations.numFound.get_associations(direct and indirect),get_association_counts,get_histophenoandget_association_table(with and without ortholog traversal) are byte-identical betweenmainand this branch.facet.queryunderdefType=edismax, where the_query_nesting had to be checked rather than assumed.backend/tests/unit/test_solr_clause_limits.pypins the property without needing Solr: clause counts bounded and flat in the number of IDs, since a bounded-only check would pass again on a reverted OR chain at small N. Reverting the helper to a chain fails 12 of its 20 tests, including exactly the171indirect case (1,026 clauses) while171direct still passes (513) — the ~170 boundary from the issue. Dropping the exception handler or the raise each fail their own tests.782 passed, 16 skipped.