Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions changelog.d/fixed/6973-browser-table-header-budget.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
`max_content_chars` now bounds the link table's opening line along with its entries, so the extraction stays inside the ceiling an operator set rather than overshooting it by that line's length.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This describes max_content_chars as now bounding the header line, but the diff doesn't change EXTRACT_CONTENT_JS_TEMPLATE's tableCost() in browser.rs — only the Rust unit test was updated to include the header cost in its own local port. As it stands this fragment describes a fix that isn't in the code yet (see the comment on browser.rs); worth holding until the JS side actually changes, or the release notes will claim a behavior change that didn't ship.


Generated by Claude Code

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bullet ("max_content_chars now bounds the link table's opening line along with its entries, so the extraction stays inside the ceiling...") describes a behavioral fix that the diff doesn't actually make. EXTRACT_CONTENT_JS_TEMPLATE's own tableCost/attempt functions in crates/librefang-runtime/src/browser.rs (the code that runs in-browser and decides the real cut) are unchanged in this PR — only the Rust unit test's local port of that algorithm was updated. See the inline comment on browser.rs:2506 for the detailed trace. If the intent is only to fix the test's blind spot (leaving the JS fix for later), this fragment should say that instead of claiming the ceiling is now honored, since this text goes into the release notes verbatim.


Generated by Claude Code

The budget summed the entries and stopped there, but the rendered block also opens with a line naming the marker form and the base origin, and that line reaches the model with the entries — 50,093 characters against a 50,000 cap on the Rust Wikipedia article, the 93 being that line for a 24-character origin.
The test could not have caught it: it re-derived the table's cost in its own port and asserted against that same derivation, so the budget and the assertion agreed by construction whatever the renderer did.
Every ported test in this module now asserts that the template still contains the rule it models, since a port is only evidence about the script while the two agree — and nothing else would have noticed the script and its port drifting apart.
It now asserts against what `render_page_body` actually produces, which is the string that reaches the model (#6624, #6973) (@nevgenov)
96 changes: 81 additions & 15 deletions crates/librefang-runtime/src/browser.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1545,9 +1545,13 @@ const EXTRACT_CONTENT_JS_TEMPLATE: &str = r#"(() => {
}
return kept;
}
// What the table costs once rendered, matching how the tool layer prints an entry.
// What the table costs once rendered, matching how the tool layer prints it.
// The line that opens the table reaches the model with the entries, so it is part of the cost: counting entries alone put the payload over the operator's ceiling by the length of that line, 93 characters on a page served from a typical origin.
// Mirrors `render_page_body` — change one and the other has to follow.
function tableCost(kept) {
let n = 0;
if (!kept.length) return 0;
let n = '\n\nLinks (click with browser_click, e.g. ⟨1⟩)\n'.length;
if (origin) n += ('; paths are relative to ' + origin).length;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The header text now lives in three independent copies: this JS literal, the equivalent literal in render_page_body_inner (around line 1670), and the test's own copy in table_cost inside test_budget_search_fills_the_cap_without_exceeding_it (around line 2551).
The two .contains() checks added in that test only look for short prefixes — "Links (click with browser_click" and "paths are relative to" — so a future wording change to the trailing part of either literal (the e.g. ⟨1⟩) example text, or a stray extra space around ; paths are relative to) would pass those checks while silently reintroducing a length mismatch between what this function counts and what render_page_body actually renders.
That is the same failure class this PR fixes, just relocated one level down: the port and the thing it ports agreeing by construction rather than by a check that would fail if they diverged.
Worth considering deriving the header text from a single Rust-side constant and threading it into the template the way __MAX_CONTENT_CHARS__ already is, or, short of that, tightening the .contains() checks to the full literal string rather than a prefix.


Generated by Claude Code

for (const e of kept) n += ('⟨' + e.id + '⟩ ' + e.url + '\n').length;
return n;
}
Expand Down Expand Up @@ -1892,6 +1896,13 @@ mod tests {
/// Ported from the JS the script runs, because the traversal itself needs a live DOM.
#[test]
fn test_nested_lists_keep_their_own_bullets() {
// The port models the script; this checks the script still does what is modelled.
// Reverting the fold to the two-filter form would leave the port validating itself and passing.
assert!(
EXTRACT_CONTENT_JS_TEMPLATE.contains("emit((opened ? ' ' : '- ') + own, true)"),
"the item's own text must be emitted in runs, so a sub-list keeps its place between them"
);

/// A list item as the page writes it: runs of its own text and sub-lists, in source order.
enum Part {
Text(&'static str),
Expand Down Expand Up @@ -2001,6 +2012,16 @@ mod tests {
/// Anchoring on the first article's ancestors instead of searching the tree misses a grid that sits beside a featured card rather than under it.
#[test]
fn test_root_selection_finds_a_container_of_repeated_cards() {
// As above: the port is only evidence while it and the script agree on the rule.
assert!(
EXTRACT_CONTENT_JS_TEMPLATE.contains("candidates.push({ node, carriers })"),
"selection must collect every qualifying node, not stop at the first branch that has one"
);
assert!(
EXTRACT_CONTENT_JS_TEMPLATE.contains("c.node.contains(o.node)"),
"a candidate containing another must lose to it, which is what keeps selection off the whole page"
);

struct Node {
tag: &'static str,
name: &'static str,
Expand Down Expand Up @@ -2477,8 +2498,22 @@ mod tests {
///
/// Ported from the JS the script runs, because the extraction itself needs a live browser — same shape as `test_truncation_marker_fits_inside_the_cap`.
/// The subtract-the-overflow form this replaces stayed under the cap but landed at 34k against 50k on a Wikipedia-shaped page, because cutting prose drops markers and so drops table entries faster than the prose itself shrank.
///
/// The ceiling is asserted against what `render_page_body` actually produces rather than against the ported cost function.
/// Re-deriving the cost on both sides is self-consistent by construction, so it cannot see the budget and the renderer disagreeing — which is exactly what happened: the budget counted entries and the renderer also emitted a line introducing them, putting the payload 93 characters past the operator's number.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This diff only touches mod tests. EXTRACT_CONTENT_JS_TEMPLATE's own tableCost() (around line 1549) still sums only the per-link entries:

function tableCost(kept) {
    let n = 0;
    for (const e of kept) n += ('⟨' + e.id + '⟩ ' + e.url + '\n').length;
    return n;
}

It never adds the header line ("\n\nLinks (click with browser_click, e.g. ⟨1⟩)" plus "; paths are relative to {origin}" when set) that render_page_body_inner renders alongside it — the exact gap this PR's description measures at 50,093 vs. the 50,000 cap.

The test's local table_cost port here was updated to add that header cost, and the assertion now checks against the real render_page_body, which is a good change on its own. But because the production attempt()/tableCost() in the JS template wasn't updated to match, the test's budget search is no longer a port of what actually runs in the browser — it validates a hypothetical corrected budget function against render_page_body, not the real extraction script's output. Passing this test doesn't demonstrate the live overshoot is fixed; a live Wikipedia extraction against this branch should still land at ~50,093 for a 50,000 cap.

Looks like the intended change to the JS tableCost() didn't make it into the commit. Could you add the header-cost addition there (mirroring render_page_body_inner's format exactly, including the origin/base line) and re-verify against a live page the way you did for the numbers in the PR description?


Generated by Claude Code

#[test]
fn test_budget_search_fills_the_cap_without_exceeding_it() {
// The port below models the script; this checks the script still does what is modelled.
// A port is only evidence about the script while the two agree, and nothing else here would notice if the header cost were dropped from the template — the port would go on validating a corrected budget against `render_page_body` and pass.
assert!(
EXTRACT_CONTENT_JS_TEMPLATE.contains("Links (click with browser_click"),
"the script's own table cost must include the line that opens the table, or this test measures something the browser never runs"
);
assert!(
EXTRACT_CONTENT_JS_TEMPLATE.contains("paths are relative to"),
"that line carries the origin, whose length varies per page and so must be measured rather than assumed"
);

/// One `⟨n⟩` marker per line, so a cut drops markers the way a real page does.
fn page(n_links: usize, prose_len: usize, url_len: usize) -> (String, Vec<String>) {
let links: Vec<String> = (0..n_links)
Expand All @@ -2500,34 +2535,58 @@ mod tests {
let keep = budget.saturating_sub(marker.chars().count());
format!("{}{marker}", text.chars().take(keep).collect::<String>())
}
fn table_cost(text: &str, links: &[String]) -> usize {
(0..links.len())
/// The extraction's own view of what the table costs, header included.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness: this PR only patches the Rust-side test's local reimplementation of the budget search — it never touches the production EXTRACT_CONTENT_JS_TEMPLATE, which is the code that actually runs in the browser and enforces max_content_chars at runtime.

Look at tableCost inside the template (same file, lines 1549-1553):

function tableCost(kept) {
    let n = 0;
    for (const e of kept) n += ('⟨' + e.id + '⟩ ' + e.url + '\n').length;
    return n;
}

and the search that uses it (lines 1569-1584, attempt/best) — neither was changed by this diff. It still sums only per-entry cost, with no header line, exactly as before. The header itself is added downstream, once, by render_page_body_inner (lines 1666-1670), which is called on every real extraction result via browser_tools.rs:21.

So the fixed table_cost here (with header accounted for) and the render_page_body-based assertion model a hypothetical corrected algorithm that the shipped JS does not implement. The test now passes, but real browser extractions will still overshoot max_content_chars by the header line's length — the exact 93-char-on-Wikipedia overshoot this PR's own description reports — because the JS budget search that decides how much prose/table to keep never learns about the header cost.

The pre-existing test_link_table_is_budgeted_against_the_cap (line 2465) still asserts the literal, unmodified JS string "out.length + tableCost(kept)", which corroborates that the production side was never touched.

This looks like the PR title/changelog claim a behavioral fix ("max_content_chars now bounds the link table's opening line... so the extraction stays inside the ceiling") that the diff does not actually deliver — the JS template needs the header's length folded into its own tableCost/attempt before this is true.


Generated by Claude Code

fn table_cost(text: &str, links: &[String], origin: &str) -> usize {
let entries: usize = (0..links.len())
.filter(|i| text.contains(&format!("\u{27e8}{}\u{27e9}", i + 1)))
.map(|i| {
format!("\u{27e8}{}\u{27e9} {}\n", i + 1, links[i])
.chars()
.count()
})
.sum()
.sum();
if entries == 0 {
return 0;
}
let mut header = "\n\nLinks (click with browser_click, e.g. \u{27e8}1\u{27e9})\n"
.chars()
.count();
if !origin.is_empty() {
header += format!("; paths are relative to {origin}").chars().count();
}
entries + header
}
/// The extraction's response for a given prose cut, as `render_page_body` receives it.
fn response(content: &str, links: &[String], origin: &str) -> serde_json::Value {
let kept: Vec<serde_json::Value> = (0..links.len())
.filter(|i| content.contains(&format!("\u{27e8}{}\u{27e9}", i + 1)))
.map(|i| serde_json::json!({"id": i + 1, "url": links[i]}))
.collect();
serde_json::json!({"content": content, "links": kept, "links_base": origin})
}
/// `(total, was_truncated)` — a page that fits outright is not expected to fill the cap.
fn budgeted(content: &str, links: &[String], cap: usize) -> (usize, bool) {
/// `(rendered length, was_truncated)` — a page that fits outright is not expected to fill the cap.
fn budgeted(content: &str, links: &[String], cap: usize, origin: &str) -> (usize, bool) {
let rendered = |budget: usize| {
let out = cut(content, budget);
render_page_body(&response(&out, links, origin))
.chars()
.count()
};
let total = |budget: usize| {
let out = cut(content, budget);
out.chars().count() + table_cost(&out, links)
out.chars().count() + table_cost(&out, links, origin)
};
if total(cap) <= cap {
return (total(cap), false);
return (rendered(cap), false);
}
let (mut lo, mut hi, mut best) = (0usize, cap, total(0));
let (mut lo, mut hi, mut best) = (0usize, cap, rendered(0));
for _ in 0..24 {
if lo >= hi {
break;
}
let mid = (lo + hi).div_ceil(2);
let t = total(mid);
if t <= cap {
best = t;
if total(mid) <= cap {
best = rendered(mid);
lo = mid;
} else {
hi = mid - 1;
Expand All @@ -2544,10 +2603,11 @@ mod tests {
(2000, 200_000, 1_000, 80),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The four cases here all clear the cap with comfortable margin: the live Wikipedia measurement in the PR description lands 19 characters under 50,000, and the synthetic cases are sized similarly loosely, well inside the 95% tolerance band the truncated branch checks.
None of the four pins the specific edge this PR fixes: a case where the header cost is what tips a link in or out, i.e. where the budgeted total sits within one entry's length of the cap once the header is included.
A fifth, deterministic case would cover that directly: one link, a fixed short URL, and cap set to exactly prose_len + header_len_with_origin + entry_len (93 chars for the https://en.wikipedia.org origin used here, per the PR's own measurement) so the link exactly fits, plus a second cap one less than that so it exactly doesn't — asserting the entry survives in the first case and is dropped with no header charged in the second.
That would catch a header-cost regression of even a single character, which the current loose-tolerance assertions would not.


Generated by Claude Code

] {
let (content, links) = page(n_links, prose_len, url_len);
let (total, truncated) = budgeted(&content, &links, cap);
// A real origin, since the table's opening line carries it and so its length counts against the cap.
let (total, truncated) = budgeted(&content, &links, cap, "https://en.wikipedia.org");
assert!(
total <= cap,
"cap {cap}: prose plus table is {total} chars, past the ceiling the operator set"
"cap {cap}: the rendered page is {total} chars, past the ceiling the operator set"
);
// Only where the page actually overflows: the HN-shaped row above fits whole at 23,904 and must not be padded out to the cap to satisfy this.
if truncated {
Expand Down Expand Up @@ -2594,6 +2654,12 @@ mod tests {
/// Ported from the JS the script runs, because the extraction itself needs a live browser: an operator sizing `max_content_chars` to a context window gets a real ceiling, and the marker reports the pre-truncation length so the model can tell how much it is missing rather than only that something was lost (#6624).
#[test]
fn test_truncation_marker_fits_inside_the_cap() {
// As above: this port predates the others and had nothing tying it to the script either.
assert!(
EXTRACT_CONTENT_JS_TEMPLATE.contains("... (truncated, ' + total + ' chars total)"),
"the marker must report the pre-truncation length, which is what makes it worth its own characters"
);

fn truncate(content: &str, cap: usize) -> String {
if content.chars().count() <= cap {
return content.to_string();
Expand Down