Skip to content

fix: make text parsers safe for long lines#128

Merged
siyul-park merged 16 commits into
mainfrom
fix-long-lines-109
Jul 6, 2026
Merged

fix: make text parsers safe for long lines#128
siyul-park merged 16 commits into
mainfrom
fix-long-lines-109

Conversation

@siyul-park

@siyul-park siyul-park commented Jul 6, 2026

Copy link
Copy Markdown
Owner

Summary

  • Configure bufio.Scanner buffers directly in instr.ParseAll and program.Parse with a 1 MiB maximum line size.
  • Keep the parser line-size policy local to each parser; removed the separate internal/textparse helper.
  • Wrap oversized-line scanner failures with an actionable minivm parser error.
  • Add regression coverage for long valid lines and oversized input in instruction and program parsers.

Fixes #109.

Validation

  • Not run locally: this environment cannot clone the repository through the terminal; changes were applied and inspected through the GitHub connector.

Summary by CodeRabbit

  • Bug Fixes
    • Parsing now enforces a maximum input line/token size to prevent failures on extremely large lines.
    • Oversized input now fails with clearer, line-specific error messages indicating the line exceeded the maximum allowed size.
    • Very long code lines with trailing whitespace are handled correctly and continue to parse successfully.
    • Line numbers reported for errors are now consistent across parsing phases.

@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Adds explicit bufio.Scanner line-size caps to instr.ParseAll and program.Parse, reports oversized lines with line-aware errors, and extends both parsers’ regression coverage for long and oversized inputs.

Changes

Scanner buffer size fix

Layer / File(s) Summary
instr.ParseAll line limit handling
instr/parse.go, instr/parse_test.go
Adds maxParseLineBytes, configures scanner.Buffer(...), tracks line numbers, returns a line-specific oversized-input error, keeps the comment-only symbol edits, and adds multi-line long/oversized parsing tests.
program.Parse line limit handling
program/parse.go, program/parse_test.go
Adds maxParseLineBytes, configures scanner.Buffer(...), preserves line numbering across both scan phases, returns line-specific oversized-input errors, and adds long/oversized code-line tests.

Estimated code review effort: 2 (Simple) | ~12 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant Parser as instr.ParseAll / program.Parse
  participant Scanner as bufio.Scanner

  Caller->>Parser: parse input reader
  Parser->>Scanner: Buffer(buf, maxParseLineBytes)
  loop scan lines
    Scanner-->>Parser: Scan() token
    Parser->>Parser: increment line counter
  end
  Parser->>Scanner: Err()
  alt token too long
    Scanner-->>Parser: "token too long"
    Parser-->>Caller: line N exceeds maximum allowed size
  else other/no error
    Scanner-->>Parser: raw error or nil
    Parser-->>Caller: raw error or success
  end
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The changes meet #109 by configuring scanner buffers in both parsers, adding clear oversized-line errors, and covering long-line regressions.
Out of Scope Changes check ✅ Passed No clear unrelated changes are introduced; the comment edits and test additions support the parser-size limit work.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Title check ✅ Passed The title clearly states the main change: making text parsers safe for long lines.
Description check ✅ Passed It covers the key changes, links issue #109, and includes validation; only the template headings differ from the repository's format.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix-long-lines-109

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Jul 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 68.75000% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 46.83%. Comparing base (c84dd91) to head (ed0c69c).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
program/parse.go 60.00% 4 Missing ⚠️
instr/parse.go 83.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #128      +/-   ##
==========================================
+ Coverage   46.80%   46.83%   +0.02%     
==========================================
  Files          77       77              
  Lines       17340    17352      +12     
==========================================
+ Hits         8116     8126      +10     
- Misses       8306     8309       +3     
+ Partials      918      917       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/textparse/scanner.go`:
- Around line 19-25: LineError currently drops line-number context for every
scanner error except bufio.ErrTooLong, regressing the messages produced by
instr.ParseAll and program.Parse. Update LineError to wrap all non-nil errors
with the line prefix while still special-casing ErrTooLong for the max-size
message, and use errors.Is instead of direct equality when checking
bufio.ErrTooLong.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e022795b-333f-49e1-a0c5-97e1023ce3b2

📥 Commits

Reviewing files that changed from the base of the PR and between d764182 and 22e132e.

📒 Files selected for processing (5)
  • instr/parse.go
  • instr/parse_test.go
  • internal/textparse/scanner.go
  • program/parse.go
  • program/parse_test.go

Comment thread internal/textparse/scanner.go Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
instr/parse.go (1)

151-154: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Prefer errors.Is over string matching for the scanner error.

strings.Contains(err.Error(), "token too long") is fragile compared to checking against the exported bufio.ErrTooLong sentinel error with errors.Is.

♻️ Proposed fix
 import (
 	"bufio"
+	"errors"
 	"fmt"
 	"io"
 	"math"
 	"strconv"
 	"strings"
 )
 ...
 	if err := scanner.Err(); err != nil {
-		if strings.Contains(err.Error(), "token too long") {
+		if errors.Is(err, bufio.ErrTooLong) {
 			return nil, fmt.Errorf("line %d exceeds maximum allowed size of %d bytes", line, maxParseLineBytes)
 		}
 		return nil, err
 	}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@instr/parse.go` around lines 151 - 154, The scanner error check in parse
logic relies on string matching, which should be replaced with the exported
sentinel error check. Update the error handling around the line-size validation
in the parsing flow to use errors.Is against bufio.ErrTooLong instead of
strings.Contains(err.Error(), "token too long"), while keeping the existing
wrapped message for the oversized line case and preserving the fallback return
of the original error.
program/parse.go (1)

39-42: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Same fragile string match for scanner error, occurring twice in this file.

Both scan phases detect the oversized-line condition via strings.Contains(err.Error(), "token too long"). Use errors.Is(err, bufio.ErrTooLong) instead, which is more robust and idiomatic for this stdlib sentinel error.

♻️ Proposed fix
 import (
 	"bufio"
+	"errors"
 	"fmt"
 	"io"
 	"strings"

 	"github.com/siyul-park/minivm/instr"
 	"github.com/siyul-park/minivm/types"
 )
 ...
 	if err := scanner.Err(); err != nil {
-		if strings.Contains(err.Error(), "token too long") {
+		if errors.Is(err, bufio.ErrTooLong) {
 			return nil, fmt.Errorf("line %d exceeds maximum allowed size of %d bytes", lineNum+1, maxParseLineBytes)
 		}
 		return nil, err
 	}
 ...
 	if err := scanner.Err(); err != nil {
-		if strings.Contains(err.Error(), "token too long") {
+		if errors.Is(err, bufio.ErrTooLong) {
 			return nil, fmt.Errorf("line %d exceeds maximum allowed size of %d bytes", lineNum+1, maxParseLineBytes)
 		}
 		return nil, err
 	}

Also applies to: 84-87

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@program/parse.go` around lines 39 - 42, The oversized-line handling in
parse.go is using a fragile string check for the scanner error in both scan
phases. Update the error handling in the relevant parsing logic around the
scanner/line processing paths to use errors.Is with bufio.ErrTooLong instead of
strings.Contains(err.Error(), "token too long"), and apply the same change
wherever this check appears in the file so the behavior stays consistent.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@instr/parse.go`:
- Around line 151-154: The scanner error check in parse logic relies on string
matching, which should be replaced with the exported sentinel error check.
Update the error handling around the line-size validation in the parsing flow to
use errors.Is against bufio.ErrTooLong instead of strings.Contains(err.Error(),
"token too long"), while keeping the existing wrapped message for the oversized
line case and preserving the fallback return of the original error.

In `@program/parse.go`:
- Around line 39-42: The oversized-line handling in parse.go is using a fragile
string check for the scanner error in both scan phases. Update the error
handling in the relevant parsing logic around the scanner/line processing paths
to use errors.Is with bufio.ErrTooLong instead of strings.Contains(err.Error(),
"token too long"), and apply the same change wherever this check appears in the
file so the behavior stays consistent.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 7e319d49-f2be-4c44-84fb-4011dafd033e

📥 Commits

Reviewing files that changed from the base of the PR and between 22e132e and e66b3ea.

📒 Files selected for processing (4)
  • instr/parse.go
  • instr/parse_test.go
  • program/parse.go
  • program/parse_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • instr/parse_test.go

@siyul-park
siyul-park merged commit 3edddac into main Jul 6, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make text parsers safe for large instruction and program lines

1 participant