Skip to content

Allow foreachref parse max tokens from 4*64KB to 4MB #36414

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lunny merged 4 commits into from
Jan 21, 2026
Merged

Allow foreachref parse max tokens from 4*64KB to 4MB #36414

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
02b102f
Allow foreachref parse max token from 4*64KB to 16*64KB
lunny Jan 20, 2026
d9ff35a
Update modules/git/foreachref/parser.go
lunny Jan 20, 2026
1732829
fix
wxiaoguang Jan 21, 2026
a1640e2
Merge branch 'main' into lunny/grow_foreachref_maxtoken
lunny Jan 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions modules/git/foreachref/parser.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,11 @@ type Parser struct {
func NewParser(r io.Reader, format Format) *Parser {
scanner := bufio.NewScanner(r)

// default MaxScanTokenSize = 64 kiB may be too small for some references,
// so allow the buffer to grow up to 4x if needed
scanner.Buffer(nil, 4*bufio.MaxScanTokenSize)
// default Scanner.MaxScanTokenSize = 64 kiB may be too small for some references,
// so allow the buffer to be large enough in case the ref has long content (e.g.: a tag with long message)
// as long as it doesn't exceed some reasonable limit (4 MiB here, or MAX_DISPLAY_FILE_SIZE=8MiB), it is OK
// there are still some choices: 1. add a config option for the limit; 2. don't use scanner and write our own parser to fully handle large contents
scanner.Buffer(nil, 4*1024*1024)

// in addition to the reference delimiter we specified in the --format,
// `git for-each-ref` will always add a newline after every reference.
Expand Down