Repository navigation
IDLE editor crashes opening a file with million+ character lines #103089
Description
Bug report
IDLE crashes if you try to open a .py file with ridiculously long nested dicts in it.
To reproduce
- Copy and paste the contents of this file into a
.pyfile: repro.txt - Attempt to open the file with IDLE.
Observed behaviour
The first time I tried doing this, IDLE opened the file, but then immediately crashed. All subsequent times I have tried doing this, Python has hanged on me requesting IDLE to open the file. (I don't know how long it hangs for; I've interrupted it using CTRL-C every time.)
Comments
This is pretty weird code. The reason why it's so weird is that it's partially generated code: it's the result of a codemod by stubdefaulter.
IDLE is far from the only tool that crashes or hangs when encountering this code. This code also causes black, flake8 and mypy to hang indefinitely. VSCode does open the file, and has no performance issues, but it takes a very long time for pyright's analysis of the file to complete in the background when you open the file with VSCode.
Your environment
- CPython versions tested on:
- 3.11.2
- CPython
mainbranch @ 7703def
- Operating system and architecture: Windows
Linked PRs
- gh-103089: Colorize only the beginning of very long lines in IDLE #157647
- [3.15] gh-103089: Colorize only the beginning of very long lines in IDLE (GH-157647) #158021
- [3.14] gh-103089: Colorize only the beginning of very long lines in IDLE (GH-157647) #158022
- [3.13] gh-103089: Colorize only the beginning of very long lines in IDLE (GH-157647) #158023
Metadata
Metadata
Assignees
Projects
- Status
Activity
AlexWaygood commentedon Mar 28, 2023
Cc. @terryjreedy for IDLE
[-]IDLE editor crashes when trying to open a file with ridiculously long nested dicts[/-][+]IDLE editor crashes opening a file with million+ character lines[/+]terryjreedy commentedon Mar 28, 2023
The expert-IDLE tag emails me.
The file is 15 lines, not all the same length, with about 15 million chars. IDLE read() the file in iomenu.load_file, line 142, and inserts it into an empty tk text widget at line 170. Tk text widgets are known to get sluggish with lines somewhere around 2000 chars and unusable at about 10,000 chars. I expect the crash and hang are 3rd party issues. I am leaving this open to a) verify this and b) try to mitigate it a bit by checking
charsfor total length and longest and average line length and issuing a crash/hang warning when 'excessive'. (While doing so, I should check the effect of wrapping or not. Also consider same check for pasted text and reconsider squeezer check and warning.)When you say "IDLE loaded the file" did you actually see 15 lines in a normal editor window? For your experiments, was the extension .py or .txt as loaded here? I believe IDLE will highlight syntax only for the former. (15M chars should take maybe a minute.)
EDIT: Alex actually said above 'opened', not 'loaded', which opening was followed by a crash. So I asked whether anything appeared before the crash, and he says 'yes' below.
AlexWaygood commentedon Mar 28, 2023
Yes.
A
.pyextension, but GitHub doesn't let you upload.pyfiles into bug reports. (This is why I said "copy and paste the contents of this file into a.pyfile" in my instructions to reproduce the issue.)StanFromIreland commentedon Feb 9, 2025
Tested on Linux. Takes well over 20 minutes to load (I forgot when exactly I started it) but it gets there (and then its very slow...). I would vote to close as I doubt this will ever realistically pose an issue. 1984 has roughly half a million chars.
AlexWaygood commentedon Feb 9, 2025
Well, it did pose a real issue for me. I opened this issue because I ran a codemod tool, found Black hung on my code after the codemod had been applied, and then tried opening the file with IDLE to see if I could figure out why Black was hanging -- only for IDLE to then also crash/hang. This was all pretty confusing and less than ideal!
I'm not suggesting that IDLE needs to be able to actually work well as a text editor when opening files like this. But a better behaviour might be to detect if the file is too long for IDLE to realistically cope with and refuse to open it, printing an error message to the terminal. Or, alternatively, to detect if trying to open the file is taking too long and abort. There are lots of behaviours that would be better than simply hanging without any explanation!
terryjreedy commentedon Feb 10, 2025
Closing would be a legitimate response, but hanging is really annoying. So I will leave open for now.
The main issue with long lines is user code output, and we our current imperfect solution is the shell squeezer, inserted between fetching from a socket and inserting into a text widget. We could similarly insert something between
chars = f.read()(inmenu 127) andself.text.insert("1.0", chars)(170). The virtue of a timed popup,self.text.after(1500, popup), would be not having to predict a problem. But I don't know if 'after' events can trigger during an insert call.While pasting could be an issue, it is currently an atomic operation in that clipboard content is directly moved to the text widget. This might be separable, but a 15M paste first requires a 15M copy, which I think even less likely than a 15M file write.
serhiy-storchaka commentedon Sep 16, 2026
The Tk text widget itself handles such lines (inserting 15 MB with
wrap='none'takes well under a second). The hang is in the colorizer: adding a tag to a line takes time proportional to the number of tags already in the line, so colorizing a line is quadratic in the number of tokens: a 2000-character line with 400 tokens takes about 6 ms, a 10,000-character line with 2000 tokens about 100 ms, and a line with 200,000 tokens about half an hour. The file has nine such lines.#157647 colorizes only the first 2000 characters of a line. With it the file opens in about 8 s, most of which is
IndentSearchertokenizing the whole file in search of the first indented block; that can be limited separately.terryjreedy commentedon Sep 23, 2026
For first indent search, if files were read with readlines, newlines would already be detected is C and only the first char of each line would need to be checked for whitespace, until found.
gh-103089: Colorize only the beginning of very long lines in IDLE (#1…
[3.13] gh-103089: Colorize only the beginning of very long lines in I…
[3.14] gh-103089: Colorize only the beginning of very long lines in I…
[3.15] gh-103089: Colorize only the beginning of very long lines in I…