Repository navigation
argparse.FileType opens a file and never closes it #58032
Description
| BPO | 13824 |
|---|---|
| Nosy | @merwok, @mitar, @matrixise, @MojoVampire, @remilapeyre, @septatrix |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2012-01-19.12:25:39.321>
labels = ['type-bug', 'library']
title = 'argparse.FileType opens a file and never closes it'
updated_at = <Date 2021-07-25.09:27:11.812>
user = 'https://bugs.python.org/DavidLayton'bugs.python.org fields:
activity = <Date 2021-07-25.09:27:11.812>
actor = 'Nils Kattenbeck'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2012-01-19.12:25:39.321>
creator = 'David.Layton'
dependencies = []
files = ['31852']
hgrepos = []
issue_num = 13824
keywords = ['patch']
message_count = 15.0
messages = ['151615', '152518', '152519', '152530', '166068', '198336', '198376', '215298', '341670', '341761', '341764', '341852', '342153', '342503', '398184']
nosy_count = 13.0
nosy_names = ['bethard', 'eric.araujo', 'mitar', 'Paolo.Elvati', 'manveru', 'Stefan.Pfeiffer', 'paul.j3', 'David.Layton', 'matrixise', 'josh.r', 'remi.lapeyre', 'sebix', 'Nils Kattenbeck']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13824'
versions = ['Python 2.7']Linked PRs
Metadata
Metadata
Assignees
Projects
- Status
Activity
paulj3 commentedon May 11, 2019
While I continue to follow argparse issues, I'm not doing development (with my own repository, etc).
I haven't revisited the issue since 2013, and don't know that much more about Python file handling. Occasionally 'FileType' issues come up on StackOverflow, and my most common suggestion is accept a string, and open the file yourself.
In writing a solution, keep backward compatibility foremost in mind. We don't want to mess things up for existing code. And keep FileType simple, consistent with Steven's original intent - as a tool for simple input/output scripts.
As for the idea of
2 - Making the argument namespace itself support context management
someone could experiment with a new Namespace class with the proper enter/exit methods. The use of a custom Namespace class is documented
https://docs.python.org/3/library/argparse.html#the-namespace-object
So the idea can be implemented and thoroughly tested without altering the default Namespace class and its use. (With one caution, subparsers won't use this custom class.)
matrixise commentedon May 14, 2019
Hi,
Are you interested to write test cases? They could be useful for the fix.
Thank you
Septatrix commentedon Jul 25, 2021
A good alternative would be to use pathlib.Path. The only downside would be that one has to manually handle
-but besides that it solves most problems.Still the fact that file descriptors are kept open should be added as a warning to the docs
merwok commentedon Jul 6, 2022
How so? FileType lets you specify that the argument should be readable or writable, so this would be a loss of functionality.
serhiy-storchaka commentedon Jan 1, 2024
The current design has too many flaws. Open files are leaked if
argparseitself fails or the user code fails or quits before it has chance to close all files. Output files are always rewritten in case of error. Input files can remain open for too long time and consume the limited resource of file descriptors. And solving it in the user code is difficult even if we ignore errors that can occur during parsing (see for example the pickletools CLI #85567).All solutions (making file opening lazy (and maybe explicit) or making the parser or the namespace object a context manager) require changing the user code. Minimizing changes in the user code increases chance of writing user code that does not do all thing properly.
On other hand, explicit opening and closing files is not much more complex that proper use of FileType or its replacement. The only non-trivial thing that FileType does in comparison with the plain open() is returning the standard stream (depending on mode) if the argument is "-". We can provide an utility function for this, although the code is only non-trivial because it is general, for any concrete mode string it is simpler.
I propose to deprecate
argparse.FileType. Its existence in the stdlib implies that it is a preferable way to handle file arguments, but it is not so. You have less issues if you do not use it.gvanrossum commentedon Jan 1, 2024
+1. Well argued. I’m not even sure we need a helper for ‘-‘ ; that API would be a bit convoluted, since closing stdin is fraught.
pythongh-58032: Do not use argparse.FileType in module CLIs and scripts
gh-58032: Do not use argparse.FileType in module CLIs and scripts (GH…
pythongh-58032: Do not use argparse.FileType in module CLIs and scrip…
pythongh-58032: Do not use argparse.FileType in module CLIs and scrip…
pythongh-58032: Do not use argparse.FileType in module CLIs and scrip…
picnixz commentedon Sep 7, 2024
@serhiy-storchaka Should we keep this one opened to deprecate
FileTypeor do you want to close it (and not deprecate FileType for now)?pythongh-58032: Deprecate the argparse.FileType type converter
pythongh-58032: Deprecate the argparse.FileType type converter
serhiy-storchaka commentedon Sep 27, 2024
#124664 deprecates
argparse.FileType. Since it is widely used, no warning is emitted at runtime. A runtime warning will be added later, perhaps in 3.16, and finally it will be removed.Mariatta commentedon Sep 27, 2024
Sorry, the sentence is a bit ambiguous.
Do you mean to add the warning in 3.16 (so that the earliest removal is at least 3.18), or to have it finally removed in 3.16?
Mariatta commentedon Sep 27, 2024
If say, we have it removed in 3.18, does it mean we will keep this ticket open for 4 years more from now?
Mariatta commentedon Sep 27, 2024
I gathered that we just need to add this to
pending-removal-in-future.rstand then we can close this ticket (after the PR is merged).serhiy-storchaka commentedon Sep 28, 2024
Sorry for ambiguous wording. I meant that in future (perhaps in 3.16) we will add a runtime warning and start normal deprecation period, so the feature could be removed two years later, in 3.18. This way authors will have more time to remove the usage of FileType and do not annoy users with deprecation warnings.
Mariatta commentedon Sep 28, 2024
Thank you, the timeline is clearer to me now.
I'm wondering two things:
serhiy-storchaka commentedon Oct 10, 2024
These are very good questions.
I am afraid of the amount of user code this may affect.
FileTypeis no longer directly used in the Python stdlib or scripts, but it is used in the CLI of some vendor packages (charset,charset_normalizer,distlib) used to build the documentation. It may add some noise in the build output if they are used via a CLI. I am sure thatFileTypeis used in many user scripts despite its flaws. For simple script it may be not critical that it leaks or clamps files on errors or closes stdin or stdout just before quitting. Deprecation warning will add a noise in normal cases, when all works.This is why I wanted to first make the deprecation silent, documentation-only. To give more time to authors without annoying users. This would also send signal to linters. I forgot about PendingDeprecationWarning, it mitigates the issue with annoying warnings.
I will add PendingDeprecationWarning. But now I need to modify a number of tests in test_argparse to silence the warning or check for it. My laziness was the other reason.
hugovk commentedon Oct 10, 2024
Searching the top 8k PyPI projects finds 384 matching lines in 138 projects for
argparse.*FileType, which is ~1.7%.Details
gh-58032: Deprecate the argparse.FileType type converter (GH-124664)
gh-58032: Docs: Sort argparse alphabetically (GH-125871)
pythongh-58032: Docs: Sort argparse alphabetically (pythonGH-125871)
pythongh-58032: Deprecate the argparse.FileType type converter (pytho…
pythongh-58032: Docs: Sort argparse alphabetically (pythonGH-125871)
Remove use of argparse.FileType
Remove use of argparse.FileType
Remove use of argparse.FileType
Remove use of argparse.FileType
Do not use argparse.FileType
henryiii commentedon Feb 6, 2026
The docs don't really have an example for how to migrate away from this and keep
-support. Having an example with a common mode or two would be helpful, I think. pylint detects usage of this now (code W4904), by the way, which is helpful.drcraig commentedon May 10, 2026
I'm giving a talk on argparse at PyCon US next week (last minute addition from the waitlist, https://us.pycon.org/2026/schedule/presentation/165/, still hasn't appeared on the master schedule on the website as of 5/9), and I just now learned of the deprecation while reading the docs to prepare the talk. I'm trying now to think of a good alternative to advise people to use for the use case where a CLI program wants to read either one or more files or default to stdin.
The use cases to replace are the 0 or 1 files version
and the similar 0 or many files version
The first thing I can think of is to have a helper function that you use to open the file, or pass through the existing stdin/out/err file object. Something like
Maybe also need to pair that with a couple helpers to use with the
typeargument ofadd_argumentthat will handle translating "-" to sys.stdin or out. Something likeSo you'd use it like
Any reactions to that? I hesitate to stand up in front of a crowd and promote something I just made up and haven't used yet. But we need something to suggest to people now that the old way is officially deprecated.
MojoVampire commentedon May 16, 2026
@drcraig: I'm inclined to ask that the old way be undeprecated. I actually wrote my own subclass of
argparse.FileTypeat work that works withcontextlib.ExitStack, and it's extremely usable, aside from the newPendingDeprecationWarning, and integrating the feature intoFileTypeitself is easy now that I've already done it once and explored the edge cases. Usage would boil down to:The behavior varies slightly for receiving
-instead of a filename, only insofar as, instead ofFileType.__call__invokingself._stack.enter_contexton the result of anopencall, it instead invokesself._stack.callback(lambda: fileobj.closed or fileobj.flush())when opened for write (and does nothing at all when it's opened for read), so sys.stdout (or its buffer, based on the mode) is guaranteed to be flushed (if not already closed) when the with block is exited.The two main objections I can see are:
contextlibandargparse) as the recommended way of usingargparse. Butcontextlib.ExitStackis an amazing tool that does its one job very well, and reinventing it piecemeal seems wrong, and beyond that, we're not averse to providing recipes combining modules elsewhere in the docs (theitertoolsrecipes at this point are built on top ofitertools,collections,contextlib,functools,heapq,math, andoperatorat this point).The one thing it doesn't handle is cleaning up when a file is opened for write, and then a subsequent argument parsing failure closes it while leaving the empty file on disk. I'm largely okay with that; the user does know what they typed in after all, so the mystery file appearing isn't confusing.
MojoVampire commentedon May 19, 2026
Opened gh-150125 to propose restoring
FileType.fix: Cease usage of argparse.FileType
fix: Cease usage of argparse.FileType
fix: Cease usage of argparse.FileType
fix: Cease usage of argparse.FileType (#170)