argparse: Problem with defaults for variable nargs when using choices

@rhettinger

BPO	9625
Nosy	@rhettinger, @macfreek, @ericvsmith, @merwok, @cedk, @berkerpeksag, @lanzz, @jameshcorbett, @zumoshi
Files	issue9625.diff issue9625.patch issue9625_1.patch issue9625_2.patch notes.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2010-08-17.09:19:03.116>
labels = ['type-bug', 'library', '3.9']
title = 'argparse: Problem with defaults for variable nargs when using choices'
updated_at = <Date 2020-12-21.23:36:46.142>
user = 'https://bugs.python.org/thesociable'

bugs.python.org fields:

activity = <Date 2020-12-21.23:36:46.142>
actor = 'rhettinger'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2010-08-17.09:19:03.116>
creator = 'thesociable'
dependencies = []
files = ['26471', '27858', '30709', '30762', '35128']
hgrepos = []
issue_num = 9625
keywords = ['patch']
message_count = 17.0
messages = ['114108', '151780', '151807', '151857', '166086', '174642', '191224', '191932', '192258', '192716', '217677', '285865', '347742', '356859', '357064', '374581', '374592']
nosy_count = 16.0
nosy_names = ['rhettinger', 'bethard', 'macfreek', 'eric.smith', 'eric.araujo', 'ced', 'Sworddragon', 'thesociable', 'regis', 'berker.peksag', 'paul.j3', 'sebix', 'lanzz', 'Jan Huta\xc5\x99', 'jameshcorbett', 'zumoshi']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9625'
versions = ['Python 3.9']

Linked PRs

gh-53834: Fix support of arguments with choices in argparse #124495

Variable argument count plays badly with choices.

Example:
========

>>> import argparse
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('choices', nargs='*', default='a', choices=['a',
'b', 'c'])
>>> args = parser.parse_args()
>>> print type(args.choices)
<type 'str'>
>>> args = parser.parse_args(['a'])
>>> print type(args.choices)
<type 'list'>

If the user specifies the value on the command line then a list is used, but if the value comes from the default a string is used.
Unfortunately, changing default to a list value gives an error:
error: argument choices: invalid choice: ['a'] (choose from 'a', 'b',
'c')

Additionally, this means it is also not possible to use default=['a', 'c'].

The current workaround is to create a custom type:

def my_type(string):
    if string not in ['a', 'b', 'c']:
        raise TypeError
    return string

Maybe it will sound strange, but what is this task REALLY about? I mean - I can see two problems here, but no clear information about which problem is a real problem and - if it is - what is the expected behavior.

Problems I can see are:

Type of returned value changes depending on the value source (list for user provided or list for default)
It's impossible to set list as 'default'

I understand that this task concentrates on 1st one, however changing behavior described in 2nd would - in my oppinion - fix 1st one too - am I right? User would "define" the returned type by defining the 'default' as a list or string.

But - if we assume that we only discuss 1st problem here - what is the expected behavior? Provided workaround is suggesting that we expect string, but - basing on current nargs behavior and "intuition" - I'd rather expect to get a list, when I define nargs with "*" or "+". This sounds "natural" for me.

Could someone explain me the problem and the expected behavior in a clear way?

Sorry if I'm not clear, but it's my first post here and maybe I've missed something in the description or I assume something that is not true (especially that the task is quite old... ;) ).

Of course I've made a mistake:

"list for user provided or list for default"

should be:

"list for user provided or STRING for default"

The real issue is that the choices flag does not work with a default flag and * nargs.

The following works as expected:
>>> parser.add_argument('chosen', nargs='*', default=['a'])
>>> print(parser.parse_args())
Namespace(chosen=['a'])
>>> print(parser.parse_args(['a', 'b']))
Namespace(chosen=['a', 'b'])

Introducing a choices constraint breaks down when using the defaults:
>>> parser.add_argument('chosen', nargs='*', default=['a'], choices=['a', 'b'])
>>> print(parser.parse_args(['a']))
Namespace(chosen=['a'])
>>> print(parser.parse_args())
error: argument chosen: invalid choice: ['a'] (choose from 'a', 'b')

I would expect instead to have Namespace.chosen populated with the default list as before, but the choices constraint check does not validate correctly.

I think that changing the choices constraint logic to iterate over the default values if nargs results in a list would be a possible solution.

I agree that this looks like a bug. I think the fix is something like the attached patch, but it needs some tests to make sure that it fixes your problem.

Here is a new version of the patch with tests

ping

I've added 2 more tests,

one with default='c', which worked before.

one with default=['a','b'], which only works with this change.

http://bugs.python.org/issue16878 is useful reference, since it documents
the differences between nargs="?" and nargs="*", and their handling of
their defaults.

Change "choices='abc'" to "choices=['a', 'b', 'c']", as discussed in bpo-16977 (use of string choices is a bad example)

The patch I just posted to http://bugs.python.org/issue16468 uses this fix.

There's a complicating issue - should these default values be passed through the type function?

In most cases in _get_values, the string first goes through _get_value, and then to _check_value.

For example the 'else:' case:

            value = [self._get_value(action, v) for v in arg_strings]
            for v in value:
                self._check_value(action, v)

The '*' positional case could coded the same way, allowing:

parser.add_argument('foo',
     nargs='*',
     type=int,
     choices=range(5),
     default=['0',1,'2'])

and objecting to

         default=[6,'7','a'] # out of range string or int or invalid value

This does impose a further constraint on the 'type' function, that it accepts a converted value. e.g. int(1) is as valid as int('1').

But we need to be careful that this case is handled in a way that is consistent with other defaults (including the recent change that delayed evaluating defaults till the end).

Paul, what is your current thinking on this?

Another workaround for who might ever need it. The benefit of this solution comparing to a custom type is that argparse will generate the help string properly with the choices, and this solution does workaround the place when the bug happens:
    class Choices(tuple):
    ...

Thanks @JooEiras, this is a very nice workaround, but I think it can be simplified:

class Choices(tuple):

    def __init__(self, _iterable=None, default=None):
        # _iterable is already handled by tuple.__new__
        self.default = default or []

    def __contains__(self, item):
        return super().__contains__(item) or item == self.default

This has the added benefit of being able to set default when creating the object, and working when omitting default=choices.default from parser.add_argument(...), if no default is set for the Choices object.

Any concerns about this change?

PS: I think Choices.__init__ is actually ran twice in the original workaround. I don't think there's a reason for this.

pythongh-53834: Fix support of arguments with choices in argparse

I think that we should allow arbitrary default and should omit checks if the default value is used. It can fail only if the programmer provides default that does not pass the check. The check does not check input, it restricts the programmer. In this case it always fail because default and choices are always incompatible if nargs='*'.

This also allows to distinguish the default value from the specified value if nargs='?'.

gh-53834: Fix support of arguments with choices in argparse (GH-124495)

Navigation Menu

Repository navigation

argparse: Problem with defaults for variable nargs when using choices #53834

Linked PRs

8 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

argparse: Problem with defaults for variable nargs when using choices #53834

Description

Linked PRs

Activity

thesociable commented on Aug 17, 2010

regis commented on Jan 22, 2012

regis commented on Jan 23, 2012

thesociable commented on Jan 23, 2012

bethard commented on Jul 21, 2012

cedk commented on Nov 3, 2012

cedk commented on Jun 15, 2013

paulj3 commented on Jun 27, 2013

paulj3 commented on Jul 3, 2013

paulj3 commented on Jul 9, 2013

paulj3 commented on May 1, 2014

8 remaining items

rhettinger commented on Jul 29, 2020

NichtJens commented on Apr 17, 2024

serhiy-storchaka commented on Sep 25, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions