Multi-value options override the value returned from callback when not all specified

[mxndtaylor]

I discovered this when I wanted to add a flag_value-style option to shortcut a custom str parameter option.

I want to have 2 options:

1. one that accepts a parameter of any str value like the name of a file or something
2. another that is a flag that fetches the value from a remote server instead of me specifying it

Since these both represent the same thing, my initial thought was just to pass them under the same parameter name to the function, which seems to almost work.

the problem seems to be that if the "custom" value is not specified while the "flag" option is set, then click will use the flag_value from the "flag" as the default value for the "custom" value.

And click seems to process unspecified options last (which I can see why this was done), but since "custom" is not specified, it processes the default after the "flag" option has already set a value for the function parameter that they share.

# bad.py
import click

SENTINEL = "$_fetch"

def fetch():
    # some logic dependent on an external resource
    # return requests.get("...").json()["value")
    print('resource fetched')
    return "foo"

def callback(ctx, param, value):
    if value is SENTINEL:
        return fetch()
    return value

@click.command()
@click.option("--custom", "custom")
@click.option("--fetch", "custom", flag_value=SENTINEL, callback=callback)
def cli(custom):
    print(f'custom={custom}')

if __name__ == '__main__':
    cli()

results in the SENTINEL internal value being passed to the function when --fetch is specified alone:

# call with just --custom <arg>, get <arg>, as expected
$ python bad.py --custom bar
custom=bar

# call with just --fetch, get SENTINEL, the internal value that should not ever appear in the values
$ python bad.py --fetch
resource fetched
custom=$_fetch

# call with --custom <arg> first and --fetch after, get the fetched resource, as expected
$ python bad.py --custom bar --fetch
resource fetched
custom=foo

It's that last use-case that really rubs the dirt in; paradoxically, I have to specify the longer option I was trying to avoid using in order to get my "shortcut" flag option to work.

However, we can use a workaround to fix it, sort of, by passing the same callback to both options:

# bad.py
# ..
@click.option("--custom", "custom", callback=callback)
@click.option("--fetch", "custom", flag_value=SENTINEL, callback=callback)
# ..

results in fetch being called twice when --fetch is the last specified option:

# call with just --custom <arg>, get <arg>, as expected
$ python bad.py --custom bar
custom=bar

# call with just --fetch, get foo, but the external resource is pinged twice
$ python bad.py --fetch
resource fetched
resource fetched
custom=foo

# call with --custom <arg> first and --fetch after, get the fetched resource, as expected, resource is pinged once
$ python bad.py --custom bar --fetch
resource fetched
custom=foo

I could have some caching mechanism to avoid the repeated call to the external resource, but that should not be necessary, I should not have to add the callback argument to both options.

I can write a custom Option class that resolved this though:

# good.py
import click

class FixOption(click.Option):
    def handle_parse_result(self, ctx, opts, args):
        val = super().handle_parse_result(ctx, opts, args)
        # even though opts is type-hinted as non-mutable, this still works
        opts[self.name] = val[0]
        return val

SENTINEL = "$_fetch"

def fetch():
    # some logic dependent on an external resource
    # return requests.get("...").json()["value")
    print('resource fetched')
    return "foo"

def callback(ctx, param, value):
    if value is SENTINEL:
        return fetch()
    return value

@click.command()
@click.option("--custom", "custom")
@click.option("--fetch", "custom", flag_value=SENTINEL, callback=callback, cls=FixOption)
def cli(custom):
    print(f'custom={custom}')

if __name__ == '__main__':
    cli()

But I'd prefer not to have to keep a whole class around just for this, especially when it does work without one, just only if I pass both my options.

Environment:

• Python version: 3.12.5
• Click version: 8.1.7

[mxndtaylor]

just to emphasize where the root problem seems to be to me:

the problem seems to be that if the "custom" value is not specified while the "flag" option is set, then click will use the flag_value from the "flag" as the default value for the "custom" value.

I'm not really sure where this is happening in the library, or why it isn't finding the value returned by the callback instead.