I will pick this up and get it merged in a bit. Some notes for myself:

The current logic uses IPAddress.Any, which is strictly an ipv4 address under the hood. This should be IPAddress.IPv6Any, or the current options.Listen() call should be changed to options.ListenAnyIP() which appears to be the better of the two options:

    /// <summary>
    /// Listens on all IPs using IPv6 [::], or IPv4 0.0.0.0 if IPv6 is not supported.
    /// </summary>
    public void ListenAnyIP(int port, Action<ListenOptions> configure)
    {
        ArgumentNullException.ThrowIfNull(configure);

        var listenOptions = new AnyIPListenOptions(port);
        ApplyEndpointDefaults(listenOptions);
        configure(listenOptions);
        CodeBackedListenOptions.Add(listenOptions);
    }

If a user wanted to listen on just localhost the guidance would be to supply the proper loopback address for their system (::1 or 127.0.0.1), which should be reasonable if a user has a strong opinion about this.

The logic in the kestrel config is then:

If the user has specified any value for ip_address, use options.Listen(<provided ip4/6 address>), otherwise use options.ListenAny()

The only uncovered use case would be instances where the user wanted to listen on multiple, specific addresses but not 'any', in which case another code/config change would be needed. When CIDRs can be supplied for API keys or the relay, they can be supplied as a comma separated list. I suppose this option should have that ability as well.

The option will need a custom validator that iterates over the supplied addresses to make sure they are valid.