A PoC patch for pip to bring this functionality in fridex/pip#1

@fridex you can now simplify the specification part by referencing https://packaging.python.org/en/latest/specifications/direct-url-data-structure/.

@fridex you can now simplify the specification part by referencing https://packaging.python.org/en/latest/specifications/direct-url-data-structure/.

Thanks, added to the references section for now - not sure if it should be directly used in the specification section as provenance_url.json diverges from direct_url.json as stated in the PEP-610/PyPA's spec page. The biggest difference are mostly fields that MUST/SHOULD be present in the provenance_url.json. See @gpshead's comment here, specifically hash and hashes differences. This might open a question whether DirectUrl should be reused in pip and whether there is a need to provide similar spec to the Direct URL Data Structure.

EDIT: The hash field is completely dropped in this PEP.

@fridex thanks for the update. I personally think it is better to reference the direct url data structure in the specification section, mentioning that only archive_info is relevant and applicable for provenance_url.json.

By reusing the same specification, implementers can safely assume they can reuse the same code without having to dissect and compare the specs. It will also simplify maintenance if/when the spec evolves.

The hash key is clearly mentioned there as deprecated so it should not be an issue.

@fridex thanks for the update. I personally think it is better to reference the direct url data structure in the specification section, mentioning that only archive_info is relevant and applicable for provenance_url.json.

By reusing the same specification, implementers can safely assume they can reuse the same code without having to dissect and compare the specs. It will also simplify maintenance if/when the spec evolves.

The hash key is clearly mentioned there as deprecated so it should not be an issue.

Thanks, this seems reasonable as it follows what is already implemented in pip and is closer to the proposed pip PR and PEP-610. On the other hand, the proposed PEP is shifting away from this based on these two discussions with @gpshead:

Old text in this draft PEP, taken from PEP-610:

• The hashes key SHOULD be present as a dictionary mapping a hash name to a hex encoded digest of the file.

A comment by @gpshead (link):

I'd change this to MUST. There is no point in this data if hash information is not available. This being a new file, it shouldn't need carry along the legacy "hash": "algorithm_name=hex_value" baggage.

Which resulted in a change to this wording:

The value of archive_info MUST be a dictionary with a single key hashes.

Old text in this draft PEP, taken from PEP-610:

When both the hash and hashes keys are present, the hash represented in
the hash key MUST also be present in the hashes dictionary, so consumers can consider the hashes key only if it is present, and fall back to hash otherwise.

A comment by @gpshead (link):

Very good! This is important. But this is the implementation complexity I'd rather we not even have. Code can be written that deals with this JSON but does not do this and does not validate equality of hashes specified in more than one place, and blindly uses the simpler hash field... which is then a security problem. get rid of the legacy field entirely in this file and the chance of that drops.

granted someone is still going to include the field. and will pass these data structures elsewhere to code that uses that instead of a hashes dict. but unless we require that code parsing these validate explicitly that the legacy "hash" key is not present, that unintended consequences bug cannot be avoided.

This led to a complete removal of the hash field.

It looks like we will need to make a consensus here.

BTW we should move this discussion to discuss.python.org as requested by Paul. It seems like it will make it easier for Paul to review it and also include other folks. I try to keep the draft PEP text up to date there.

BTW we should move this discussion to discuss.python.org

@fridex I already raised this point at https://discuss.python.org/t/draft-pep-recording-provenance-of-installed-packages/24838/15?u=sbidoul

I don't think there is any incompatibility with what @gpshead said, though:

• Regarding hash: the direct URL data structure spec already says A deprecated hash key (type string) MAY be present for backwards compatibility purposes,, so it is already implicit that new tools should not generate it. But it does not harm to reinforce this here.
• Whether hashes is mandatory is a matter of opinion I guess. I personally think it's better if direct_url.json and provenance_url.json have the same behaviour in this respect. But even if we decide that a difference is desirable that does not prevent referencing the direct url data structure spec and add an additional requirement here.

BTW we should move this discussion to discuss.python.org

@fridex I already raised this point at https://discuss.python.org/t/draft-pep-recording-provenance-of-installed-packages/24838/15?u=sbidoul

I don't think there is any incompatibility with what @gpshead said, though:

• Regarding hash: the direct URL data structure spec already says A deprecated hash key (type string) MAY be present for backwards compatibility purposes,, so it is already implicit that new tools should not generate it. But it does not harm to reinforce this here.
• Whether hashes is mandatory is a matter of opinion I guess. I personally think it's better if direct_url.json and provenance_url.json have the same behaviour in this respect. But even if we decide that a difference is desirable that does not prevent referencing the direct url data structure spec and add an additional requirement here.

Let's have a sync once the PEP is actually submitted. I kept the hash key out of the spec and included it in the rejected ideas section. The hashes field is mandatory in the spec now. The compatibility with direct_url.json might be nice, then the question might be enforcing availability of these fields, which might be worth discussion.

@sbidoul thanks again for your very valuable feedback!

Replaced with python#3076