Skip to content

HEAD requests should be HEAD requests upon redirect #99730

Closed
Closed
HEAD requests should be HEAD requests upon redirect#99730
Labels
stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@haampie
haampie
opened on Nov 23, 2022

Bug report

Currently the following is False

from urllib.request import Request, urlopen

len(urlopen(Request("http://google.com", method="HEAD")).read()) == 0  # False

But this is True

len(urlopen(Request("http://www.google.com", method="HEAD")).read()) == 0 # True

This is because http://google.com redirects with 302 to http://www.google.com.

This means that checking for existence of some file by URL will actually download the file when the URL responds with a redirect. This makes no sense. Also the HTTP spec says nothing about changing HEAD requests into GET requests; it just says that everything but GET and HEAD requests should require user interaction on redirect, which Python violates, but there's a comment on that explaining it's an active choice to violate the spec there.

To me it seems like this is an oversight. Note that curl -LI http://google.com also sticks to HEAD requests.

Linked PRs

Activity

  1. haampie
    added
    type-bugAn unexpected behavior, bug, or error
    on Nov 23, 2022

  2. bcail commented on Oct 11, 2023

    @bcail
    bcail
    on Oct 11, 2023

    @haampie is there anything I can do to help with this issue/PR?

  3. haampie commented on Oct 12, 2023

    @haampie
    haampie
    on Oct 12, 2023
    ContributorAuthor

    I guess it should be compared to other tools. A while after reporting this issue and implementing my own redirect handler, I hit some issue where sending a HEAD request to a server that redirects to an AWS bucket with temporary credentials would only work if the redirect was using a GET request, and error with Unauthorized if it was HEAD. Dunno how common this is.

  4. bcail commented on Oct 12, 2023

    @bcail
    bcail
    on Oct 12, 2023

    OK, here's confirmation that the requests package stays with HEAD:

    >>> r = requests.head('http://google.com', allow_redirects=True)
>>> r.url
'http://www.google.com/'
>>> len(r.content)
0

    And I'm seeing curl stay with the HEAD request as well.

    Here's an MDN doc that doesn't come down strongly on one side or the other.

  5. bcail commented on Oct 24, 2023

    @bcail
    bcail
    on Oct 24, 2023

    @orsenthil any thoughts on this issue, whether the request handling can be updated to not change a HEAD to a GET on a redirect?

  6. iritkatriel
    added
    stdlibStandard Library Python modules in the Lib/ directory
    on Nov 28, 2023

  7. orsenthil commented on May 1, 2024

    @orsenthil
    orsenthil
    on May 1, 2024
    Member

    @orsenthil any thoughts on this issue, whether the request handling can be updated to not change a HEAD to a GET on a redirect?

    The PR #99731 is approved. We will merge this change, and make the behavior change.

  8. encukou
    added a commit that references this issue on May 1, 2024

    gh-99730: urllib.request: Keep HEAD method on redirect (GH-99731)

    759e8e7

  9. encukou commented on May 1, 2024

    @encukou
    encukou
    on May 1, 2024
    Member

    Merged; it'll be in 3.13.
    Since this is a behaviour change, I don't think backporting to maintenance branches is appropriate.

  10. encukou
    closed this as completedon May 1, 2024
  11. SonicField
    added a commit that references this issue on May 8, 2024

    pythongh-99730: urllib.request: Keep HEAD method on redirect (pythonG…

    1a1771b

  12. Jean-Luc-Picard-2021 commented on Oct 1, 2024

    @Jean-Luc-Picard-2021
    Jean-Luc-Picard-2021
    on Oct 1, 2024

    Ok, cool, will test later!

    The change might be incomplete, since it only keeps HEAD.
    But the protocoll is such that all methods are preserved,
    not only HEAD, but also DELETE, PUT, etc..

    And there are cases where HEAD is indeed promoted to GET.
    It all depends on the particular 30x response status.

    See also here:

    Note: In HTTP/1.0, the status codes 301 (Moved Permanently) and 302 (Found) were originally defined as method-preserving ([HTTP/1.0], Section 9.3) to match their implementation at CERN;

    303 (See Other) was defined for a redirection that changed its method to GET. However, early user agents split on whether to redirect POST requests as POST (according to then-current specification) or as GET (the safer alternative when redirected to a different site). Prevailing practice eventually converged on changing the method to GET.

    307 (Temporary Redirect) and 308 (Permanent Redirect) [RFC7538] were later added to unambiguously indicate method-preserving redirects, and status codes 301 and 302 have been adjusted to allow a POST request to be redirected as GET.

    And here:

    https://stackoverflow.com/questions/77600025/how-does-http-redirect-work-with-http-methods

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions