Skip to content

expose the offset of a zipfile from the start of the file as a public API #84481

Open
Open
expose the offset of a zipfile from the start of the file as a public API#84481
Labels
3.14bugs and security fixesstdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

Description

@massimosala
massimosala
mannequin
opened on Apr 16, 2020
BPO 40301
Nosy @Yhg1s, @stevendaprano, @serhiy-storchaka, @danifus, @massimosala
Files
  • py27_zipfile.patch: patch for Python 2.7.17
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2020-04-16.13:59:27.010>
labels = ['type-feature', 'library', '3.9']
title = 'zipfile module: new feature (two lines of code), useful for test, security and forensics'
updated_at = <Date 2020-04-18.14:35:44.170>
user = 'https://github.com/massimosala'

    bugs.python.org fields:

    activity = <Date 2020-04-18.14:35:44.170>
actor = 'massimosala'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-04-16.13:59:27.010>
creator = 'massimosala'
dependencies = []
files = ['49067']
hgrepos = []
issue_num = 40301
keywords = ['patch']
message_count = 13.0
messages = ['366597', '366601', '366685', '366689', '366690', '366695', '366709', '366711', '366712', '366713', '366714', '366717', '366720']
nosy_count = 6.0
nosy_names = ['twouters', 'alanmcintyre', 'steven.daprano', 'serhiy.storchaka', 'dhillier', 'massimosala']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue40301'
versions = ['Python 3.9']

    Linked PRs

    Activity

    1. massimosala commented on Apr 16, 2020

      @massimosala
      massimosalamannequin
      on Apr 16, 2020
      MannequinAuthor

      module zipfile

      Tag "Components": I am not sure "Library (Lib)" is the correct one. If it isn't, please fix.

      I use python to check zip files against malware.
      In these files the are binary blobs outside the ZIP archive.
      The malware payload isn't inside the ZIP file structure.
      Example: a file "openme.zip" with this content :
      [blob from offset 0 to offset 5678]
      [ZIP archive from offset 5679 to end of file]

      zipfile already handles this, finding the ZIP structure inside the file.

      My change is just to add a new public property, to expose an internal variable: the file offset of the ZIP structure.

      I know, I am after the code freeze of Python 2.7.18.
      But the change is really trivial, see the diff.
      I hope you can approve this patch for all the Python versions, also for 2.7, to have consistency. For 2.7 this is the last call.

    2. massimosala
      added
      3.7 (EOL)end of life
      3.8 (EOL)end of life
      3.9 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      type-featureA feature request or enhancement
      on Apr 16, 2020
    3. massimosala
      changed the title [-]zipfile module: new feature (two lines of code)[/-] [+]zipfile module: new feature (two lines of code), useful for test, security and forensics[/+] on Apr 16, 2020
    4. massimosala
      changed the title [-]zipfile module: new feature (two lines of code)[/-] [+]zipfile module: new feature (two lines of code), useful for test, security and forensics[/+] on Apr 16, 2020

    5. stevendaprano commented on Apr 16, 2020

      @stevendaprano
      stevendaprano
      on Apr 16, 2020
      Member

      This is a new feature and cannot be added to older versions which are in feature-freeze.

      Adding the feature to (say) Python 2.7.18 would be inconsistent, because it wouldn't exist in 2.7.0 through .17. Likewise for all the other versions before 3.9.

      Personally, this sounds like a nice feature to have, and your use-case sounds convincing to me.

    6. stevendaprano
      removed
      3.7 (EOL)end of life
      3.8 (EOL)end of life
      on Apr 16, 2020

    7. massimosala commented on Apr 17, 2020

      @massimosala
      massimosalamannequin
      on Apr 17, 2020
      MannequinAuthor

      Hi Steven

      Every software "ecosystem" has its guidelines and I am a newbie about
      python development.

      Mmh I see your concerns. I agree about your deletions of all py 3 versions
      before the latest 3.9.

      About Py 2, I remark these facts:

      • there are a lot of forensics tools still written for py 2;
      • python 2.7.18 will be forever the last python 2 and I think is it fine to
        end-users to have zipfile with this feature both in py 2.7 and py 3.9;
      • in the code there isn't any new routine to test: the change is just to
        expose one internal variable.

      I agree my request is an exception but I think you have to agree this
      situation is exceptional.
      IMHO rules must exist to help us and I think this request doesn't carry any
      burden.

      I ask you please

      • to reconsider my request
      • anyway, to put me in contact with zipfile mainteners, I don't know how to
        reach them but I want to hear them about this.

      Many thanks, Massimo

    8. danifus commented on Apr 18, 2020

      @danifus
      danifusmannequin
      on Apr 18, 2020
      Mannequin

      Could something similar be achieved by looking for the earliest file header offset?

      def find_earliest_header_offset(zf):
    earliest_offset = None
    for zinfo in zf.infolist():
        if earliest_offset is None:
            earliest_offset = zinfo.header_offset
        else:
            earliest_offset = min(zinfo.header_offset, earliest_offset)
    return earliest_offset

      You could also adapt this using

      zinfo.compress_size + len(zinfo.FileHeader())

      to see if there were any sections inside the archive which were not referenced from the central directory. Not sure if zip files with arbitrary bytes inside the archive would be valid everywhere, but I think they are using zipfile.

      You can also have zipped content inside an archive which has a valid fileheader but no reference from the central directory. Those entries are discoverable by implementations which process content serially from the start of the file but not implementations which rely on the central directory.

    9. 38 remaining items

    10. hroncok commented on Apr 14, 2025

      @hroncok
      hroncok
      on Apr 14, 2025
      Contributor

      Do the test files need to be executable?

    11. vstinner commented on Apr 14, 2025

      @vstinner
      vstinner
      on Apr 14, 2025
      Member

      Python and Python test suite don't need the script to be executable, but you can run these 2 test scripts.

      Example:

      $ Lib/test/archivetestdata/exe_with_zip $PWD/python
Opening Lib/test/archivetestdata/exe_with_zip as a zipfile.
Favorite number in executable: 5

    12. befeleme commented on Apr 14, 2025

      @befeleme
      befeleme
      on Apr 14, 2025
      Contributor

      Would you accept a PR removing the executable bit? (I don't know if Fedora is the only distribution mangling the shebangs, but I imagine this can unexpectedly hit other downstream packagers).

    13. hroncok commented on Apr 14, 2025

      @hroncok
      hroncok
      on Apr 14, 2025
      Contributor

      Python and Python test suite don't need the script to be executable...

      Turns out test_execute_zip64 and other similar tests need that.

    14. emmatyping
      added 2 commits that reference this issue on Jul 21, 2025

      Revert "pythongh-84481: Make ZipFile.data_offset more robust (python#…

      8eb63c9

      Revert "pythongh-84481: Add ZipFile.data_offset attribute (python#132165

      4221022
    15. gpshead
      added a commit that references this issue on Jul 21, 2025

      gh-136170: Revert adding `ZipFile.data_offset` (GH-136950)

      6bf1c0a
    16. miss-islington
      added a commit that references this issue on Jul 21, 2025

      pythongh-136170: Revert adding `ZipFile.data_offset` (pythonGH-136950)

      c87fb19

    17. gpshead commented on Jul 22, 2025

      @gpshead
      gpshead
      on Jul 22, 2025
      Member

      reopening as we pulled .data_offset from 3.14 (see PR and issues).

    18. gpshead
      reopened this on Jul 22, 2025
    19. gpshead
      added a commit that references this issue on Jul 22, 2025

      [3.14] gh-136170: Revert adding `ZipFile.data_offset` (GH-136950) (#1…

      caef946
    20. danifus
      marked Alter ZipFile.data_offset calculation #136170 as a duplicate of this issue on Jul 22, 2025
    21. taegyunkim
      added a commit that references this issue on Aug 4, 2025

      pythongh-136170: Revert adding `ZipFile.data_offset` (pythonGH-136950)

      e589644
    22. Agent-Hellboy
      added a commit that references this issue on Aug 19, 2025

      pythongh-136170: Revert adding `ZipFile.data_offset` (pythonGH-136950)

      81c41c6
    23. kumaraditya303
      added a commit that references this issue on Sep 9, 2025

      [3.14] pythongh-136170: Revert adding `ZipFile.data_offset` (pythonGH…

      6239ef8
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.14bugs and security fixesstdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

      Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions