Skip to content

Tarfile is unnecessarily slow #121267

Closed
Closed
Tarfile is unnecessarily slow#121267
Labels
performancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

Description

@jforberg
jforberg
opened on Jul 2, 2024

Bug report

Bug description:

There is room for improvement in tarfile write performance. In a simple benchmark I find that tarfile spends most of its time doing repeated user name/group name queries.
https://gist.github.com/jforberg/86af759c796199740c31547ae828aef2

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

Activity

  1. jforberg
    added
    type-bugAn unexpected behavior, bug, or error
    on Jul 2, 2024
  2. jforberg
    added 2 commits that reference this issue on Jul 2, 2024

    pythongh-121267: Improve performance of tarfile (python#121267)

    fb0b6c3

    pythongh-121267: Improve performance of tarfile (python#121267)

    41397ce

  3. gaogaotiantian commented on Jul 2, 2024

    @gaogaotiantian
    gaogaotiantian
    on Jul 2, 2024
    Member

    Could you paste your cProfile output? I was a bit surprised that most of the time is spent on reading the pwd and grp.

  4. jforberg commented on Jul 2, 2024

    @jforberg
    jforberg
    on Jul 2, 2024
    ContributorAuthor

    @gaogaotiantian Here is the output:

    cprofile.txt

  5. jforberg
    added 2 commits that reference this issue on Jul 3, 2024

    pythongh-121267: Improve performance of tarfile (python#121267)

    8d2f912

    pythongh-121267: Improve performance of tarfile (python#121267)

    c5eee91
  6. hauntsaninja
    added a commit that references this issue on Oct 30, 2024

    gh-121267: Improve performance of tarfile (#121267) (#121269)

    2b2d607
  7. ethanfurman
    closed this as completedon Oct 30, 2024
  8. picnixz
    added
    type-featureA feature request or enhancement
    performancePerformance or resource usage
    stdlibStandard Library Python modules in the Lib/ directory
    and removed
    type-bugAn unexpected behavior, bug, or error
    on Oct 30, 2024

  9. picnixz commented on Oct 30, 2024

    @picnixz
    picnixz
    on Oct 30, 2024
    Member

    Recategorizing this issue as per #121269 (comment).

  10. picnixz
    added a commit that references this issue on Dec 8, 2024

    pythongh-121267: Improve performance of tarfile (python#121267) (pyth…

    4d78977
  11. ebonnal
    added a commit that references this issue on Jan 12, 2025

    pythongh-121267: Improve performance of tarfile (python#121267) (pyth…

    d57b9a4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions