Writeup

When I was looking into rebuilding the package fonts version 0.0.3, I noticed that it was using a version of setuptools (56.2.0) that was not available at the time of publication (2018, ~40 was setuptools latest).

Diving into this, I debugged routes being captured by timewarp and noticed that timewarp was never reached during the build. This was because there was no redirection added for pip which rerouted the pypi request.

Solving this was trivial since pip allowed for the setting of the environment variable export PIP_INDEX_URL=http://localhost:8081/ which would redirect pypi calls to timewarp.

However, this revealed some issues with the expected network responses to and from pip:

• timewarp expects pip to send pypi and time information sent in the authentication ("pypi": "##:##:####")
• pip expects timewarp to send back a list of potential releases available for a package in the Simple API but only the package page JSON is sent currently.

Below I list my proposed changes and the implementation which I used to get the time gated versions

Proposed Network Solutions

My main goal was to leave pip as unaltered as possible. Rolling out a custom pip felt like overkill, especially if we were had to revert the pip version as well.

Since I did not want to change how pip created and authenticated it's internal routes, I added a new endpoint for capturing date information

• Before doing any installs, rebuild sends a request where we set the auth to "time": "##:##:####"
  • This route returns a hash of the date as a text response
  • This hash is then appended to the PIP_INDEX_URL and later parsed when a install is required
    • e.g. pip install setuptools would become the route http://localhost:8081/<HASH>/setuptools
  • The hash added to the url gets parsed and converted to a date when the install is requested

Once the date is gathered, timewarp redirects to pypi and gets the releases from the package page json

• This is default behavior and worked well!
  • The filtering worked to come up with a list of release versions that fit within the time range
• However, sending the data back to pip had issues
  • The pip version attached to the alpine docker selected for my local runs required a Simple API response
  • Currently, timewarp sends a JSON of the package page of the latest version for that time back
    • This had issues for two reasons:
      1. pip wanted the page with downloadable packages for the versions allowed, which is different than the general pypi package page
      2. A range is requested so that pip can internally handle specified installs (e.g. setuptools==40.0.0)
• The fix for this still uses the default behavior to get the package page for releases, but now reaches out to the Simple API to get the package list html page
  • timewarp then filters the html on this list to only include the releases gathered before.
  • Sending this response to pip causes it to evaluate and download the dependency at that time frame

Proof of concept implementation

I did not create a pull request for this since a lot of the implementation is local based (for now). If you want to do some testing yourself, I can create a branch on my fork for this.

internal/timewarp/timewarp.go patch gist for better highlighting