feat(pypi): add support for JSON-based Simple API (PEP 691)

[felipecrs]

Changes

Renovate has currently two methods for looking up pypi updates:

1. https://pypi.org/simple/<package>, which only supports listing versions
2. https://pypi.org/pypi/<package>/json, which supports releaseTimestamp, and sourceUrl detection

The problem is:

For method 2, Renovate relies on the releases key in the JSON response. The releases key is deprecated (ref), and https://pypi.org/pypi/<package>/json may drop it at any time. When it does, this method will stop working entirely.

This is already the case for some private registries like Artifactory. For Artifactory remote repositories, the releases key is still there since it proxies https://pypi.org/pypi/<package>/json. But for local repositories, Artifactory already omits releases from the response, making Renovate unable to lookup releases in such case.

The only alternative for now is to use registryUrl=https://example.com/artifactory/api/pypi/example-pypi-local/simple/ instead, but then Renovate will not extract release timestamps (meaning minimumReleaseAge won't work).

I believe the ultimate solution for this is:

1. Support JSON-based Simple API (docs) (introduced as PEP 691) when resolving /simple/<package> .

   • This is what this PR does.
   • It includes upload-time which can be used to detect releaseTimestamp.
   • https://pypi.org/simple/<package> already supports it.
   • Artifactory supports it on an opt-in basis.

2. Refactor PyPI JSON to extract release metadata from https://pypi.org/pypi/<package>/json/<package>/<version> (docs)

   • I.e. instead of making a single request to https://pypi.org/pypi/<package>/json, first retrieve the available versions from the PyPI Simple (as suggested by their documentation), and then query every individual release through https://pypi.org/pypi/<package>/json/<package>/<version>
   • Not as great as before but, well, that's life.

As mentioned above, this PR implements solution 1. If the Simple API returns JSON upon request, we use it and extract releaseTimestamps. If not, we fallback to HTML parsing like before.

Relates to #20070

PS: I started working on this PR before realizing #41916, of which the author stated that he had a working PR (#44182). Apologies!

Context

Please select one of the following:

• This closes an existing Issue, Closes:
• This doesn't close an Issue, but I accept the risk that this PR may be closed if maintainers disagree with its opening or implementation

AI assistance disclosure

Did you use AI tools to create any part of this pull request?

Please select one option and, if yes, briefly describe how AI was used (e.g., code, tests, docs) and which tool(s) you used.

• No — I did not use AI for this contribution.
• Yes — minimal assistance (e.g., IDE autocomplete, small code completions, grammar fixes).
• Yes — substantive assistance (AI-generated non‑trivial portions of code, tests, or documentation).
• Yes — other (please describe):

Claude Opus 4.6 was used to generate the initial code changes.

Documentation (please check one with an [x])

• I have updated the documentation, or
• No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

• Code inspection only, or
• Newly added/modified unit tests, or
• No unit tests, but ran on a real repository, or
• Both unit tests + ran on a real repository

The public repository: https://github.com/felipecrs/renovate-repro-pip-pep-691

Important

To verify this against https://pypi.org/simple/, this line needs to be changed in lib/modules/datasource/pypi/index.ts:

    const hostUrl = ensureTrailingSlash(
-    registryUrl!.replace('https://pypi.org/simple', 'https://pypi.org/pypi'),
+    registryUrl!,
   );

I believe we should consider removing that from Renovate as a separate PR, as that prevents users from setting their own registryUrl=https://pypi.org/simple/. As mentioned above, retrieving releases from https://pypi.org/pypi/ may stop working at any time.

[nvoq-lblaney]

I swear @felipecrs, everyone I go on GitHub, I run into you =). You might remember me by my non-work user @shadycuz. Thanks for this fix, this issue has been a headache at work.

[felipecrs]

Of course I remember you! From groovy-guru!

[psalaberria002]

Looking forward to this one!

[ahippler]

same as #44182. I’ll close my PR so @viceice has fewer to review.

[RahulGautamSingh]

Thanks for the improvements!

Code looks good to me and approach 1 is the way to go, we have not implemented approach 2 for other datasources as well, as it needs too many API requests.

[felipecrs]

Thanks a lot for the review, @RahulGautamSingh!

But I'll incorporate some changes from @ahippler's #44182 and add him as co-author. Marking as draft for now.

EDIT: Done.

[RahulGautamSingh]

lgtm otherwise.