Improve resolution speed of version ranges in remotes using the name as pattern by memsharded · Pull Request #5433 · conan-io/conan

memsharded · 2019-07-02T18:19:09Z

Changelog: Fix: Improve resolution of version ranges for remotes
Docs: omit

Partially address, in the client side #5432

Separate work must be done in Artifactory

Mark-Hatch-Bose · 2019-07-02T18:59:12Z

I installed this from your branch, am not seeing a difference in search times to Artifactory.
Maybe I'm doing something wrong?

{"_action": "REST_API_CALL", "duration": 3.1186702251434326, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.20.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562093776.1418612, "url": "<REMOTE>/artifactory/api/conan/barbarian/v2/conans/search?ignorecase=False&q=<PACKAGE>%2F%2A%40BoseCorp%2Fbose-stable"}
{"_action": "REST_API_CALL", "duration": 3.0984485149383545, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.20.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562093779.2458003, "url": "<REMOTE>/artifactory/api/conan/barbarian/v2/conans/search?ignorecase=False&q=<PACKAGE>%2F%2A%40BoseCorp%2Fbose-master"}
{"_action": "REST_API_CALL", "duration": 3.133497953414917, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.20.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562093782.3835235, "url": "<REMOTE>/artifactory/api/conan/barbarian/v2/conans/search?ignorecase=False&q=><PACKAGE>%2F%2A%40BoseCorp%2Fmaster"}
{"_action": "REST_API_CALL", "duration": 3.0755865573883057, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.20.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562093785.4605715, "url": "<REMOTE>/artifactory/api/conan/barbarian/v2/conans/search?ignorecase=False&q=<PACKAGE>%2F%2A%40BoseCorp%2Fbose-master"}
{"_action": "REST_API_CALL", "duration": 3.1029679775238037, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.20.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562093788.5661237, "url": "<REMOTE>/artifactory/api/conan/barbarian/v2/conans/search?ignorecase=False&q=><PACKAGE>%2F%2A%40BoseCorp%2Fmaster"}
{"_action": "REST_API_CALL", "duration": 3.113391399383545, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.20.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562093791.682815, "url": "<REMOTE>/artifactory/api/conan/barbarian/v2/conans/search?ignorecase=False&q=><PACKAGE>%2F%2A%40BoseCorp%2Fmaster"}
{"_action": "REST_API_CALL", "duration": 3.3839454650878906, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.20.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562093795.0717094, "url": "<REMOTE>/artifactory/api/conan/barbarian/v2/conans/search?ignorecase=False&q=><PACKAGE>%2F%2A%40BoseCorp%2Fmaster"}
{"_action": "REST_API_CALL", "duration": 3.1916117668151855, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.20.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562093798.2700784, "url": "<REMOTE>/artifactory/api/conan/barbarian/v2/conans/search?ignorecase=False&q=><PACKAGE>%2F%2A%40BoseCorp%2Fmaster"}
{"_action": "REST_API_CALL", "duration": 3.2044949531555176, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.20.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562093801.4781427, "url": "<REMOTE>/artifactory/api/conan/barbarian/v2/conans/search?ignorecase=False&q=><PACKAGE>%2F%2A%40BoseCorp%2Fmaster"}
{"_action": "REST_API_CALL", "duration": 3.16910719871521, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.20.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562093804.6536212, "url": "<REMOTE>/artifactory/api/conan/barbarian/v2/conans/search?ignorecase=False&q=><PACKAGE>%2F%2A%40BoseCorp%2Fmaster"}
{"_action": "REST_API_CALL", "duration": 3.162350654602051, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.20.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562093807.8195848, "url": "<REMOTE>/artifactory/api/conan/barbarian/v2/conans/search?ignorecase=False&q=><PACKAGE>%2F%2A%40BoseCorp%2Fmaster"}
{"_action": "REST_API_CALL", "duration": 3.284928798675537, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.20.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562093811.1089065, "url": "<REMOTE>/artifactory/api/conan/barbarian/v2/conans/search?ignorecase=False&q=><PACKAGE>%2F%2A%40BoseCorp%2Fmaster"}

memsharded · 2019-07-02T20:34:37Z

Uhm, yes, I still see the full package reference in your URLs, instead of the shorter one that should result. No idea why, I guess that you have already checked, but just in case, could you add a couple of prints() to the modified code in the PR in your clone, to make sure it is entering there?

Mark-Hatch-Bose · 2019-07-02T20:47:19Z

I'm a fool, I hadn't checked out your branch after cloning... 💯
I see MUCH improved performance now!

{"_action": "REST_API_CALL", "duration": 0.09672117233276367, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.21.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562100269.735404, "url": "<URL_REDACTED>/artifactory/api/conan/barbarian/v2/conans/search?q=<PACKAGE>ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 0.11811661720275879, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.21.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562100269.8555326, "url": "<URL_REDACTED>/artifactory/api/conan/barbarian/v2/conans/search?q=<PACKAGE>ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 0.04911184310913086, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.21.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562100269.910386, "url": "<URL_REDACTED>/artifactory/api/conan/barbarian/v2/conans/search?q=<PACKAGE>ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 0.05893063545227051, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.21.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562100269.9704657, "url": "<URL_REDACTED>/artifactory/api/conan/barbarian/v2/conans/search?q=<PACKAGE>ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 0.05200672149658203, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.21.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562100270.0241113, "url": "<URL_REDACTED>/artifactory/api/conan/barbarian/v2/conans/search?q=<PACKAGE>&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 0.052623748779296875, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.21.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562100270.07776, "url": "<URL_REDACTED>/artifactory/api/conan/barbarian/v2/conans/search?q=<PACKAGE>&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 0.059350013732910156, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.21.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562100270.138302, "url": "<URL_REDACTED>/artifactory/api/conan/barbarian/v2/conans/search?q=<PACKAGE>&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 0.05178999900817871, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.21.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562100270.1919434, "url": "<URL_REDACTED>/artifactory/api/conan/barbarian/v2/conans/search?q=<PACKAGE>&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 0.05158591270446777, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.21.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562100270.24533, "url": "<URL_REDACTED>/artifactory/api/conan/barbarian/v2/conans/search?q=<PACKAGE>&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 0.044409751892089844, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.21.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562100270.2912815, "url": "<URL_REDACTED>/artifactory/api/conan/barbarian/v2/conans/search?q=<PACKAGE>&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 0.04635310173034668, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.21.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562100270.3388534, "url": "<URL_REDACTED>/artifactory/api/conan/barbarian/v2/conans/search?q=<PACKAGE>ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 0.0608220100402832, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.21.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562100270.400658, "url": "<URL_REDACTED>/artifactory/api/conan/barbarian/v2/conans/search?q=<PACKAGE>ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 0.06620454788208008, "headers": {"Authorization": "**********", "User-Agent": "Conan/1.17.0-dev (Python 3.5.2) python-requests/2.21.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "<USER>"}, "method": "GET", "time": 1562100270.469506, "url": "<URL_REDACTED>/artifactory/api/conan/barbarian/v2/conans/search?q=<PACKAGE>ignorecase=False"}

memsharded · 2019-07-02T21:19:23Z

Ok, thanks for the update @Mark-Hatch-Bose, those are great news!

Conan 1.17 is about to be closed, so it seems almost impossible that it will make it, but lets wait for @lasote feedback.

lasote · 2019-07-03T06:18:21Z

From 3 secs to 0.1! Wow, such an improvement cannot be delayed to 1.18 ;)

memsharded added 2 commits July 2, 2019 14:14

faster version ranges resolution in remotes

7c1534f

faster version

a931e17

memsharded mentioned this pull request Jul 2, 2019

Performance of version ranges search worse than regular search #5432

Closed

fix broken tests

d50f074

memsharded assigned lasote Jul 2, 2019

lasote added this to the 1.17 milestone Jul 3, 2019

lasote merged commit db0e7c3 into conan-io:develop Jul 3, 2019

memsharded deleted the feature/faster_version_ranges branch July 3, 2019 13:35

jgsogo mentioned this pull request Aug 28, 2020

Unable to connect to conan-center #7573

Closed

maikelvdh mentioned this pull request Sep 7, 2020

[bug] Search call is leaking package name details to Conan Center Index #7672

Closed

czoido mentioned this pull request Sep 8, 2020

Use patterns in server query when resolving version ranges #7673

Merged

5 tasks

Navigation Menu

Improve resolution speed of version ranges in remotes using the name as pattern - #5433

Improve resolution speed of version ranges in remotes using the name as pattern#5433
lasote merged 3 commits into
conan-io:developfrom
memsharded:feature/faster_version_ranges

memsharded commented Jul 2, 2019

Uh oh!

Mark-Hatch-Bose commented Jul 2, 2019

Uh oh!

memsharded commented Jul 2, 2019

Uh oh!

Mark-Hatch-Bose commented Jul 2, 2019

Uh oh!

memsharded commented Jul 2, 2019

Uh oh!

lasote commented Jul 3, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

memsharded commented Jul 2, 2019

Uh oh!

Mark-Hatch-Bose commented Jul 2, 2019

Uh oh!

memsharded commented Jul 2, 2019

Uh oh!

Mark-Hatch-Bose commented Jul 2, 2019

Uh oh!

memsharded commented Jul 2, 2019

Uh oh!

lasote commented Jul 3, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants