1939786 - Downloads from m-c have become very slow from 1Nov2024 to the point that doing biusection is not really possible

Status: RESOLVED WORKSFORME

(Testing :: mozregression, defect)

Description

[Mayank Bansal]

When doing bisection, the download of builds from m-c take several minutes. I am on a 150mbps broadband connection.
This is a recent regression, since maybe Dec2024.

This is a very annoying issue. It severely prohibits me from doing bisections.
Please can this be fixed.

Comment 1

[Mayank Bansal]

Attachment: about:support

Comment 2

[Mayank Bansal]

Debug log from downloading a single Win64 m-c build of 2Jan2024 :

2025-01-04T10:26:30.284000: DEBUG : redo: attempt 1/3
2025-01-04T10:26:30.284000: DEBUG : redo: retry: calling get with args: ('https://api.github.com/repos/mozilla/mozregression/releases/latest',), kwargs: {}, attempt #1
2025-01-04T10:26:30.288000: DEBUG : urllib3.connectionpool: Starting new HTTPS connection (1): api.github.com:443
2025-01-04T10:26:30.658000: DEBUG : urllib3.connectionpool: https://api.github.com:443 "GET /repos/mozilla/mozregression/releases/latest HTTP/11" 200 None
2025-01-04T10:26:36.754000: DEBUG : redo: attempt 1/3
2025-01-04T10:26:36.754000: DEBUG : redo: retry: calling get with args: ('https://hg.mozilla.org/mozilla-central/json-tags',), kwargs: {}, attempt #1
2025-01-04T10:26:36.756000: DEBUG : urllib3.connectionpool: Starting new HTTPS connection (1): hg.mozilla.org:443
2025-01-04T10:26:38.695000: DEBUG : urllib3.connectionpool: https://hg.mozilla.org:443 "GET /mozilla-central/json-tags HTTP/11" 200 None
2025-01-04T10:26:45.319000: DEBUG : Sending usage ping
2025-01-04T10:26:45.330000: DEBUG : Get URLs for 2025-01-02
2025-01-04T10:26:45.330000: DEBUG : redo: attempt 1/3
2025-01-04T10:26:45.330000: DEBUG : redo: retry: calling _default_get with args: ('https://archive.mozilla.org/pub/firefox/nightly/2025/01/',), kwargs: {'auth': None}, attempt #1
2025-01-04T10:26:45.336000: DEBUG : urllib3.connectionpool: Starting new HTTPS connection (1): archive.mozilla.org:443
2025-01-04T10:26:45.659000: DEBUG : urllib3.connectionpool: https://archive.mozilla.org:443 "GET /pub/firefox/nightly/2025/01/ HTTP/11" 200 6173
2025-01-04T10:26:45.665000: DEBUG : got build_urls ['https://archive.mozilla.org/pub/firefox/nightly/2025/01/2025-01-02-21-27-43-mozilla-central/', 'https://archive.mozilla.org/pub/firefox/nightly/2025/01/2025-01-02-09-18-53-mozilla-central/']
2025-01-04T10:26:45.666000: DEBUG : Fetching build info from https://archive.mozilla.org/pub/firefox/nightly/2025/01/2025-01-02-21-27-43-mozilla-central/
2025-01-04T10:26:45.666000: DEBUG : redo: attempt 1/3
2025-01-04T10:26:45.666000: DEBUG : redo: retry: calling _default_get with args: ('https://archive.mozilla.org/pub/firefox/nightly/2025/01/2025-01-02-21-27-43-mozilla-central/',), kwargs: {'auth': None}, attempt #1
2025-01-04T10:26:45.669000: DEBUG : Fetching build info from https://archive.mozilla.org/pub/firefox/nightly/2025/01/2025-01-02-09-18-53-mozilla-central/
2025-01-04T10:26:45.669000: DEBUG : redo: attempt 1/3
2025-01-04T10:26:45.669000: DEBUG : redo: retry: calling _default_get with args: ('https://archive.mozilla.org/pub/firefox/nightly/2025/01/2025-01-02-09-18-53-mozilla-central/',), kwargs: {'auth': None}, attempt #1
2025-01-04T10:26:45.672000: DEBUG : urllib3.connectionpool: Starting new HTTPS connection (2): archive.mozilla.org:443
2025-01-04T10:26:45.958000: DEBUG : urllib3.connectionpool: https://archive.mozilla.org:443 "GET /pub/firefox/nightly/2025/01/2025-01-02-21-27-43-mozilla-central/ HTTP/11" 200 53694
2025-01-04T10:26:45.990000: DEBUG : urllib3.connectionpool: https://archive.mozilla.org:443 "GET /pub/firefox/nightly/2025/01/2025-01-02-09-18-53-mozilla-central/ HTTP/11" 200 53694
2025-01-04T10:26:46.018000: DEBUG : got valid_builds [(0, {'build_txt_url': 'https://archive.mozilla.org/pub/firefox/nightly/2025/01/2025-01-02-21-27-43-mozilla-central/firefox-135.0a1.en-US.win64.txt', 'build_url': 'https://archive.mozilla.org/pub/firefox/nightly/2025/01/2025-01-02-21-27-43-mozilla-central/firefox-135.0a1.en-US.win64.zip'}), (1, {'build_txt_url': 'https://archive.mozilla.org/pub/firefox/nightly/2025/01/2025-01-02-09-18-53-mozilla-central/firefox-135.0a1.en-US.win64.txt', 'build_url': 'https://archive.mozilla.org/pub/firefox/nightly/2025/01/2025-01-02-09-18-53-mozilla-central/firefox-135.0a1.en-US.win64.zip'})]
2025-01-04T10:26:46.018000: DEBUG : Update build info from {'build_txt_url': 'https://archive.mozilla.org/pub/firefox/nightly/2025/01/2025-01-02-21-27-43-mozilla-central/firefox-135.0a1.en-US.win64.txt', 'build_url': 'https://archive.mozilla.org/pub/firefox/nightly/2025/01/2025-01-02-21-27-43-mozilla-central/firefox-135.0a1.en-US.win64.zip'}
2025-01-04T10:26:46.019000: DEBUG : Fetching txt info from https://archive.mozilla.org/pub/firefox/nightly/2025/01/2025-01-02-21-27-43-mozilla-central/firefox-135.0a1.en-US.win64.txt
2025-01-04T10:26:46.019000: DEBUG : redo: attempt 1/3
2025-01-04T10:26:46.019000: DEBUG : redo: retry: calling _default_get with args: ('https://archive.mozilla.org/pub/firefox/nightly/2025/01/2025-01-02-21-27-43-mozilla-central/firefox-135.0a1.en-US.win64.txt',), kwargs: {}, attempt #1
2025-01-04T10:26:46.031000: DEBUG : urllib3.connectionpool: https://archive.mozilla.org:443 "GET /pub/firefox/nightly/2025/01/2025-01-02-21-27-43-mozilla-central/firefox-135.0a1.en-US.win64.txt HTTP/11" 200 99
2025-01-04T10:26:47.215000: DEBUG : urllib3.connectionpool: https://archive.mozilla.org:443 "GET /pub/firefox/nightly/2025/01/2025-01-02-21-27-43-mozilla-central/firefox-135.0a1.en-US.win64.zip HTTP/11" 200 107834527
2025-01-04T10:28:06.055000: INFO : Running mozilla-central build for 2025-01-02
2025-01-04T10:28:08.981000: INFO : Launching C:\Users\mayan\AppData\Local\Temp\tmpxrgjsf30\firefox\firefox.exe
2025-01-04T10:28:08.985000: INFO : Application command: C:\Users\mayan\AppData\Local\Temp\tmpxrgjsf30\firefox\firefox.exe --wait-for-browser -profile C:\Users\mayan\AppData\Local\Temp\tmp8k2yjk3x.mozrunner
2025-01-04T10:28:09.007000: INFO : application_buildid: 20250102212743
2025-01-04T10:28:09.007000: INFO : application_changeset: 94c05b2b36188c127c71f09614343df6844c8f7b
2025-01-04T10:28:09.008000: INFO : application_display_name: Firefox Nightly
2025-01-04T10:28:09.008000: INFO : application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
2025-01-04T10:28:09.008000: INFO : application_name: Firefox
2025-01-04T10:28:09.008000: INFO : application_remotingname: firefox-nightly
2025-01-04T10:28:09.008000: INFO : application_repository: https://hg.mozilla.org/mozilla-central
2025-01-04T10:28:09.008000: INFO : application_vendor: Mozilla
2025-01-04T10:28:09.008000: INFO : application_version: 135.0a1
2025-01-04T10:28:09.008000: INFO : platform_buildid: 20250102212743
2025-01-04T10:28:09.008000: INFO : platform_changeset: 94c05b2b36188c127c71f09614343df6844c8f7b
2025-01-04T10:28:09.008000: INFO : platform_repository: https://hg.mozilla.org/mozilla-central
2025-01-04T10:28:09.008000: INFO : platform_version: 135.0a1

Comment 3

[Zeid [:zeid]]

Hey Mayank -- curious if you try downloading the builds directly from archive.mozilla.org, do they download faster? Just want to rule out an issue with archive.mozilla.org itself or other network issues.

Comment 4

[Mayank Bansal]

Downloads from archive.mozilla.org are faster. though still not saturate my networkds full bandwidth. I get download speed of ~1.5mbps. I have a 150mbps fiber connection.

Note teh bug i have added in see also bug 1940016.

Both these started around 1Nov2024. Note my comment in https://chat.mozilla.org/#/room/#treeherder:mozilla.org on 1Nov.
All of these issues seem to be tied up to something that changed in treeherder.

Comment 6

[Mayank Bansal]

Edited.

Comment 7

[Mayank Bansal]

WFM now.