Fix mozilla-download to use https instead of ftp

RESOLVED FIXED

Status

RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: gerard-majax, Assigned: gerard-majax)

Tracking

unspecified
ARM
Gonk (Firefox OS)
Dependency tree / graph

Firefox Tracking Flags

(blocking-b2g:2.2r+, b2g-v2.2 fixed, b2g-v2.2r fixed)

Details

Attachments

(5 attachments)

(Assignee)

Description

3 years ago
Running gaia-try on v2.2r is totally broken: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=dce50a884e3b39f1fb676cf72f4e6c62adeb6368

This is because the mozilla-download (and mozilla-get-url) there relies on the FTP protocol which was shutdown.

The master branch of mozilla-download is of no help since it downloads from TaskCluster while the B2G builds for b2g37_v2.2r branch are being produced by Buildbot. So no build available from this master either.

This technically means any uplift on Gaia for v2.2r is not able to get any test being run.
(Assignee)

Updated

3 years ago
Blocks: 1181505
Josh, this is actively blocking 2.2r patches from landing. Can you please help move it along?
Flags: needinfo?(jocheng)

Comment 2

3 years ago
Hi LISSY,
I am not sure who is the owner of mozilla-download that can help here. Do you know?
Thanks!
Flags: needinfo?(lissyx+mozillians)
(Assignee)

Comment 3

3 years ago
(In reply to Josh Cheng [:josh] from comment #2)
> Hi LISSY,
> I am not sure who is the owner of mozilla-download that can help here. Do
> you know?
> Thanks!

If I'd knew I would have already pinged them.
Flags: needinfo?(lissyx+mozillians)
This is blocking a sec-moderate security fix backport for 2.2 of bug 1181571, and I have another fix backport incoming. Should these be landed blindly?
Blocks: 1181571
(Assignee)

Comment 6

3 years ago
Blocking KillSwitch ...
Flags: needinfo?(vkrishnamoorthy)
Flags: needinfo?(mtreese)
Chris, is this something you can help with?
Flags: needinfo?(catlee)
I'm not familiar with mozilla-download, but I've been told that perhaps Henrik can help.
Flags: needinfo?(catlee) → needinfo?(hskupin)
I don't know what mozilla-download is, but I assume you mean mozdownload? If it does not work for you I assume you are using a version <1.17. I would encourage you to upgrade to v1.17 which is not using ftp.mozilla.org anymore but archive.mozilla.org, and it works perfectly:

$ mozdownload -t daily
 INFO | Retrieving the build status file from https://archive.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/
 INFO | Retrieving list of builds from https://archive.mozilla.org/pub/mozilla.org/firefox/nightly/2015/09
 INFO | Found 1 build: 2015-09-01-03-02-26-mozilla-central
 INFO | Selected build: 2015-09-01-03-02-26-mozilla-central
 INFO | Downloading from: https://archive.mozilla.org/pub/mozilla.org/firefox/nightly/2015/09/2015-09-01-03-02-26-mozilla-central/firefox-43.0a1.en-US.linux-x86_64.tar.bz2
 INFO | Saving as: /mozilla/code/firefox/inbound/testing/mozharness/2015-09-01-03-02-26-mozilla-central-firefox-43.0a1.en-US.linux-x86_64.tar.bz2
^C1% |#############                                                                                                                  | ETA:  00:00:29   1.75 M/s
Download interrupted by the user
Flags: needinfo?(hskupin)
So I digged in the logs and have seen that mozilla-download has nothing to do with mozdownload. This tool is based on node.js and is maintained in https://github.com/mozilla-b2g/mozilla-download.
Thanks catlee & Henrik, I have pinged Gareth about this one.
Flags: needinfo?(gaye)
Blocks: 1142877
blocking-b2g: --- → 2.2r+
(Assignee)

Comment 13

3 years ago
(In reply to Gareth Aye [:gaye] (back from PTO) from comment #12)
> We can try to uplift this
> https://github.com/mozilla-b2g/gaia/commit/
> 30462283ce20a2481b9f6bd918218d65943548ec

This will download from TaskCluster, but I have been told we have no B2G Desktop v2.2/v2.2r build there, it is still on Buildbot.
You can still use https://archive.mozilla.org/ for downloading builds via a FTP connection.
Assignee: nobody → gaye

Updated

3 years ago
Flags: needinfo?(mtreese)

Comment 15

3 years ago
gaye is helping. Clear my NI.
Flags: needinfo?(jocheng)
Blocks: 1202484
Blocks: 1201530
Is there anything different about archive.mozilla.org? I'm pointing the mozilla-download ftp client at the new host and I'm not getting any responses. Also

gareth@albee:~/Documents/mozilla-get-url$ ftp -dv archive.mozilla.org
ftp: connect: Connection timed out
Flags: needinfo?(catlee)
This is not the FTP protocol anymore, HTTPS. Please see my comment 9.
(In reply to Henrik Skupin (:whimboo) from comment #14)
> You can still use https://archive.mozilla.org/ for downloading builds via a
> FTP connection.

^^ I am so confused
Sorry, I may have to stop commenting late in the night. FTP definitely doesn't work, right.
correct - we have discontinued support for the ftp protocol
Flags: needinfo?(catlee)
(Assignee)

Comment 23

3 years ago
(In reply to Henrik Skupin (:whimboo) from comment #20)
> Sorry, I may have to stop commenting late in the night. FTP definitely
> doesn't work, right.

(In reply to Chris AtLee [:catlee] from comment #21)
> correct - we have discontinued support for the ftp protocol

(In reply to Shing Lyu [:slyu] from comment #22)
> Some context, not sure if is the exact same thing:
> https://blog.mozilla.org/it/2015/07/27/product-delivery-migration-what-is-
> changing-when-its-changing-and-the-impacts/

Thank you! That is exactly what I said in comment 0 !

Comment 24

3 years ago
(In reply to Gareth Aye [:gaye] (back from PTO) from comment #16)
> Is there anything different about archive.mozilla.org? I'm pointing the
> mozilla-download ftp client at the new host and I'm not getting any
> responses. Also
> 
> gareth@albee:~/Documents/mozilla-get-url$ ftp -dv archive.mozilla.org
> ftp: connect: Connection timed out

Hi gaye,
Could you try modify mozilla-download to use HTTPS connect to archive.mozilla.org as mentioned by Shing?
Thanks!
Flags: needinfo?(gaye)
I think we'd be better off just using the python http client. Would someone get that set up for us?
Flags: needinfo?(hskupin)
Flags: needinfo?(gaye)
Flags: needinfo?(catlee)
I could give advises and mentor how to integrate mozdownload if wanted, but I don't have the time to do that work. Sorry. Feel free to also ask on IRC in the #automation channel.
Flags: needinfo?(hskupin)
I talked with Gareth and we can get to this one next week. Josh, if you are able to find someone else who can work on it sooner, we can provide guidance on it. 

Alex, is this specific to the 2.2R branch? What's the situation on 2.2?
Flags: needinfo?(lissyx+mozillians)

Updated

3 years ago
Flags: needinfo?(jocheng)

Comment 28

3 years ago
Hi Dylan,
After some check w/ team, I currently not able to find anyone available now, we would appreciate your help when gaye can check this next week. Thanks!
Flags: needinfo?(jocheng)
(Assignee)

Comment 29

3 years ago
(In reply to Dylan Oliver [:doliver] from comment #27)
> I talked with Gareth and we can get to this one next week. Josh, if you are
> able to find someone else who can work on it sooner, we can provide guidance
> on it. 
> 
> Alex, is this specific to the 2.2R branch? What's the situation on 2.2?

Anything except gaia-try on master is impacted and broken. Yes. All gaia < master.
Flags: needinfo?(lissyx+mozillians)
(Assignee)

Comment 30

3 years ago
Since there is no move, I have hacked something to at least get gaia-try working: https://github.com/lissyx/gaia/commit/0c0db03186ba442dc8569a14839e6c22d7ae2205
:gerard-majax hack is fine for me on v2.2 branch
(Assignee)

Comment 32

3 years ago
(In reply to Gareth Aye [:gaye] (back from PTO) from comment #31)
> :gerard-majax hack is fine for me on v2.2 branch

Right but thiw hack will break local development. We might be able to easily refine it, but it will still be a workaround.
If you still agree, I'll go down this road ... :)
Assignee: gaye → lissyx+mozillians
Flags: needinfo?(gaye)
(Assignee)

Comment 33

3 years ago
Ok, I may try something similar to what we do for the |b2g_sdk| target. This should work for local development and also for gaia-try. We might even easily uplift this.
(Assignee)

Comment 34

3 years ago
(In reply to Alexandre LISSY :gerard-majax from comment #33)
> Ok, I may try something similar to what we do for the |b2g_sdk| target. This
> should work for local development and also for gaia-try. We might even
> easily uplift this.

Well, problem solved: there are only linux64_gecko builds in https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ for the mozilla-b2g branches we are interested in ...
Created attachment 8661148 [details] [review]
[gaia] lissyx:bug1195872_gaia_v2.2r > mozilla-b2g:v2.2r
Created attachment 8661149 [details] [review]
[gaia] lissyx:bug1195872_gaia_v2.2 > mozilla-b2g:v2.2
(Assignee)

Comment 37

3 years ago
Comment on attachment 8661148 [details] [review]
[gaia] lissyx:bug1195872_gaia_v2.2r > mozilla-b2g:v2.2r

[Approval Request Comment]
[Bug caused by] (feature/regressing bug #): FTP shutdown
[User impact] if declined: no gaia-try
[Testing completed]: gaia-try works
[Risk to taking this patch] (and alternatives if risky): low
[String changes made]: none
Attachment #8661148 - Flags: review?(gaye)
Attachment #8661148 - Flags: approval-gaia-v2.2r?
(Assignee)

Comment 38

3 years ago
Comment on attachment 8661149 [details] [review]
[gaia] lissyx:bug1195872_gaia_v2.2 > mozilla-b2g:v2.2

[Approval Request Comment]
[Bug caused by] (feature/regressing bug #): FTP shutdown
[User impact] if declined: no gaia-try
[Testing completed]: gaia-try works
[Risk to taking this patch] (and alternatives if risky): low
[String changes made]: none
Attachment #8661149 - Flags: review?(gaye)
Attachment #8661149 - Flags: approval-gaia-v2.2?

Comment 39

3 years ago
Comment on attachment 8661149 [details] [review]
[gaia] lissyx:bug1195872_gaia_v2.2 > mozilla-b2g:v2.2

Hack approved. Will merge shortly.
Flags: needinfo?(gaye)
Attachment #8661149 - Flags: review?(gaye) → review+

Comment 40

3 years ago
Comment on attachment 8661148 [details] [review]
[gaia] lissyx:bug1195872_gaia_v2.2r > mozilla-b2g:v2.2r

Hack approved. Landing shortly.
Attachment #8661148 - Flags: review?(gaye) → review+

Comment 41

3 years ago
Comment on attachment 8661148 [details] [review]
[gaia] lissyx:bug1195872_gaia_v2.2r > mozilla-b2g:v2.2r

This can land with a=test-only.
Attachment #8661148 - Flags: approval-gaia-v2.2r?

Comment 42

3 years ago
Comment on attachment 8661149 [details] [review]
[gaia] lissyx:bug1195872_gaia_v2.2 > mozilla-b2g:v2.2

This can land with a=test-only.
Attachment #8661149 - Flags: approval-gaia-v2.2?
Created attachment 8661389 [details] [review]
[gaia] nullaus:bug1195872 > mozilla-b2g:v2.2r
Created attachment 8661393 [details] [review]
[gaia] nullaus:bug1195872-2 > mozilla-b2g:v2.2

Comment 45

3 years ago
Commit (v2.2r): https://github.com/mozilla-b2g/gaia/commit/5380c2226684d753059839092fbfef9b9da3e640
status-b2g-v2.2: --- → affected
status-b2g-v2.2r: --- → fixed
Created attachment 8661398 [details] [review]
[gaia] nullaus:v2.2 > mozilla-b2g:v2.2

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Cleaning off old ni's.
Flags: needinfo?(vkrishnamoorthy)
Flags: needinfo?(catlee)
You need to log in before you can comment on or make changes to this bug.