Closed Bug 1195872 Opened 5 years ago Closed 5 years ago

Fix mozilla-download to use https instead of ftp

Categories

(Firefox OS Graveyard :: Gaia, defect)

ARM
Gonk (Firefox OS)
defect
Not set
normal

Tracking

(blocking-b2g:2.2r+, b2g-v2.2 fixed, b2g-v2.2r fixed)

RESOLVED FIXED
blocking-b2g 2.2r+
Tracking Status
b2g-v2.2 --- fixed
b2g-v2.2r --- fixed

People

(Reporter: gerard-majax, Assigned: gerard-majax)

References

Details

Attachments

(5 files)

Running gaia-try on v2.2r is totally broken: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=dce50a884e3b39f1fb676cf72f4e6c62adeb6368

This is because the mozilla-download (and mozilla-get-url) there relies on the FTP protocol which was shutdown.

The master branch of mozilla-download is of no help since it downloads from TaskCluster while the B2G builds for b2g37_v2.2r branch are being produced by Buildbot. So no build available from this master either.

This technically means any uplift on Gaia for v2.2r is not able to get any test being run.
Blocks: 1181505
Josh, this is actively blocking 2.2r patches from landing. Can you please help move it along?
Flags: needinfo?(jocheng)
Hi LISSY,
I am not sure who is the owner of mozilla-download that can help here. Do you know?
Thanks!
Flags: needinfo?(lissyx+mozillians)
(In reply to Josh Cheng [:josh] from comment #2)
> Hi LISSY,
> I am not sure who is the owner of mozilla-download that can help here. Do
> you know?
> Thanks!

If I'd knew I would have already pinged them.
Flags: needinfo?(lissyx+mozillians)
This is blocking a sec-moderate security fix backport for 2.2 of bug 1181571, and I have another fix backport incoming. Should these be landed blindly?
Blocking KillSwitch ...
Flags: needinfo?(vkrishnamoorthy)
Flags: needinfo?(mtreese)
Chris, is this something you can help with?
Flags: needinfo?(catlee)
I'm not familiar with mozilla-download, but I've been told that perhaps Henrik can help.
Flags: needinfo?(catlee) → needinfo?(hskupin)
I don't know what mozilla-download is, but I assume you mean mozdownload? If it does not work for you I assume you are using a version <1.17. I would encourage you to upgrade to v1.17 which is not using ftp.mozilla.org anymore but archive.mozilla.org, and it works perfectly:

$ mozdownload -t daily
 INFO | Retrieving the build status file from https://archive.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/
 INFO | Retrieving list of builds from https://archive.mozilla.org/pub/mozilla.org/firefox/nightly/2015/09
 INFO | Found 1 build: 2015-09-01-03-02-26-mozilla-central
 INFO | Selected build: 2015-09-01-03-02-26-mozilla-central
 INFO | Downloading from: https://archive.mozilla.org/pub/mozilla.org/firefox/nightly/2015/09/2015-09-01-03-02-26-mozilla-central/firefox-43.0a1.en-US.linux-x86_64.tar.bz2
 INFO | Saving as: /mozilla/code/firefox/inbound/testing/mozharness/2015-09-01-03-02-26-mozilla-central-firefox-43.0a1.en-US.linux-x86_64.tar.bz2
^C1% |#############                                                                                                                  | ETA:  00:00:29   1.75 M/s
Download interrupted by the user
Flags: needinfo?(hskupin)
So I digged in the logs and have seen that mozilla-download has nothing to do with mozdownload. This tool is based on node.js and is maintained in https://github.com/mozilla-b2g/mozilla-download.
Thanks catlee & Henrik, I have pinged Gareth about this one.
Flags: needinfo?(gaye)
blocking-b2g: --- → 2.2r+
(In reply to Gareth Aye [:gaye] (back from PTO) from comment #12)
> We can try to uplift this
> https://github.com/mozilla-b2g/gaia/commit/
> 30462283ce20a2481b9f6bd918218d65943548ec

This will download from TaskCluster, but I have been told we have no B2G Desktop v2.2/v2.2r build there, it is still on Buildbot.
You can still use https://archive.mozilla.org/ for downloading builds via a FTP connection.
Assignee: nobody → gaye
Flags: needinfo?(mtreese)
gaye is helping. Clear my NI.
Flags: needinfo?(jocheng)
Is there anything different about archive.mozilla.org? I'm pointing the mozilla-download ftp client at the new host and I'm not getting any responses. Also

gareth@albee:~/Documents/mozilla-get-url$ ftp -dv archive.mozilla.org
ftp: connect: Connection timed out
Flags: needinfo?(catlee)
This is not the FTP protocol anymore, HTTPS. Please see my comment 9.
(In reply to Henrik Skupin (:whimboo) from comment #14)
> You can still use https://archive.mozilla.org/ for downloading builds via a
> FTP connection.

^^ I am so confused
Sorry, I may have to stop commenting late in the night. FTP definitely doesn't work, right.
correct - we have discontinued support for the ftp protocol
Flags: needinfo?(catlee)
(In reply to Henrik Skupin (:whimboo) from comment #20)
> Sorry, I may have to stop commenting late in the night. FTP definitely
> doesn't work, right.

(In reply to Chris AtLee [:catlee] from comment #21)
> correct - we have discontinued support for the ftp protocol

(In reply to Shing Lyu [:slyu] from comment #22)
> Some context, not sure if is the exact same thing:
> https://blog.mozilla.org/it/2015/07/27/product-delivery-migration-what-is-
> changing-when-its-changing-and-the-impacts/

Thank you! That is exactly what I said in comment 0 !
(In reply to Gareth Aye [:gaye] (back from PTO) from comment #16)
> Is there anything different about archive.mozilla.org? I'm pointing the
> mozilla-download ftp client at the new host and I'm not getting any
> responses. Also
> 
> gareth@albee:~/Documents/mozilla-get-url$ ftp -dv archive.mozilla.org
> ftp: connect: Connection timed out

Hi gaye,
Could you try modify mozilla-download to use HTTPS connect to archive.mozilla.org as mentioned by Shing?
Thanks!
Flags: needinfo?(gaye)
I think we'd be better off just using the python http client. Would someone get that set up for us?
Flags: needinfo?(hskupin)
Flags: needinfo?(gaye)
Flags: needinfo?(catlee)
I could give advises and mentor how to integrate mozdownload if wanted, but I don't have the time to do that work. Sorry. Feel free to also ask on IRC in the #automation channel.
Flags: needinfo?(hskupin)
I talked with Gareth and we can get to this one next week. Josh, if you are able to find someone else who can work on it sooner, we can provide guidance on it. 

Alex, is this specific to the 2.2R branch? What's the situation on 2.2?
Flags: needinfo?(lissyx+mozillians)
Flags: needinfo?(jocheng)
Hi Dylan,
After some check w/ team, I currently not able to find anyone available now, we would appreciate your help when gaye can check this next week. Thanks!
Flags: needinfo?(jocheng)
(In reply to Dylan Oliver [:doliver] from comment #27)
> I talked with Gareth and we can get to this one next week. Josh, if you are
> able to find someone else who can work on it sooner, we can provide guidance
> on it. 
> 
> Alex, is this specific to the 2.2R branch? What's the situation on 2.2?

Anything except gaia-try on master is impacted and broken. Yes. All gaia < master.
Flags: needinfo?(lissyx+mozillians)
Since there is no move, I have hacked something to at least get gaia-try working: https://github.com/lissyx/gaia/commit/0c0db03186ba442dc8569a14839e6c22d7ae2205
:gerard-majax hack is fine for me on v2.2 branch
(In reply to Gareth Aye [:gaye] (back from PTO) from comment #31)
> :gerard-majax hack is fine for me on v2.2 branch

Right but thiw hack will break local development. We might be able to easily refine it, but it will still be a workaround.
If you still agree, I'll go down this road ... :)
Assignee: gaye → lissyx+mozillians
Flags: needinfo?(gaye)
Ok, I may try something similar to what we do for the |b2g_sdk| target. This should work for local development and also for gaia-try. We might even easily uplift this.
(In reply to Alexandre LISSY :gerard-majax from comment #33)
> Ok, I may try something similar to what we do for the |b2g_sdk| target. This
> should work for local development and also for gaia-try. We might even
> easily uplift this.

Well, problem solved: there are only linux64_gecko builds in https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ for the mozilla-b2g branches we are interested in ...
Comment on attachment 8661148 [details] [review]
[gaia] lissyx:bug1195872_gaia_v2.2r > mozilla-b2g:v2.2r

[Approval Request Comment]
[Bug caused by] (feature/regressing bug #): FTP shutdown
[User impact] if declined: no gaia-try
[Testing completed]: gaia-try works
[Risk to taking this patch] (and alternatives if risky): low
[String changes made]: none
Attachment #8661148 - Flags: review?(gaye)
Attachment #8661148 - Flags: approval-gaia-v2.2r?
Comment on attachment 8661149 [details] [review]
[gaia] lissyx:bug1195872_gaia_v2.2 > mozilla-b2g:v2.2

[Approval Request Comment]
[Bug caused by] (feature/regressing bug #): FTP shutdown
[User impact] if declined: no gaia-try
[Testing completed]: gaia-try works
[Risk to taking this patch] (and alternatives if risky): low
[String changes made]: none
Attachment #8661149 - Flags: review?(gaye)
Attachment #8661149 - Flags: approval-gaia-v2.2?
Comment on attachment 8661149 [details] [review]
[gaia] lissyx:bug1195872_gaia_v2.2 > mozilla-b2g:v2.2

Hack approved. Will merge shortly.
Flags: needinfo?(gaye)
Attachment #8661149 - Flags: review?(gaye) → review+
Comment on attachment 8661148 [details] [review]
[gaia] lissyx:bug1195872_gaia_v2.2r > mozilla-b2g:v2.2r

Hack approved. Landing shortly.
Attachment #8661148 - Flags: review?(gaye) → review+
Comment on attachment 8661148 [details] [review]
[gaia] lissyx:bug1195872_gaia_v2.2r > mozilla-b2g:v2.2r

This can land with a=test-only.
Attachment #8661148 - Flags: approval-gaia-v2.2r?
Comment on attachment 8661149 [details] [review]
[gaia] lissyx:bug1195872_gaia_v2.2 > mozilla-b2g:v2.2

This can land with a=test-only.
Attachment #8661149 - Flags: approval-gaia-v2.2?
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Cleaning off old ni's.
Flags: needinfo?(vkrishnamoorthy)
Flags: needinfo?(catlee)
You need to log in before you can comment on or make changes to this bug.