1180949 - `mach artifact install` fails without wget installed

Status: RESOLVED WORKSFORME

(Firefox for Android Graveyard :: General, defect)

Description

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Attachment: Failure output

My OS X install (10.10) did not have wget pre-installed.

I installed via brew and `mach artifact install` was able to complete.

I believe `curl` comes pre-installed - perhaps that should be used if wget is not available?

Comment 1

[Nick Alexander :nalexander [he/him]]

We're using wget specifically so it will handle --timestamp for us.  (Curl doesn't make this as easy.)

I'd prefer that we actually use Python to do the fetch and file caching.  Using something like

https://github.com/mozilla/mozregression/blob/master/mozregression/download_manager.py
https://github.com/mozilla/mozregression/blob/master/mozregression/limitedfilecache.py

would be good, I think.  This is a great contributor ticket, although maybe not a great first ticket.

Comment 2

[Nick Alexander :nalexander [he/him]]

You can find the relevant code by using:

nalexander@chocho ~/M/gecko> ./mach mach-debug-commands artifact
artifact
========

File: /Users/nalexander/Mozilla/gecko/mobile/android/mach_commands.py
Class: PackageFrontend
Method: artifact

In this case, that's a wrapper around the real code, which is at https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/artifacts.py.

The relevant code is at https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/artifacts.py#261.  The caching is a little tricky, so you might want to disable it entirely at first (by removing the @cachedmethod wrapper.)

Comment 3

[rshenthar]

Thanks Nick! I searched mozilla-central but could not find any use of mozregression modules in it. Could you point me to a file where download manager and LimitedFileCache are used?

Are the highlighted code areas the ones that we need to change in the below link?
https://pastebin.mozilla.org/8839224

Also I am not familiar with caching - for the wget or for any download, could you please give a brief explanation of how the process works.(with respect to https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/artifacts.py#261 would be even simpler.)

Comment 4

[Nick Alexander :nalexander [he/him]]

(In reply to rshenthar from comment #3)
> Thanks Nick! I searched mozilla-central but could not find any use of
> mozregression modules in it. Could you point me to a file where download
> manager and LimitedFileCache are used?

There are no uses.  mozregression is in pypi (https://pypi.python.org/pypi/mozregression/); see the |mach artifact| command to see examples of using pip to install pypi dependencies.  You should be able to add the dependency and then use the code in the command easily.  This is how we use pylru, for example.
 
> Are the highlighted code areas the ones that we need to change in the below
> link?
> https://pastebin.mozilla.org/8839224

Yes, mostly.  You'll need to rewrite this method entirely to not use wget and to use the mozregression modules.  (Assuming that it's possible; I haven't investigated the mozregression API thoroughly.  You might need to work on mozregression to make it work for your situation as well.)

> Also I am not familiar with caching - for the wget or for any download,
> could you please give a brief explanation of how the process works.(with
> respect to
> https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/
> artifacts.py#261 would be even simpler.)

The caching is built into the mozregression download_manager.  You shouldn't need to know too much here.  A demo that didn't cache (but used mozregression) would be a big help.

Comment 5

[rshenthar]

Unassigning as I am occupied with high priority bugs for now. Free to take and work on.

Comment 6

[Alex Johnson (:alex_johnson)]

Can I work on this one?  Just a fore warning, I haven't worked with Python a whole lot.  But, I defintely want to learn.

Comment 7

[Nick Alexander :nalexander [he/him]]

(In reply to Alex Johnson (:alex_johnson) from comment #6)
> Can I work on this one?  Just a fore warning, I haven't worked with Python a
> whole lot.  But, I defintely want to learn.

I think this has been fixed organically.  Now we're using mozdownload, so this isn't an issue:

https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/artifacts.py#265

Alex, you might be interested in https://bugzilla.mozilla.org/show_bug.cgi?id=1221200.  That ticket has significant work in progress but has stalled.  I'd want to check in with Harshit before jumping on that, but take a look at it.