1036064 - --installer-url and --test-url should accept local files

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Gregory Szorc [:gps]]

I'm running mozharness locally and want to make it easier to point it at installers/tests generated from local builds. Patch will be uploaded momentarily.

Comment 1

[Aki Sasaki (not active)]

--installer-path ?
We don't have an equivalent for --test-zip-path but I don't think it would be difficult to add.

Comment 2

[Gregory Szorc [:gps]]

Attachment: Accept local paths to --installer-url and --test-url

If the argument given to --installer-url and --test-url looks like a
path (we only support UNIX paths), we prepend file:// to the value. The
existing URL handling code transparently "downloads" these URLs by
copying the file.

As part of testing this, I realized my local build wasn't producing a
jsshell archive and failed "downloads" would retry. I added some logic
to the download error handling logic to detect local filesystem errors
so this error is treated as fatal.

The .errno comparison should probably catch a few more failures, such as
permissions failures. But I'm not sure which ones are exclusive to
file:// vs other protocols. I suppose we could have certain protocol
handlers not retry. But this was the simplest patch.

Comment 3

[Gregory Szorc [:gps]]

(In reply to Aki Sasaki [:aki] from comment #1)
> --installer-path ?
> We don't have an equivalent for --test-zip-path but I don't think it would
> be difficult to add.

Well, I tried --installer-path with the code path I was using and it didn't like it! I think there are differences between --installer-path and --installer-url. I didn't want to track those down.

I think computers should take the cognitive load of "url" vs "path" away from the user. IMO we should trend towards a single argument (--installer/--tests-zip) that accepts either both URLs and paths. This patch gets us closer to that.

Comment 4

[Aki Sasaki (not active)]

Comment on attachment 8452626 [details] [diff] [review]
Accept local paths to --installer-url and --test-url

Hm, I thought file:// didn't work, but now it does.  Not sure if this was faulty testing back then or a change in urllib2.

Comment 5

[Gregory Szorc [:gps]]

https://hg.mozilla.org/build/mozharness/rev/34ac30d649b6

Comment 6

[Aki Sasaki (not active)]

pyflakes sez:
mozharness/base/script.py:208: undefined name 'errno'

Comment 7

[Pete Moore [:pmoore][:pete]]

Merged to production branch of mozharness.
Is comment 6 a problem? Do we need to pull it out?

Comment 8

[Gregory Szorc [:gps]]

https://hg.mozilla.org/build/mozharness/rev/fd063cc86fb6

Comment 9

[Gregory Szorc [:gps]]

It's not a problem for automation use - only local use. I pushed a trivial followup (missing import of a stdlib module).

I keep forgetting I don't have my Mercurial extension for running pyflakes on commit globally installed. I need to fix that...

Comment 10

[Chris Cooper [:coop] (he/him)]

Merged to production, and deployed.

Comment 11

[Gregory Szorc [:gps]]

If this is deployed to production, why is the bug still open?

Comment 12

[Aki Sasaki (not active)]

We leave it to the owner to resolve, since it's not always clear whether a bug is closable after the initial set of patches is landed.

Comment 13

[Gregory Szorc [:gps]]

Ahh, didn't know that.

In that case...

Comment 14

[Aki Sasaki (not active)]

mozharness/base/script.py:215: [E, ScriptMixin._download_file] Module 'errno' has no 'NOENT' member

https://docs.python.org/2/library/errno.html

Did you mean errno.ENOENT ?

Comment 15

[Gregory Szorc [:gps]]

https://hg.mozilla.org/build/mozharness/rev/519a7b89282d

Comment 16

[Aki Sasaki (not active)]

In production.