1252608 - |mach external-media-tests| ignores --binary option

Status: RESOLVED FIXED

(Testing Graveyard :: external-media-tests, defect)

Description

[Syd Polk :sydpolk]

+++ This bug was initially created as a clone of Bug #1252586 +++

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:44.0) Gecko/20100101 Firefox/44.0
Build ID: 20160210153822

Steps to reproduce:

Ran ./mach external-media-tests --binary /Applications/Firefox.app/Contents/MacOS/firefox


Actual results:

Mach fails with an error,
Exception: Binary expected at path_to_obj-x86_64-apple-darwin15.3.0/dist/Nightly.app/Contents/MacOS/firefox does not exist.


Expected results:

The tests should run with the binary specified with the --binary option.

Comment 1

[Syd Polk :sydpolk]

Attachment: MozReview Request: Bug 1252608 - --binary does not work with |mach external-media-tests| - r?whimboo, r?gps

Review commit: https://reviewboard.mozilla.org/r/37465/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37465/

Comment 2

[Gregory Szorc [:gps]]

Comment on attachment 8725404 [details]
MozReview Request: Bug 1252608 - --binary does not work with |mach external-media-tests| - r?whimboo, r?gps

https://reviewboard.mozilla.org/r/37465/#review34045

::: dom/media/test/external/mach_commands.py:69
(Diff revision 1)
> +        if kwargs['binary'] is None:

Same nit as bug 1252586.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8725404 [details]
MozReview Request: Bug 1252608 - --binary does not work with |mach external-media-tests| - r?whimboo, r?gps

https://reviewboard.mozilla.org/r/37465/#review34097

With the nits fixed before landing. r=me.

Comment 4

[Syd Polk :sydpolk]

There is a well-established, well-documented procedure using virtualenv and python setup.py that will allow users to specify arbitrary binaries and test directories. There are also mozharness scripts that will allow users to give urls to installers and test packages. Having the ./mach command do the same thing, when it requires a local build anyway, is not worth the hassle.

The only advantage is that if you have multiple object directories from your build, you could use --binary to point to the other one. Unlikely the media team will need this.

Bryce, Chris, Ralph, what do you think?

Comment 5

[Chris Pearce [:cpearce (Not reading bugmail)]]

Can't multiple objdirs be supported by specifying/changing mk_add_options MOZ_OBJDIR=$foo in mozconfig?

The only advantage I can see is that it makes it easier to run the Netflix tests, as you can point the test harness at a Mozilla-built Nightly with a voucher.bin and vouched-for plugin-container.exe and the Netflix tests will work. I'd settle for the harness refusing to run with an appropriate error message if there is not a voucher.bin in the bin dir. Bonus points if we also check whether the plugin-container.exe in the objdir has a creation timestamp that is similar (say within 1 hour) to the voucher.bin's timestamp, as that would indicate whether the plugin-container.exe matches the voucher.

Comment 6

[Syd Polk :sydpolk]

If you run |mach configure| first, this should work with --binary pointing to other binaries, and is a valid use case.

Comment 7

[Syd Polk :sydpolk]

Comment on attachment 8725404 [details]
MozReview Request: Bug 1252608 - --binary does not work with |mach external-media-tests| - r?whimboo, r?gps

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37465/diff/1-2/

Comment 8

[Syd Polk :sydpolk]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6f2ba39efd9bae76a5a33abd971c132d01aeb90b
Bug 1252608 - --binary does not work with |mach external-media-tests| - r=whimboo, r=gps

Comment 9

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/6f2ba39efd9b