1197224 - Write mozharness script to run firefox-media-tests in buildbot

Status: RESOLVED FIXED

(Testing Graveyard :: external-media-tests, defect)

Description

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Adapt the existing mozharness script used for running firefox-media-test jobs in pf-jenkins [1], using a config specific to buildbot.

We're keeping firefox-media-tests in Github for now (with firefox-ui-tests as a submodule) so a 'checkout' step is needed in the mozharness script, in addition to the other changes required to work with buildbot.


[1] https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/run_media_tests.py

Comment 1

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Here's a draft of the script (firefox_media_tests.py) along with a stab at a config file for buildbot (firefox_media_tests_buildbot.py): https://gist.github.com/mjzffr/f2c7612d2efd3c0c1304

Might be easier to look at the diff: https://gist.github.com/mjzffr/f2c7612d2efd3c0c1304/revisions

I added a checkout step to get the tests, as well as calls to self.buildbot_status. To run the default YouTube playback tests, you'd run something like |python path/to/firefox_media_tests.py --test-timeout 400 --tests ./build/firefox_media_tests/playback/youtube/manifest.ini --no-treeherding -c path/to/firefox_media_tests_buildbot.py|, assuming the Firefox installer and crashreporter-symbols are magically provided by buildbot. 

Some questions:
* How does buildbot affect what I have to specify for the binary, etc.? (e.g. here https://gist.github.com/mjzffr/f2c7612d2efd3c0c1304/6644e4913b3d5c9b1e7adb7667bc5f0a29050062#file-firefox_media_tests-py-L530,L532)
* Are there any expectations about log output? (e.g. https://gist.github.com/mjzffr/f2c7612d2efd3c0c1304/6644e4913b3d5c9b1e7adb7667bc5f0a29050062#file-firefox_media_tests-py-L546)
* How does the config for buildbot vary per platform? How does that affect the exes to specify, etc? All aspects of this config are generally a mystery to me.

Comment 2

[Jonathan Griffin (:jgriffin)]

Hey Maja,

The config file looks about right for Windows.  You'll need a separate config file for osx/linux, with different values for exes and virtualenv fields; you can cargo-cult these values from e.g., https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/marionette/prod_config.py

Other comments:

The buildbot version of the script will have to do some things that the Jenkins version won't, and vice versa; I think you might be better off defining a common script and then buildbot and Jenkins subclasses; more on this below.

You need to add 'read-buildbot-config' to all_actions; default_actions must be a subset of that.

self.binary_path will work 'magically' if you add the equivalent of https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/marionette.py#180

--virtualenv creation--
For Jenkins, I think you're using a pre-created virtualenv.  For buildbot, you'll need a create-virtualenv method that installs all the necessary dependencies from the filesystem, as much as possible.  This means installing mozbase components and Marionette from tests.zip, and then installing firefox-ui-tests and then firefox-media-tests.  You can do the mozbase/marionette install easily using https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/marionette.py#228-235

At the same time, we want to avoid installing the unused modules in TreeherdingMixin, which is another reason I suggested splitting this script into buildbot and Jenkins subclasses.

--downloading and installing the build--
All of the logic you have in download_and_extract is unnecessary in buildbot; the right things will happen automatically there.  That logic should be saved for a Jenkins-specific subclass.

--logs--
Marionette's default logging works fine with Treeherder.  In particular, there's no need to pass --log-tbpl.  For all of the logs that are generated, getting them uploaded so that they're visible in Treeherder requires using blobber.  Using blobber is pretty simple:

1 - Add BlobUploadMixin to your class (https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/marionette.py?offset=200#29)
2 - define a 'abs_blob_upload_dir' directory, ala https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/marionette.py?offset=700#219
3 - write logs you want to upload in that directory
4 - logs and automatically uploaded at the end of the script and appear in the "job details" panel of Treeherder

Testing this kind of buildbot script isn't too hard.  Generally, you create a copy of your config file and remove the 'read-buildbot-config' step from it then invoke your script like so:

python my_script.py --config my_config.py --installer-url http://installerurl --test-packages-url http://testjsonurl

where testjsonurl is something like http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64/1440539618/test_packages.json
and installurl is something like http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64/1440539618/firefox-43.0a1.en-US.linux-x86_64.tar.bz2

Comment 3

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Thanks for the detailed feedback! Time for Round 2. Refactored into FirefoxMediaTestsBuildbot + FirefoxMediaTestsBase: https://gist.github.com/mjzffr/f2c7612d2efd3c0c1304#file-firefox_media_tests_buildbot-py

It works locally. Let me know how I can test it in buildbot itself.  

Note: Unlike the example configs you've pointed me to, I think I need to add "gittool.py" to "exes" for the "checkout" step to work. Is that a problem?

Comment 4

[Jonathan Griffin (:jgriffin)]

Nice!  Yes, feel free to add 'gittool.py' to 'exes'.

By convention, the config file you currently have (which is for Windows) should be named something like buildbot_windows_config.py, and the posix version can be buildbot_posix_config.py, if no differences are needed between osx and linux.

To test this in buildbot, I need to write a buildbot patch, which I'll do today in bug 1185014.  This needs to be put into production by releng, which will likely happen next Tuesday.  You need to land the mozharness script and config file in gecko, under testing/mozharness, and then we'll be able to see results start showing up at https://treeherder.mozilla.org/#/jobs?repo=cedar

These two steps can happen in either order, so feel free to make a gecko patch for the mozharness bits and attach here for official review any time.  Since this is your first gecko patch (AFAIK), you may find this useful, or ping me or anyone else on the team: https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/How_to_Submit_a_Patch

Comment 5

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Attachment: MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

Bug 1197224 - Add mozharness script to run firefox-media-tests in buildbot; r?jgriffin

Comment 6

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

https://reviewboard.mozilla.org/r/18729/#review16765

Looks good, let's give it a shot!

::: testing/mozharness/mozharness/mozilla/testing/firefox_media_tests.py:135
(Diff revision 1)
> +    def __init__(self, config_options=[], all_actions=[], **kwargs):

Passing a list as a default arg can have unintended consequences; better to use an empty tuple or None.

Comment 7

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

Bug 1197224 - Add mozharness script to run firefox-media-tests in buildbot; r?jgriffin

Comment 8

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

https://reviewboard.mozilla.org/r/18729/#review16917

Comment 9

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/d2457f064ba3

Comment 10

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/d2457f064ba3

Comment 11

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

Bug 1197224 - Part 2 Fix path for gittool.py and git on Windows; r?jgriffin

Comment 12

[Jonathan Griffin (:jgriffin)]

landed on cedar for testing:

remote: View your change here:
remote:   https://hg.mozilla.org/projects/cedar/rev/b821cdc017a8
remote: 
remote: Follow the progress of your build on Treeherder:
remote:   https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=b821cdc017a8

Comment 13

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

Bug 1197224 - Part 2 Fix path for gittool.py and git on Windows; r?jgriffin

Comment 14

[Jonathan Griffin (:jgriffin)]

remote: View your change here:
remote:   https://hg.mozilla.org/projects/cedar/rev/35b623bc0b77
remote: 
remote: Follow the progress of your build on Treeherder:
remote:   https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=35b623bc0b77

Comment 15

[Jonathan Griffin (:jgriffin)]

Apparently, we don't have git on the Windows slaves.  Maja filed this as bug 1206877.  If this takes a long time to fix (which is my guess), the alternative is to package firefox-media-tests and dependencies, and upload them as needed to internal pypi.  This would remove the need to clone anything with git.

Comment 16

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

We do have git on the Windows slaves -- the checkout step works -- it's just not in the default PATH, and a particular call to 'pip freeze' the in VirtualenvMixin relies on the default PATH. 

OS X 10.6 slaves actually don't have git installed. We could just not test that platform.

Comment 17

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

Bug 1197224 - Part 2 Fix path for gittool.py and git on Windows; r?jgriffin

Comment 18

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Attachment: MozReview Request: Bug 1197224 - Part 3 Don't install packages in editable mode; r?jgriffin

Bug 1197224 - Part 3 Don't install packages in editable mode; r?jgriffin

Comment 19

[Jonathan Griffin (:jgriffin)]

remote: View your change here:
remote:   https://hg.mozilla.org/projects/cedar/rev/78f26721ef4e
remote: 
remote: Follow the progress of your build on Treeherder:
remote:   https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=78f26721ef4e

Comment 20

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

The 'checkout' and 'create-virtualenv' steps succeeded on Windows. Next problem: the call to firefox-media-tests fails, but whimboo pointed out that it looks like Bug 1198448. I'll try the same fix as my next step.

Comment 21

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8665079 [details]
MozReview Request: Bug 1197224 - Part 3 Don't install packages in editable mode; r?jgriffin

Bug 1197224 - Part 3 Don't install packages in editable mode; r?jgriffin

Comment 22

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Attachment: MozReview Request: Bug 1197224 - Part 4 Call tests via Python entry-point; r?jgriffin

Bug 1197224 - Part 4 Call tests via Python entry-point; r?jgriffin

Similarly to Bug 1198448, calling the firefox-media-tests binary
on Windows fails because it triggers a UAC prompt. Instead, we
call the same entry-point script via Python.

Comment 23

[Jonathan Griffin (:jgriffin)]

remote: View your change here:
remote:   https://hg.mozilla.org/projects/cedar/rev/769a602cfe63
remote: 
remote: Follow the progress of your build on Treeherder:
remote:   https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=769a602cfe63

Comment 24

[James Graham [:jgraham]]

https://hg.mozilla.org/projects/cedar/rev/210f3ed93368c0ba781b55faf3341520059ccc4f
Bug 1197224 - Part 3 Don't install packages in editable mode; r?jgriffin

Comment 25

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

All jobs are now green, except on OS X 10.6, which does not have git installed. I propose that we do not include this platform given the small percentage of active users. (I'm told it's 1.5% of *Mac* Firefox users, which is < 1% of all Firefox users.) Does that sound good, jgriffin?

https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=210f3ed93368&filter-job_type_symbol=Mt

Comment 26

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

Correct path for gittool.py in Windows config. Don't install packages
in editable mode. Call media tests via Python entry point: similarly
to Bug 1198448, calling the firefox-media-tests binary
on Windows fails because it triggers a UAC prompt. Instead, we
call the same entry-point script via Python.

Comment 27

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

https://reviewboard.mozilla.org/r/18729/#review18487

lgtm

Comment 28

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

Correct path for gittool.py in Windows config. Don't install packages
in editable mode. Call media tests via Python entry point: similarly
to Bug 1198448, calling the firefox-media-tests binary
on Windows fails because it triggers a UAC prompt. Instead, we
call the same entry-point script via Python.

Comment 29

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

The naming-scheme of branches in the firefox-ui-tests repo just changed in bug 1209011, so I need to replace 'master' with 'mozilla-central' for the checkout step to work.

Comment 30

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

https://reviewboard.mozilla.org/r/18729/#review18569

Comment 31

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/992ee69d7c2e

Comment 32

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/992ee69d7c2e