Write mozharness script to run firefox-media-tests in buildbot

RESOLVED FIXED in Firefox 43

Status

defect
RESOLVED FIXED
4 years ago
a year ago

People

(Reporter: maja_zf, Assigned: maja_zf)

Tracking

unspecified
mozilla44
Dependency tree / graph

Firefox Tracking Flags

(firefox43 fixed, firefox44 fixed)

Details

Attachments

(1 attachment, 2 obsolete attachments)

Adapt the existing mozharness script used for running firefox-media-test jobs in pf-jenkins [1], using a config specific to buildbot.

We're keeping firefox-media-tests in Github for now (with firefox-ui-tests as a submodule) so a 'checkout' step is needed in the mozharness script, in addition to the other changes required to work with buildbot.


[1] https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/run_media_tests.py
Assignee

Comment 1

4 years ago
Here's a draft of the script (firefox_media_tests.py) along with a stab at a config file for buildbot (firefox_media_tests_buildbot.py): https://gist.github.com/mjzffr/f2c7612d2efd3c0c1304

Might be easier to look at the diff: https://gist.github.com/mjzffr/f2c7612d2efd3c0c1304/revisions

I added a checkout step to get the tests, as well as calls to self.buildbot_status. To run the default YouTube playback tests, you'd run something like |python path/to/firefox_media_tests.py --test-timeout 400 --tests ./build/firefox_media_tests/playback/youtube/manifest.ini --no-treeherding -c path/to/firefox_media_tests_buildbot.py|, assuming the Firefox installer and crashreporter-symbols are magically provided by buildbot. 

Some questions:
* How does buildbot affect what I have to specify for the binary, etc.? (e.g. here https://gist.github.com/mjzffr/f2c7612d2efd3c0c1304/6644e4913b3d5c9b1e7adb7667bc5f0a29050062#file-firefox_media_tests-py-L530,L532)
* Are there any expectations about log output? (e.g. https://gist.github.com/mjzffr/f2c7612d2efd3c0c1304/6644e4913b3d5c9b1e7adb7667bc5f0a29050062#file-firefox_media_tests-py-L546)
* How does the config for buildbot vary per platform? How does that affect the exes to specify, etc? All aspects of this config are generally a mystery to me.
Flags: needinfo?(jgriffin)
Hey Maja,

The config file looks about right for Windows.  You'll need a separate config file for osx/linux, with different values for exes and virtualenv fields; you can cargo-cult these values from e.g., https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/marionette/prod_config.py

Other comments:

The buildbot version of the script will have to do some things that the Jenkins version won't, and vice versa; I think you might be better off defining a common script and then buildbot and Jenkins subclasses; more on this below.

You need to add 'read-buildbot-config' to all_actions; default_actions must be a subset of that.

self.binary_path will work 'magically' if you add the equivalent of https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/marionette.py#180

--virtualenv creation--
For Jenkins, I think you're using a pre-created virtualenv.  For buildbot, you'll need a create-virtualenv method that installs all the necessary dependencies from the filesystem, as much as possible.  This means installing mozbase components and Marionette from tests.zip, and then installing firefox-ui-tests and then firefox-media-tests.  You can do the mozbase/marionette install easily using https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/marionette.py#228-235

At the same time, we want to avoid installing the unused modules in TreeherdingMixin, which is another reason I suggested splitting this script into buildbot and Jenkins subclasses.

--downloading and installing the build--
All of the logic you have in download_and_extract is unnecessary in buildbot; the right things will happen automatically there.  That logic should be saved for a Jenkins-specific subclass.

--logs--
Marionette's default logging works fine with Treeherder.  In particular, there's no need to pass --log-tbpl.  For all of the logs that are generated, getting them uploaded so that they're visible in Treeherder requires using blobber.  Using blobber is pretty simple:

1 - Add BlobUploadMixin to your class (https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/marionette.py?offset=200#29)
2 - define a 'abs_blob_upload_dir' directory, ala https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/marionette.py?offset=700#219
3 - write logs you want to upload in that directory
4 - logs and automatically uploaded at the end of the script and appear in the "job details" panel of Treeherder

Testing this kind of buildbot script isn't too hard.  Generally, you create a copy of your config file and remove the 'read-buildbot-config' step from it then invoke your script like so:

python my_script.py --config my_config.py --installer-url http://installerurl --test-packages-url http://testjsonurl

where testjsonurl is something like http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64/1440539618/test_packages.json
and installurl is something like http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64/1440539618/firefox-43.0a1.en-US.linux-x86_64.tar.bz2
Flags: needinfo?(jgriffin)
Assignee

Comment 3

4 years ago
Thanks for the detailed feedback! Time for Round 2. Refactored into FirefoxMediaTestsBuildbot + FirefoxMediaTestsBase: https://gist.github.com/mjzffr/f2c7612d2efd3c0c1304#file-firefox_media_tests_buildbot-py

It works locally. Let me know how I can test it in buildbot itself.  

Note: Unlike the example configs you've pointed me to, I think I need to add "gittool.py" to "exes" for the "checkout" step to work. Is that a problem?
Flags: needinfo?(jgriffin)
Nice!  Yes, feel free to add 'gittool.py' to 'exes'.

By convention, the config file you currently have (which is for Windows) should be named something like buildbot_windows_config.py, and the posix version can be buildbot_posix_config.py, if no differences are needed between osx and linux.

To test this in buildbot, I need to write a buildbot patch, which I'll do today in bug 1185014.  This needs to be put into production by releng, which will likely happen next Tuesday.  You need to land the mozharness script and config file in gecko, under testing/mozharness, and then we'll be able to see results start showing up at https://treeherder.mozilla.org/#/jobs?repo=cedar

These two steps can happen in either order, so feel free to make a gecko patch for the mozharness bits and attach here for official review any time.  Since this is your first gecko patch (AFAIK), you may find this useful, or ping me or anyone else on the team: https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/How_to_Submit_a_Patch
Flags: needinfo?(jgriffin)
Assignee

Comment 5

4 years ago
Bug 1197224 - Add mozharness script to run firefox-media-tests in buildbot; r?jgriffin
Attachment #8658939 - Flags: review?(jgriffin)
Attachment #8658939 - Flags: review?(jgriffin) → review+
Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

https://reviewboard.mozilla.org/r/18729/#review16765

Looks good, let's give it a shot!

::: testing/mozharness/mozharness/mozilla/testing/firefox_media_tests.py:135
(Diff revision 1)
> +    def __init__(self, config_options=[], all_actions=[], **kwargs):

Passing a list as a default arg can have unintended consequences; better to use an empty tuple or None.
Assignee

Comment 7

4 years ago
Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

Bug 1197224 - Add mozharness script to run firefox-media-tests in buildbot; r?jgriffin
Attachment #8658939 - Flags: review+ → review?(jgriffin)
Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

https://reviewboard.mozilla.org/r/18729/#review16917
Attachment #8658939 - Flags: review?(jgriffin) → review+
https://hg.mozilla.org/mozilla-central/rev/d2457f064ba3
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla43
Assignee

Comment 11

4 years ago
Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

Bug 1197224 - Part 2 Fix path for gittool.py and git on Windows; r?jgriffin
Attachment #8658939 - Attachment description: MozReview Request: Bug 1197224 - Add mozharness script to run firefox-media-tests in buildbot; r?jgriffin → MozReview Request: Bug 1197224 - Part 2 Fix path for gittool.py and git on Windows; r?jgriffin
Attachment #8658939 - Flags: review+ → review?(jgriffin)
landed on cedar for testing:

remote: View your change here:
remote:   https://hg.mozilla.org/projects/cedar/rev/b821cdc017a8
remote: 
remote: Follow the progress of your build on Treeherder:
remote:   https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=b821cdc017a8
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Assignee

Comment 13

4 years ago
Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

Bug 1197224 - Part 2 Fix path for gittool.py and git on Windows; r?jgriffin
remote: View your change here:
remote:   https://hg.mozilla.org/projects/cedar/rev/35b623bc0b77
remote: 
remote: Follow the progress of your build on Treeherder:
remote:   https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=35b623bc0b77
Apparently, we don't have git on the Windows slaves.  Maja filed this as bug 1206877.  If this takes a long time to fix (which is my guess), the alternative is to package firefox-media-tests and dependencies, and upload them as needed to internal pypi.  This would remove the need to clone anything with git.
Assignee

Comment 16

4 years ago
We do have git on the Windows slaves -- the checkout step works -- it's just not in the default PATH, and a particular call to 'pip freeze' the in VirtualenvMixin relies on the default PATH. 

OS X 10.6 slaves actually don't have git installed. We could just not test that platform.
Assignee

Comment 17

4 years ago
Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

Bug 1197224 - Part 2 Fix path for gittool.py and git on Windows; r?jgriffin
Assignee

Comment 18

4 years ago
Bug 1197224 - Part 3 Don't install packages in editable mode; r?jgriffin
Attachment #8665079 - Flags: review?(jgriffin)
remote: View your change here:
remote:   https://hg.mozilla.org/projects/cedar/rev/78f26721ef4e
remote: 
remote: Follow the progress of your build on Treeherder:
remote:   https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=78f26721ef4e
Assignee

Comment 20

4 years ago
The 'checkout' and 'create-virtualenv' steps succeeded on Windows. Next problem: the call to firefox-media-tests fails, but whimboo pointed out that it looks like Bug 1198448. I'll try the same fix as my next step.
Assignee

Comment 21

4 years ago
Comment on attachment 8665079 [details]
MozReview Request: Bug 1197224 - Part 3 Don't install packages in editable mode; r?jgriffin

Bug 1197224 - Part 3 Don't install packages in editable mode; r?jgriffin
Assignee

Comment 22

4 years ago
Bug 1197224 - Part 4 Call tests via Python entry-point; r?jgriffin

Similarly to Bug 1198448, calling the firefox-media-tests binary
on Windows fails because it triggers a UAC prompt. Instead, we
call the same entry-point script via Python.
Attachment #8665597 - Flags: review?(jgriffin)
remote: View your change here:
remote:   https://hg.mozilla.org/projects/cedar/rev/769a602cfe63
remote: 
remote: Follow the progress of your build on Treeherder:
remote:   https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=769a602cfe63
Assignee

Comment 25

4 years ago
All jobs are now green, except on OS X 10.6, which does not have git installed. I propose that we do not include this platform given the small percentage of active users. (I'm told it's 1.5% of *Mac* Firefox users, which is < 1% of all Firefox users.) Does that sound good, jgriffin?

https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=210f3ed93368&filter-job_type_symbol=Mt
Assignee

Comment 26

4 years ago
Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

Correct path for gittool.py in Windows config. Don't install packages
in editable mode. Call media tests via Python entry point: similarly
to Bug 1198448, calling the firefox-media-tests binary
on Windows fails because it triggers a UAC prompt. Instead, we
call the same entry-point script via Python.
Attachment #8658939 - Attachment description: MozReview Request: Bug 1197224 - Part 2 Fix path for gittool.py and git on Windows; r?jgriffin → MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin
Assignee

Updated

4 years ago
Attachment #8665079 - Attachment is obsolete: true
Attachment #8665079 - Flags: review?(jgriffin)
Assignee

Updated

4 years ago
Attachment #8665597 - Attachment is obsolete: true
Attachment #8665597 - Flags: review?(jgriffin)
Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

https://reviewboard.mozilla.org/r/18729/#review18487

lgtm
Attachment #8658939 - Flags: review?(jgriffin) → review+
Assignee

Updated

4 years ago
Attachment #8658939 - Flags: review+ → review?(jgriffin)
Assignee

Comment 28

4 years ago
Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

Correct path for gittool.py in Windows config. Don't install packages
in editable mode. Call media tests via Python entry point: similarly
to Bug 1198448, calling the firefox-media-tests binary
on Windows fails because it triggers a UAC prompt. Instead, we
call the same entry-point script via Python.
Assignee

Comment 29

4 years ago
The naming-scheme of branches in the firefox-ui-tests repo just changed in bug 1209011, so I need to replace 'master' with 'mozilla-central' for the checkout step to work.
Attachment #8658939 - Flags: review?(jgriffin) → review+
Comment on attachment 8658939 [details]
MozReview Request: Bug 1197224 - Part 2: configure mozharness script to run media tests on Windows; r?jgriffin

https://reviewboard.mozilla.org/r/18729/#review18569
https://hg.mozilla.org/mozilla-central/rev/992ee69d7c2e
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Resolution: --- → FIXED
Target Milestone: mozilla43 → mozilla44

Updated

a year ago
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.