1197234 - [15Q4] Move firefox-media-tests in-tree

Status: RESOLVED FIXED

(Testing Graveyard :: external-media-tests, defect)

Description

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Currently, the firefox-media-tests repo is on Github: https://github.com/mjzffr/firefox-media-tests

Note that MediaTestCase extends FirefoxTestCase and therefore depends on firefox_puppeteer; we currently obtain this dependency by setting firefox-ui-tests as a submodule of firefox-media-tests.

My current understanding of how the project might look in-tree; I'm not sure where everything should go.
* testing/firefox-media-tests
* testing/firefox-ui-tests (maybe moved in-tree some time after firefox-media-tests)
* testing/firefox_puppeteer (split out from firefox-ui-tests; sync from github repo until firefox-ui-tests move in-tree? )
* testing/mozharness/scripts/firefox_media_tests.py (see bug 1197224)
* testing/mozharness/configs/firefox_media_tests_jenkins.py (config for our Jenkins jobs, including submission to Treeherder) [1]
* testing/mozharness/mozilla?
** treeherder_helper.py [2]
** treeherding.py, s3.py, parsers.py [3, 4, 5] (These should be consolidated with how firefox-ui-tests submit data to Treeherder; also useful to other non-buildbot jobs.) 

[1] https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/config/treeherder_submission.py
[2] https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/run_media_tests.py#L117-L285 Split this out of our current mozharness script; could be useful for other Tier-2/Tier-3/non-buildbot jobs.
[3] https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/treeherding.py
[4] https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/s3.py
[5] https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/parsers.py

Comment 1

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

n-i to jgriffin, whimboo, ahal for feedback

Comment 2

[Andrew Halberstadt [:ahal]]

That makes sense to me. If the mozharness setup for firefox_media_tests and firefox_ui_tests is more or less the same, you could make one script for the both of them. Then again, we have one script for all desktop unittests, and that got pretty messy.. so maybe separate is better.

Comment 3

[Jonathan Griffin (:jgriffin)]

> * testing/mozharness/scripts/firefox_media_tests.py (see bug 1197224)
> * testing/mozharness/configs/firefox_media_tests_jenkins.py (config for our Jenkins jobs, including 
> submission to Treeherder) [1]

You will also need a buildbot config file, which will specify things like the path to Python on the buildbot slaves.  Nearly all of this is boilerplate; see https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/marionette/prod_config.py for an example.

> * testing/mozharness/mozilla?
> ** treeherder_helper.py [2]
> ** treeherding.py, s3.py, parsers.py [3, 4, 5] (These should be consolidated with how firefox-
> ui-tests submit data to Treeherder; also useful to other non-buildbot jobs.) 

This makes sense, I think, for non-buildbot/TC jobs that will want to use mozharness scripts to submit data to Treeherder.  Whatever isn't generic should probably just be added to the "scripts" dir and not to mozharness/mozilla.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Maja Frydrychowicz (:maja_zf) from comment #0)
> My current understanding of how the project might look in-tree; I'm not sure
> where everything should go.
> * testing/firefox-media-tests
> * testing/firefox-ui-tests (maybe moved in-tree some time after
> firefox-media-tests)
> * testing/firefox_puppeteer (split out from firefox-ui-tests; sync from
> github repo until firefox-ui-tests move in-tree? )

This is still something we should figure out thoughtfully. Usually under /testing only find harness code but no tests. Those are usually under the appropriate component. But having a bundle of tests of different components makes it hard. Here we would definitely need info from the appropriate people. At the end it could even mean that we have to split up our firefox-ui-tests folders, which then get bundled as they look nowadays inside the tests.zip archive.

> * testing/mozharness/scripts/firefox_media_tests.py (see bug 1197224)

In case of our tests we will have several scripts here. For now it is firefox-ui-updates.py, and I will add firefox-ui-functional.py soon. So if it's only a single type of tests you have, this looks fine.

> * testing/mozharness/configs/firefox_media_tests_jenkins.py (config for our
> Jenkins jobs, including submission to Treeherder) [1]

IMHO this is something you should do in the repo for your jenkins instance. You might want to hvae a look at https://github.com/mozilla/mozmill-ci.

> * testing/mozharness/mozilla?
> ** treeherder_helper.py [2]
> ** treeherding.py, s3.py, parsers.py [3, 4, 5] (These should be consolidated
> with how firefox-ui-tests submit data to Treeherder; also useful to other
> non-buildbot jobs.) 

Something you should talk with the mozharness owners if that is wanted. The code as is might need a refactoring.

So as conclusion it sounds fine to go ahead but please make sure that we keep the code uptodate in m-c. I don't want a fork of firefox-puppeteer which is then very hard to handle once we go to mc.

Comment 5

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Here's a more up-to-date plan of where things should go. For now, the media tests themselves will also go under /testing and we'll try to move them somewhere near the code they are testing soon/later.

testing/firefox-media-tests (includes tests, harness, libraries for interacting with video elements)
testing/firefox_puppeteer (copied from firefox-ui-tests Github repo)
testing/mozharness
  configs/mediatests/
    buildbot_windows_config.py*
    buildbot_posix_config.py*	
  mozharness/mozilla/testing/firefox_media_tests.py* 		
  scripts/firefox_media_tests_buildbot.py*

I'm not including scripts related to our Jenkins instance (and therefore Treeherder submission), since that can live in https://github.com/sydvicious/mozplatformqa-jenkins, as Henrik suggests.

* Already in m-c (Bug 1197224)

Comment 6

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

jgriffin, a while a go we talked about running these media jobs in Jenkins once firefox-media-tests is in m-c and you explained that the tests could be packaged in a tests.zip, which could then be acquired by a Jenkins job. 

What about the media test harness and firefox_puppeteer? Would that also go in a tests.zip? Should I open a separate bug for packaging new things in tests.zip?

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Maja, I would like to mention that I work on mozharness scripts for our firefox-ui-tests so it can easily be run in Jenkins without any other file stored in our Jenkins config. All the work is done via bug  	1197358. I would kinda be interested to see the code you have in your patch.

Comment 8

[Jonathan Griffin (:jgriffin)]

Anything in the tree can get packaged into a tests.zip file; you can file a separate bug for that. If you need e.g., puppeteer before it's in the tree, uploading it to our internal pypi (http://pypi.pub.build.mozilla.org/pub/) is a good way to go. To upload something to internal pypi, you can file a bug similar to bug 1204676.

Comment 9

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Thanks, Henrik.

(In reply to Henrik Skupin (:whimboo) from comment #7)
> ...without any other file stored in our Jenkins config. 

I'm not sure what you're referring to here. Perhaps we're misunderstanding each other?

> I would kinda be interested to see the code you have in your patch.

* Last quarter I wrote a mozharness script for running firefox-media-tests in pf-jenkins; it takes care of Treeherder submission as well: https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/run_media_tests.py
* I recently refactored the above script (see Bug 1197224) into a base script for media tests, a BuildBot-specific script, and a Jenkins-specific script (not yet deployed)

All the above scripts can perform a Github checkout to get the media tests. (However, in our Jenkins instance, the Jenkins jobs themselves take care of cloning the Github test repo.)

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Maja Frydrychowicz (:maja_zf) from comment #9)
> > ...without any other file stored in our Jenkins config. 
> 
> I'm not sure what you're referring to here. Perhaps we're misunderstanding
> each other?

Maybe. We only talked about the way how to get the mozharness code. So I showed you the job configuration for our Jenkins instance including the archiver tool. 

Beside this script you shouldn't need any other script located in our jenkins repo. This would include s3.py for now given that there is a blob_uploader Mixin in mozharness, which can do that for you. Sadly there is nothing for treeherder yet, but that is something I want to tackle in the near future. Maybe your code can be updated once that got landed in mozbase proper.

> > I would kinda be interested to see the code you have in your patch.
> 
> * Last quarter I wrote a mozharness script for running firefox-media-tests
> in pf-jenkins; it takes care of Treeherder submission as well:
> https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/
> run_media_tests.py
> * I recently refactored the above script (see Bug 1197224) into a base
> script for media tests, a BuildBot-specific script, and a Jenkins-specific
> script (not yet deployed)

Please keep in mind that we already have a base firefox-ui-tests class available under:
http://mxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/firefox_ui_tests.py

I would assume that you could build on top of it just to reduce the amount of code duplication.

Comment 11

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Our latest conversation about this (with :AutomatedTest, :whimboo, :sydpolk) concluded that we will wait to move firefox-media-tests and firefox-ui-tests in-tree together at the same time, consolidating as appropriate.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This bug might become the tracking bug for our work in quarter Q4 to get both kind of tests into mozilla-central.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We want to complete the work this quarter.

Comment 14

[Syd Polk :sydpolk]

This has been in-tree for a while now.