Closed Bug 1197234 Opened 4 years ago Closed 4 years ago

[15Q4] Move firefox-media-tests in-tree

Categories

(Testing Graveyard :: external-media-tests, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: maja_zf, Assigned: sydpolk)

References

Details

(Keywords: meta)

Currently, the firefox-media-tests repo is on Github: https://github.com/mjzffr/firefox-media-tests

Note that MediaTestCase extends FirefoxTestCase and therefore depends on firefox_puppeteer; we currently obtain this dependency by setting firefox-ui-tests as a submodule of firefox-media-tests.

My current understanding of how the project might look in-tree; I'm not sure where everything should go.
* testing/firefox-media-tests
* testing/firefox-ui-tests (maybe moved in-tree some time after firefox-media-tests)
* testing/firefox_puppeteer (split out from firefox-ui-tests; sync from github repo until firefox-ui-tests move in-tree? )
* testing/mozharness/scripts/firefox_media_tests.py (see bug 1197224)
* testing/mozharness/configs/firefox_media_tests_jenkins.py (config for our Jenkins jobs, including submission to Treeherder) [1]
* testing/mozharness/mozilla?
** treeherder_helper.py [2]
** treeherding.py, s3.py, parsers.py [3, 4, 5] (These should be consolidated with how firefox-ui-tests submit data to Treeherder; also useful to other non-buildbot jobs.) 

[1] https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/config/treeherder_submission.py
[2] https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/run_media_tests.py#L117-L285 Split this out of our current mozharness script; could be useful for other Tier-2/Tier-3/non-buildbot jobs.
[3] https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/treeherding.py
[4] https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/s3.py
[5] https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/parsers.py
n-i to jgriffin, whimboo, ahal for feedback
Flags: needinfo?(jgriffin)
Flags: needinfo?(hskupin)
Flags: needinfo?(ahalberstadt)
That makes sense to me. If the mozharness setup for firefox_media_tests and firefox_ui_tests is more or less the same, you could make one script for the both of them. Then again, we have one script for all desktop unittests, and that got pretty messy.. so maybe separate is better.
Flags: needinfo?(ahalberstadt)
> * testing/mozharness/scripts/firefox_media_tests.py (see bug 1197224)
> * testing/mozharness/configs/firefox_media_tests_jenkins.py (config for our Jenkins jobs, including 
> submission to Treeherder) [1]

You will also need a buildbot config file, which will specify things like the path to Python on the buildbot slaves.  Nearly all of this is boilerplate; see https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/marionette/prod_config.py for an example.

> * testing/mozharness/mozilla?
> ** treeherder_helper.py [2]
> ** treeherding.py, s3.py, parsers.py [3, 4, 5] (These should be consolidated with how firefox-
> ui-tests submit data to Treeherder; also useful to other non-buildbot jobs.) 

This makes sense, I think, for non-buildbot/TC jobs that will want to use mozharness scripts to submit data to Treeherder.  Whatever isn't generic should probably just be added to the "scripts" dir and not to mozharness/mozilla.
Flags: needinfo?(jgriffin)
(In reply to Maja Frydrychowicz (:maja_zf) from comment #0)
> My current understanding of how the project might look in-tree; I'm not sure
> where everything should go.
> * testing/firefox-media-tests
> * testing/firefox-ui-tests (maybe moved in-tree some time after
> firefox-media-tests)
> * testing/firefox_puppeteer (split out from firefox-ui-tests; sync from
> github repo until firefox-ui-tests move in-tree? )

This is still something we should figure out thoughtfully. Usually under /testing only find harness code but no tests. Those are usually under the appropriate component. But having a bundle of tests of different components makes it hard. Here we would definitely need info from the appropriate people. At the end it could even mean that we have to split up our firefox-ui-tests folders, which then get bundled as they look nowadays inside the tests.zip archive.

> * testing/mozharness/scripts/firefox_media_tests.py (see bug 1197224)

In case of our tests we will have several scripts here. For now it is firefox-ui-updates.py, and I will add firefox-ui-functional.py soon. So if it's only a single type of tests you have, this looks fine.

> * testing/mozharness/configs/firefox_media_tests_jenkins.py (config for our
> Jenkins jobs, including submission to Treeherder) [1]

IMHO this is something you should do in the repo for your jenkins instance. You might want to hvae a look at https://github.com/mozilla/mozmill-ci.

> * testing/mozharness/mozilla?
> ** treeherder_helper.py [2]
> ** treeherding.py, s3.py, parsers.py [3, 4, 5] (These should be consolidated
> with how firefox-ui-tests submit data to Treeherder; also useful to other
> non-buildbot jobs.) 

Something you should talk with the mozharness owners if that is wanted. The code as is might need a refactoring.

So as conclusion it sounds fine to go ahead but please make sure that we keep the code uptodate in m-c. I don't want a fork of firefox-puppeteer which is then very hard to handle once we go to mc.
Flags: needinfo?(hskupin)
Here's a more up-to-date plan of where things should go. For now, the media tests themselves will also go under /testing and we'll try to move them somewhere near the code they are testing soon/later.

testing/firefox-media-tests (includes tests, harness, libraries for interacting with video elements)
testing/firefox_puppeteer (copied from firefox-ui-tests Github repo)
testing/mozharness
  configs/mediatests/
    buildbot_windows_config.py*
    buildbot_posix_config.py*	
  mozharness/mozilla/testing/firefox_media_tests.py* 		
  scripts/firefox_media_tests_buildbot.py*

I'm not including scripts related to our Jenkins instance (and therefore Treeherder submission), since that can live in https://github.com/sydvicious/mozplatformqa-jenkins, as Henrik suggests.

* Already in m-c (Bug 1197224)
jgriffin, a while a go we talked about running these media jobs in Jenkins once firefox-media-tests is in m-c and you explained that the tests could be packaged in a tests.zip, which could then be acquired by a Jenkins job. 

What about the media test harness and firefox_puppeteer? Would that also go in a tests.zip? Should I open a separate bug for packaging new things in tests.zip?
Flags: needinfo?(jgriffin)
Maja, I would like to mention that I work on mozharness scripts for our firefox-ui-tests so it can easily be run in Jenkins without any other file stored in our Jenkins config. All the work is done via bug  	1197358. I would kinda be interested to see the code you have in your patch.
Anything in the tree can get packaged into a tests.zip file; you can file a separate bug for that. If you need e.g., puppeteer before it's in the tree, uploading it to our internal pypi (http://pypi.pub.build.mozilla.org/pub/) is a good way to go. To upload something to internal pypi, you can file a bug similar to bug 1204676.
Flags: needinfo?(jgriffin)
Thanks, Henrik.

(In reply to Henrik Skupin (:whimboo) from comment #7)
> ...without any other file stored in our Jenkins config. 

I'm not sure what you're referring to here. Perhaps we're misunderstanding each other?

> I would kinda be interested to see the code you have in your patch.

* Last quarter I wrote a mozharness script for running firefox-media-tests in pf-jenkins; it takes care of Treeherder submission as well: https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/run_media_tests.py
* I recently refactored the above script (see Bug 1197224) into a base script for media tests, a BuildBot-specific script, and a Jenkins-specific script (not yet deployed)

All the above scripts can perform a Github checkout to get the media tests. (However, in our Jenkins instance, the Jenkins jobs themselves take care of cloning the Github test repo.)
Flags: needinfo?(hskupin)
(In reply to Maja Frydrychowicz (:maja_zf) from comment #9)
> > ...without any other file stored in our Jenkins config. 
> 
> I'm not sure what you're referring to here. Perhaps we're misunderstanding
> each other?

Maybe. We only talked about the way how to get the mozharness code. So I showed you the job configuration for our Jenkins instance including the archiver tool. 

Beside this script you shouldn't need any other script located in our jenkins repo. This would include s3.py for now given that there is a blob_uploader Mixin in mozharness, which can do that for you. Sadly there is nothing for treeherder yet, but that is something I want to tackle in the near future. Maybe your code can be updated once that got landed in mozbase proper.

> > I would kinda be interested to see the code you have in your patch.
> 
> * Last quarter I wrote a mozharness script for running firefox-media-tests
> in pf-jenkins; it takes care of Treeherder submission as well:
> https://github.com/mjzffr/firefox-media-tests/blob/pf-jenkins/
> run_media_tests.py
> * I recently refactored the above script (see Bug 1197224) into a base
> script for media tests, a BuildBot-specific script, and a Jenkins-specific
> script (not yet deployed)

Please keep in mind that we already have a base firefox-ui-tests class available under:
http://mxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/firefox_ui_tests.py

I would assume that you could build on top of it just to reduce the amount of code duplication.
Flags: needinfo?(hskupin)
Our latest conversation about this (with :AutomatedTest, :whimboo, :sydpolk) concluded that we will wait to move firefox-media-tests and firefox-ui-tests in-tree together at the same time, consolidating as appropriate.
Assignee: mjzffr → nobody
No longer blocks: 1185001
Component: firefox-media-tests → General
Summary: Move firefox-media-tests in-tree → Move firefox-media-tests and firefox-ui-tests in-tree
This bug might become the tracking bug for our work in quarter Q4 to get both kind of tests into mozilla-central.
We want to complete the work this quarter.
Component: General → Marionette
Keywords: meta
Summary: Move firefox-media-tests and firefox-ui-tests in-tree → [15Q4] Move firefox-media-tests and firefox-ui-tests in-tree
Assignee: nobody → spolk
Depends on: 1212609
Summary: [15Q4] Move firefox-media-tests and firefox-ui-tests in-tree → [15Q4] Move firefox-media-tests in-tree
Component: Marionette → firefox-media-tests
Depends on: 1235606
This has been in-tree for a while now.
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.