1159594 - Enable B and Gij test suite for TV on Treeherder

Status: RESOLVED FIXED

(Taskcluster :: General, defect)

Description

[Ricky Chien [:rickychien] (inactive)]

As title described, to protect TV features to break our code base, Gaia integration test should be enabled by default for supporting stability. 
Since build process of TV is different from original phone build, we have to pass different ENV parameters while trying a tv build.

I don't know what treeherder would look like after enabling TV tests. Do you guys have any ideas about it? 

Thanks!

Comment 1

[Chris Cooper [:coop] (he/him)]

Passing the buck to sheriffs and a-team who can answer your treeherder questions better than I.

Comment 2

[William Lachance (:wlach)]

James Lal's ([e-mail]) FirefoxOS automation team is probably the best first point of contact, as they've made other Gaia/B2G stuff submit to treeherder.

Comment 3

[William Lachance (:wlach)]

CC'ing James here

Comment 4

[Ed Morley [:emorley]]

Moved to Taskcluster, treeherder doesn't schedule tests

Comment 5

[Ricky Chien [:rickychien] (inactive)]

Hi James, do you know how a-team can help us enable new B and Gij test suite on treeherder? Thanks!

Comment 6

[Ricky Chien [:rickychien] (inactive)]

Hi Aus, could you provide some suggestions here thanks!

Comment 7

[Aus Lacroix [:aus]]

B tests should be simple. Add a new build test for the TV target or, simply change the environment and call the correct make target you wish to test. This is probably the test file you want to add your test to right here: 

Gij test run as it currently exists is defined here (https://github.com/mozilla-b2g/gaia/blob/master/tests/taskcluster/tasks/marionette_js_tests.yml). 

The best way to get your own run of the test suite with the correct environment would be to copy the existing Gij definition file and use it as the base for the TV Gij suite. The filename is important, you need it later, maybe something like marionette_js_tv_tests.yml.

You will want to edit the name and description, add your environment variables under "env:" and change the treeherder groupSymbol used (maybe something like TVGij... or whatever you think is appropriate.

Last thing, edit this file here: https://github.com/mozilla-b2g/gaia/blob/master/tests/taskcluster/tasks.yml and add a section like this:

marionette_js_tv_tests:
  chunks: 30

Make sure there's an empty line before and after the content. The filename you chose previously when copying marionette_js_tests.yml is what you use *without* the .yml part.

I'm happy to review your changes once you have a patch (pull requests / gaia-try will work to test all this too, btw)

Comment 8

[James Lal [:lightsofapollo]]

FTR- If your adding a new suite to gaia and it's green feel free to land without a R? from us if needed. This means you are owners of the suite (which is great :)) but if something needs to be turned off for some reason we will get a hold of you.

Comment 9

[GitHub Autolander Bot]

Attachment: [gaia] shamenchens:Bug1159594-EnableTvTestSuite > mozilla-b2g:master

Comment 10

[Sherman Chen [:chens] (inactive)]

Comment on attachment 8607400 [details] [review]
[gaia] shamenchens:Bug1159594-EnableTvTestSuite > mozilla-b2g:master

Aus, could you review this patch on tasks.yml and marionette_js_tv_tests.yml? thanks.

Comment 11

[Sherman Chen [:chens] (inactive)]

Comment on attachment 8607400 [details] [review]
[gaia] shamenchens:Bug1159594-EnableTvTestSuite > mozilla-b2g:master

Ricky, John, could you take a look on it and give some feedback? this patch add test options to gaia integration test and run on gij-tv test suites.

Comment 12

[John Hu [:johnhu][:johu][:醬糊小弟]]

Comment on attachment 8607400 [details] [review]
[gaia] shamenchens:Bug1159594-EnableTvTestSuite > mozilla-b2g:master

Looks good to me for those integration tests. Thanks.

Comment 13

[Ricky Chien [:rickychien] (inactive)]

Comment on attachment 8607400 [details] [review]
[gaia] shamenchens:Bug1159594-EnableTvTestSuite > mozilla-b2g:master

f+ for the patch.

I'm glad to see TC team's great work. Gaia developers could interact with these test suites straightforwardly without submitting any mozharness patch. I can see GijTV is running on treeherder now!! \o/

Comment 14

[Sherman Chen [:chens] (inactive)]

Currently tv don't have much tests to run on so removed chunks option in marionette_js_tv_tests task, we could add chunks back when tv tests grow.
One more thing to notice, currently chunks are divided and sorted by test files[1], that means tv tests will still sit in the same chunk even we increased number of chunks, we may need proper way to fix it in the future.

[1]: https://github.com/mozilla-b2g/gaia/blob/master/bin/gaia-marionette#L65-L67

Comment 15

[Aus Lacroix [:aus]]

Comment on attachment 8607400 [details] [review]
[gaia] shamenchens:Bug1159594-EnableTvTestSuite > mozilla-b2g:master

Overall this looks good but I have some questions and comments in the GitHub PR :) You can set the review flag again once we're ready to have another look.

Comment 16

[Sherman Chen [:chens] (inactive)]

Comment on attachment 8607400 [details] [review]
[gaia] shamenchens:Bug1159594-EnableTvTestSuite > mozilla-b2g:master

Aus, I've answered those question and would you like to review again? thanks!

Comment 17

[Aus Lacroix [:aus]]

Comment on attachment 8607400 [details] [review]
[gaia] shamenchens:Bug1159594-EnableTvTestSuite > mozilla-b2g:master

Looks good! Thanks for taking the time to answer my questions. It was very helpful. :)

Comment 18

[GitHub Autolander Bot]

Autolander could not locate a review from a user within the suggested reviewer list. Either the patch author or the reviewer should be in the suggested reviewer list.

Comment 19

[Sherman Chen [:chens] (inactive)]

master: https://github.com/mozilla-b2g/gaia/commit/6c92502f48a330d3be267263cba88351d60519db

Comment 20

[Pete Moore [:pmoore][:pete]]

Resetting Version and Target Milestone that accidentally got changed...