1056921 - Add support for mochitest-jetpack tests & enable on cedar

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dave Townsend [:mossop]]

We're switching the jetpack test harness to use the mochitest python code to run Firefox, this will give us many benefits like screenshot on fail, crash stacks etc.

I have landed the harness code on Gum and now I want to get the new test jobs created and running there.

The tests consist of two different jobs. jetpack-addon and jetpack-package. Both are run in exactly the same way as browser-chrome are, through runtests.py so hopefully this should be some pretty straightforward copy and pasting to get going.

I have some blind copy-paste patches that might be a good start to attach in a moment.

Once this all works on gum the plan would be to land the code on m-c and turn off the older JP tests there and turn on the new ones instead so this shouldn't use any more resources than at present.

Comment 1

[Dave Townsend [:mossop]]

Attachment: mozharness changes

Comment 2

[Dave Townsend [:mossop]]

Attachment: buildbotcustom changes

Does some additional checks for parsing the jetpack test output in the same way as browser-chrome

Comment 3

[Dave Townsend [:mossop]]

Attachment: buildbot-configs changes

Not really sure this is right and I haven't attempted to add it to gum's config as it wasn't obvious how to do that.

Comment 4

[Dave Townsend [:mossop]]

I'm not really sure who is right to look at these patches or help me get this working, catlee can you help me find someone?

Comment 5

[Chris AtLee [:catlee]]

Jordan, can you have a look?

Comment 6

[Jordan Lund (:jlund)]

sure np!

I'm just back from PTO so my queue was abnormally large today. I'll have a peak tomorrow.

Comment 7

[Jordan Lund (:jlund)]

Comment on attachment 8476783 [details] [diff] [review]
buildbotcustom changes

Review of attachment 8476783 [details] [diff] [review]:
-----------------------------------------------------------------

I think you've got a large chunk of it right but IIUC, we may need to tweak it a bit.

first, let's see if we are on the same page:

1) we currently run jetpack via UnittestPackagedBuildFactory[1]?

2) we want to stop doing that and use mochitest to run jetpack ?

if so, I think this patch is not needed. Mochitest suites are now run through mozharness. Mozharness scripts are called via ScriptFactory[2] not UnittestPackagedBuildFactory.

hopefully all we need here is to add jetpack mochitest builders (your buildbot-config patch) and then support those builders in mozharness (your mozharness patch)

I'll look at those patches now.


[1] http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#4892
[2] http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#6049

Comment 8

[Jordan Lund (:jlund)]

Comment on attachment 8476782 [details] [diff] [review]
mozharness changes

Review of attachment 8476782 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/unittests/linux_unittest.py
@@ +72,5 @@
>          "mochitest-devtools-chrome-2": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5", "--total-chunks=3", "--this-chunk=2"],
>          "mochitest-devtools-chrome-3": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5", "--total-chunks=3", "--this-chunk=3"],
>          "mochitest-devtools-chrome-chunked": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5"],
> +        "jetpack-package": ["--jetpack-package"],
> +        "jetpack-addon": ["--jetpack-addon"],

to clarify, do we want to:

1) run one job and pass --jetpack-package and --jetpack-addon to runtests.py in a single call

2) run one job and call runtests.py twice, once with --jetpack-package and once with --jetpack-addon

3) run two jobs and call runtests.py only once for each job. passing --jetpack-package for one job and --jetpack-addon for the second

up till now we have pretty much created one job for each suite (mochitest chunk 1, xpcshell, etc). Mozharness can do more than one suite(single runtests.py call) per script run(job) but if we want to do that, we might need to play with some dials: timeout limits, which suite goes first, etc.

Comment 9

[Jordan Lund (:jlund)]

Comment on attachment 8476784 [details] [diff] [review]
buildbot-configs changes

Review of attachment 8476784 [details] [diff] [review]:
-----------------------------------------------------------------

I think this puts the pieces into place but it will not be run anywhere. i.e MOCHITEST_JP is never used

you are going to want to specify which branch(es) you want these suites to run on. Right now we currently use Cedar to test out new or failing jobs. We should probably limit this out to there until we green them up. For that you will need something like: http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config.py#1912

does this make sense?

::: mozilla-tests/config.py
@@ +386,5 @@
> +MOCHITEST_JP = [
> +    ('mochitest-jetpack', {
> +        'use_mozharness': True,
> +        'script_path': 'scripts/desktop_unittest.py',
> +        'extra_args': ['--mochitest-suite', 'jetpack-package,jetpack-addon'],

this will need to change depending on your answer to the question I ask in your buildbot-config patch.

Comment 10

[Dave Townsend [:mossop]]

(In reply to Jordan Lund (:jlund) from comment #8)
> Comment on attachment 8476782 [details] [diff] [review]
> mozharness changes
> 
> Review of attachment 8476782 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: configs/unittests/linux_unittest.py
> @@ +72,5 @@
> >          "mochitest-devtools-chrome-2": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5", "--total-chunks=3", "--this-chunk=2"],
> >          "mochitest-devtools-chrome-3": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5", "--total-chunks=3", "--this-chunk=3"],
> >          "mochitest-devtools-chrome-chunked": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5"],
> > +        "jetpack-package": ["--jetpack-package"],
> > +        "jetpack-addon": ["--jetpack-addon"],
> 
> to clarify, do we want to:
> 
> 1) run one job and pass --jetpack-package and --jetpack-addon to runtests.py
> in a single call
> 
> 2) run one job and call runtests.py twice, once with --jetpack-package and
> once with --jetpack-addon
> 
> 3) run two jobs and call runtests.py only once for each job. passing
> --jetpack-package for one job and --jetpack-addon for the second
> 
> up till now we have pretty much created one job for each suite (mochitest
> chunk 1, xpcshell, etc). Mozharness can do more than one suite(single
> runtests.py call) per script run(job) but if we want to do that, we might
> need to play with some dials: timeout limits, which suite goes first, etc.

I don't know that this matters much. Right now I think JP tests run as (2). These two tests tend to test the same code so it makes sense to just have them in one box on tbpl. If there is a standard way that makes most sense then let's just do that.

(In reply to Jordan Lund (:jlund) from comment #9)
> Comment on attachment 8476784 [details] [diff] [review]
> buildbot-configs changes
> 
> Review of attachment 8476784 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> I think this puts the pieces into place but it will not be run anywhere. i.e
> MOCHITEST_JP is never used
> 
> you are going to want to specify which branch(es) you want these suites to
> run on. Right now we currently use Cedar to test out new or failing jobs. We
> should probably limit this out to there until we green them up. For that you
> will need something like:
> http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config.
> py#1912
> 
> does this make sense?
> 
> ::: mozilla-tests/config.py
> @@ +386,5 @@
> > +MOCHITEST_JP = [
> > +    ('mochitest-jetpack', {
> > +        'use_mozharness': True,
> > +        'script_path': 'scripts/desktop_unittest.py',
> > +        'extra_args': ['--mochitest-suite', 'jetpack-package,jetpack-addon'],
> 
> this will need to change depending on your answer to the question I ask in
> your buildbot-config patch.

I didn't add this anywhere yet because I didn't really understand how to. The code to land and ship these harnesses in the test package is landed on Gum and I anticipated that there might need to be some tweaking both to the mozharness side and the m-c side to get it working. If it makes more sense to do that on cedar then that works.

Once this all works these should be enabled like any other tests on m-c and related branches and then ride the trains down to release.

Comment 11

[Dave Townsend [:mossop]]

Any progress here?

Comment 12

[Jordan Lund (:jlund)]

(In reply to Dave Townsend [:mossop] from comment #11)
> Any progress here?

Oh sorry I thought you just needed feedback on your patch and I missed your follow-up comments. I am pretty busy trying to wrap up some goals so if you would like me to drive this I'll run it by my manager. Hopefully though there isn't much work. You might be very close.

Might just a matter of taking this:
'extra_args': ['--mochitest-suite', 'jetpack-package,jetpack-addon']

And instead doing:
'extra_args': ['--mochitest-suite', 'jetpack-package', --mochitest-suite', 'jetpack-addon']

To accomplish number two mentioned above. And then to a enable on Cedar or gum. I suppose gum makes the most sense. Again in a enabling on a single branch is done via: http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config.py#1912

Feel free to make those two changes and I'll review. Then we can iterate on gum :)

Comment 13

[Dave Townsend [:mossop]]

Attachment: buildbot-configs changes

Ok here are those changes.

Comment 14

[Dave Townsend [:mossop]]

Comment on attachment 8476782 [details] [diff] [review]
mozharness changes

Sounds like this bit is ready to land?

Comment 15

[Jordan Lund (:jlund)]

Comment on attachment 8476782 [details] [diff] [review]
mozharness changes

Review of attachment 8476782 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm. this should be a good start. We may need to tweak mozharness more depending on how this does on gum

Comment 16

[Jordan Lund (:jlund)]

Comment on attachment 8486818 [details] [diff] [review]
buildbot-configs changes

Review of attachment 8486818 [details] [diff] [review]:
-----------------------------------------------------------------

Awesome, this looks good to me.

I think you meant to put gum instead of Cedar (see in-line). But I'm just thinking, I may be wrong about doing this on gum. cedar has the benefit of using mozharness 'default' branch instead of production. Production is used on most branches including gum. So if we need to make future changes, at least we can see the results on Cedar before merging possible broken patches to production.

You mentioned you didn't mind where we test this so how about we leave your patch as is but fix the comment mentioning gum? r+ to land on default any time.

::: mozilla-tests/config.py
@@ +1971,5 @@
>          if slave_platform in BRANCHES['cedar']['platforms'][platform]:
>              BRANCHES['cedar']['platforms'][platform][slave_platform]['opt_unittest_suites'] += MOZBASE[:]
>              BRANCHES['cedar']['platforms'][platform][slave_platform]['debug_unittest_suites'] += MOZBASE[:]
>  
> +# Enable webapprt-chrome tests on gum

gum mentioned here

@@ +1977,5 @@
> +    for slave_platform in PLATFORMS[platform]['slave_platforms']:
> +        if slave_platform not in BRANCHES['gum']['platforms'][platform]:
> +            continue
> +        BRANCHES['cedar']['platforms'][platform][slave_platform]['opt_unittest_suites'] += MOCHITEST_JP[:]
> +        BRANCHES['cedar']['platforms'][platform][slave_platform]['debug_unittest_suites'] += MOCHITEST_JP[:]

cedar is used for the above two lines

Comment 17

[Jordan Lund (:jlund)]

We will also need a tbpl patch so the job shows correctly. Without patch it might show up as some default like M instead of a custom one like MJP. This isn't such a big deal for Cedar but it will need to be done.

I will ping Sheriffs to see if they have docs for this.

Comment 18

[Dave Townsend [:mossop]]

Landed:

https://hg.mozilla.org/build/buildbot-configs/rev/4092544e664c
https://hg.mozilla.org/build/mozharness/rev/4a63895bd8b2

Comment 19

[Simone Bruno [:simone]]

In prod with reconfig on 2014-09-15 08:30 PT

Comment 20

[Dave Townsend [:mossop]]

All the appropriate things have landed now. How do I get cedar updated to the latest m-c so we can see how the tests are running?

Comment 21

[Ed Morley [:emorley]]

(In reply to Dave Townsend [:mossop] from comment #20)
> All the appropriate things have landed now. How do I get cedar updated to
> the latest m-c so we can see how the tests are running?

hg clone https://hg.mozilla.org/projects/cedar && cd cedar && hg pull https://hg.mozilla.org/mozilla-central && hg merge && hg commit -m "Merge mozilla-central into cedar" && hg push ssh://hg.mozilla.org/projects/cedar

Comment 22

[Dave Townsend [:mossop]]

(In reply to Ed Morley [:edmorley] from comment #21)
> (In reply to Dave Townsend [:mossop] from comment #20)
> > All the appropriate things have landed now. How do I get cedar updated to
> > the latest m-c so we can see how the tests are running?
> 
> hg clone https://hg.mozilla.org/projects/cedar && cd cedar && hg pull
> https://hg.mozilla.org/mozilla-central && hg merge && hg commit -m "Merge
> mozilla-central into cedar" && hg push ssh://hg.mozilla.org/projects/cedar

Unfortunately there are merge conflicts in the web platform tests that it looks like are being experimented on on cedar right now so I'm not sure I know enough to resolve them.

Comment 23

[Ed Morley [:emorley]]

(In reply to Dave Townsend [:mossop] from comment #22)
> Unfortunately there are merge conflicts in the web platform tests that it
> looks like are being experimented on on cedar right now so I'm not sure I
> know enough to resolve them.

Ah - James I don't suppose you could do an m-c import into Cedar for Dave? :-)

Comment 24

[James Graham [:jgraham]]

Something like https://hg.mozilla.org/projects/cedar/rev/649768be4000 ?

Comment 25

[Dave Townsend [:mossop]]

(In reply to James Graham [:jgraham] from comment #24)
> Something like https://hg.mozilla.org/projects/cedar/rev/649768be4000 ?

Wonderful, thanks