The default bug view has changed. See this FAQ.

Add support for mochitest-jetpack tests & enable on cedar

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: mossop, Assigned: mossop)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 2 obsolete attachments)

(Assignee)

Description

3 years ago
We're switching the jetpack test harness to use the mochitest python code to run Firefox, this will give us many benefits like screenshot on fail, crash stacks etc.

I have landed the harness code on Gum and now I want to get the new test jobs created and running there.

The tests consist of two different jobs. jetpack-addon and jetpack-package. Both are run in exactly the same way as browser-chrome are, through runtests.py so hopefully this should be some pretty straightforward copy and pasting to get going.

I have some blind copy-paste patches that might be a good start to attach in a moment.

Once this all works on gum the plan would be to land the code on m-c and turn off the older JP tests there and turn on the new ones instead so this shouldn't use any more resources than at present.
(Assignee)

Comment 1

3 years ago
Created attachment 8476782 [details] [diff] [review]
mozharness changes
(Assignee)

Comment 2

3 years ago
Created attachment 8476783 [details] [diff] [review]
buildbotcustom changes

Does some additional checks for parsing the jetpack test output in the same way as browser-chrome
(Assignee)

Comment 3

3 years ago
Created attachment 8476784 [details] [diff] [review]
buildbot-configs changes

Not really sure this is right and I haven't attempted to add it to gum's config as it wasn't obvious how to do that.
(Assignee)

Comment 4

3 years ago
I'm not really sure who is right to look at these patches or help me get this working, catlee can you help me find someone?
Flags: needinfo?(catlee)
Jordan, can you have a look?
Flags: needinfo?(catlee) → needinfo?(jlund)

Comment 6

3 years ago
sure np!

I'm just back from PTO so my queue was abnormally large today. I'll have a peak tomorrow.
Flags: needinfo?(jlund)

Comment 7

3 years ago
Comment on attachment 8476783 [details] [diff] [review]
buildbotcustom changes

Review of attachment 8476783 [details] [diff] [review]:
-----------------------------------------------------------------

I think you've got a large chunk of it right but IIUC, we may need to tweak it a bit.

first, let's see if we are on the same page:

1) we currently run jetpack via UnittestPackagedBuildFactory[1]?

2) we want to stop doing that and use mochitest to run jetpack ?

if so, I think this patch is not needed. Mochitest suites are now run through mozharness. Mozharness scripts are called via ScriptFactory[2] not UnittestPackagedBuildFactory.

hopefully all we need here is to add jetpack mochitest builders (your buildbot-config patch) and then support those builders in mozharness (your mozharness patch)

I'll look at those patches now.


[1] http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#4892
[2] http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#6049
Attachment #8476783 - Flags: review-

Comment 8

3 years ago
Comment on attachment 8476782 [details] [diff] [review]
mozharness changes

Review of attachment 8476782 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/unittests/linux_unittest.py
@@ +72,5 @@
>          "mochitest-devtools-chrome-2": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5", "--total-chunks=3", "--this-chunk=2"],
>          "mochitest-devtools-chrome-3": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5", "--total-chunks=3", "--this-chunk=3"],
>          "mochitest-devtools-chrome-chunked": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5"],
> +        "jetpack-package": ["--jetpack-package"],
> +        "jetpack-addon": ["--jetpack-addon"],

to clarify, do we want to:

1) run one job and pass --jetpack-package and --jetpack-addon to runtests.py in a single call

2) run one job and call runtests.py twice, once with --jetpack-package and once with --jetpack-addon

3) run two jobs and call runtests.py only once for each job. passing --jetpack-package for one job and --jetpack-addon for the second

up till now we have pretty much created one job for each suite (mochitest chunk 1, xpcshell, etc). Mozharness can do more than one suite(single runtests.py call) per script run(job) but if we want to do that, we might need to play with some dials: timeout limits, which suite goes first, etc.
Attachment #8476782 - Flags: feedback+

Comment 9

3 years ago
Comment on attachment 8476784 [details] [diff] [review]
buildbot-configs changes

Review of attachment 8476784 [details] [diff] [review]:
-----------------------------------------------------------------

I think this puts the pieces into place but it will not be run anywhere. i.e MOCHITEST_JP is never used

you are going to want to specify which branch(es) you want these suites to run on. Right now we currently use Cedar to test out new or failing jobs. We should probably limit this out to there until we green them up. For that you will need something like: http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config.py#1912

does this make sense?

::: mozilla-tests/config.py
@@ +386,5 @@
> +MOCHITEST_JP = [
> +    ('mochitest-jetpack', {
> +        'use_mozharness': True,
> +        'script_path': 'scripts/desktop_unittest.py',
> +        'extra_args': ['--mochitest-suite', 'jetpack-package,jetpack-addon'],

this will need to change depending on your answer to the question I ask in your buildbot-config patch.
Attachment #8476784 - Flags: feedback+
(Assignee)

Comment 10

3 years ago
(In reply to Jordan Lund (:jlund) from comment #8)
> Comment on attachment 8476782 [details] [diff] [review]
> mozharness changes
> 
> Review of attachment 8476782 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: configs/unittests/linux_unittest.py
> @@ +72,5 @@
> >          "mochitest-devtools-chrome-2": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5", "--total-chunks=3", "--this-chunk=2"],
> >          "mochitest-devtools-chrome-3": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5", "--total-chunks=3", "--this-chunk=3"],
> >          "mochitest-devtools-chrome-chunked": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5"],
> > +        "jetpack-package": ["--jetpack-package"],
> > +        "jetpack-addon": ["--jetpack-addon"],
> 
> to clarify, do we want to:
> 
> 1) run one job and pass --jetpack-package and --jetpack-addon to runtests.py
> in a single call
> 
> 2) run one job and call runtests.py twice, once with --jetpack-package and
> once with --jetpack-addon
> 
> 3) run two jobs and call runtests.py only once for each job. passing
> --jetpack-package for one job and --jetpack-addon for the second
> 
> up till now we have pretty much created one job for each suite (mochitest
> chunk 1, xpcshell, etc). Mozharness can do more than one suite(single
> runtests.py call) per script run(job) but if we want to do that, we might
> need to play with some dials: timeout limits, which suite goes first, etc.

I don't know that this matters much. Right now I think JP tests run as (2). These two tests tend to test the same code so it makes sense to just have them in one box on tbpl. If there is a standard way that makes most sense then let's just do that.

(In reply to Jordan Lund (:jlund) from comment #9)
> Comment on attachment 8476784 [details] [diff] [review]
> buildbot-configs changes
> 
> Review of attachment 8476784 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> I think this puts the pieces into place but it will not be run anywhere. i.e
> MOCHITEST_JP is never used
> 
> you are going to want to specify which branch(es) you want these suites to
> run on. Right now we currently use Cedar to test out new or failing jobs. We
> should probably limit this out to there until we green them up. For that you
> will need something like:
> http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config.
> py#1912
> 
> does this make sense?
> 
> ::: mozilla-tests/config.py
> @@ +386,5 @@
> > +MOCHITEST_JP = [
> > +    ('mochitest-jetpack', {
> > +        'use_mozharness': True,
> > +        'script_path': 'scripts/desktop_unittest.py',
> > +        'extra_args': ['--mochitest-suite', 'jetpack-package,jetpack-addon'],
> 
> this will need to change depending on your answer to the question I ask in
> your buildbot-config patch.

I didn't add this anywhere yet because I didn't really understand how to. The code to land and ship these harnesses in the test package is landed on Gum and I anticipated that there might need to be some tweaking both to the mozharness side and the m-c side to get it working. If it makes more sense to do that on cedar then that works.

Once this all works these should be enabled like any other tests on m-c and related branches and then ride the trains down to release.
(Assignee)

Comment 11

3 years ago
Any progress here?
Flags: needinfo?(jlund)
(In reply to Dave Townsend [:mossop] from comment #11)
> Any progress here?

Oh sorry I thought you just needed feedback on your patch and I missed your follow-up comments. I am pretty busy trying to wrap up some goals so if you would like me to drive this I'll run it by my manager. Hopefully though there isn't much work. You might be very close.

Might just a matter of taking this:
'extra_args': ['--mochitest-suite', 'jetpack-package,jetpack-addon']

And instead doing:
'extra_args': ['--mochitest-suite', 'jetpack-package', --mochitest-suite', 'jetpack-addon']

To accomplish number two mentioned above. And then to a enable on Cedar or gum. I suppose gum makes the most sense. Again in a enabling on a single branch is done via: http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config.py#1912

Feel free to make those two changes and I'll review. Then we can iterate on gum :)
Flags: needinfo?(jlund)
(Assignee)

Updated

3 years ago
Attachment #8476783 - Attachment is obsolete: true
(Assignee)

Comment 13

3 years ago
Created attachment 8486818 [details] [diff] [review]
buildbot-configs changes

Ok here are those changes.
Attachment #8476784 - Attachment is obsolete: true
Attachment #8486818 - Flags: review?(jlund)
(Assignee)

Comment 14

3 years ago
Comment on attachment 8476782 [details] [diff] [review]
mozharness changes

Sounds like this bit is ready to land?
Attachment #8476782 - Flags: review?(jlund)
Comment on attachment 8476782 [details] [diff] [review]
mozharness changes

Review of attachment 8476782 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm. this should be a good start. We may need to tweak mozharness more depending on how this does on gum
Attachment #8476782 - Flags: review?(jlund) → review+
Comment on attachment 8486818 [details] [diff] [review]
buildbot-configs changes

Review of attachment 8486818 [details] [diff] [review]:
-----------------------------------------------------------------

Awesome, this looks good to me.

I think you meant to put gum instead of Cedar (see in-line). But I'm just thinking, I may be wrong about doing this on gum. cedar has the benefit of using mozharness 'default' branch instead of production. Production is used on most branches including gum. So if we need to make future changes, at least we can see the results on Cedar before merging possible broken patches to production.

You mentioned you didn't mind where we test this so how about we leave your patch as is but fix the comment mentioning gum? r+ to land on default any time.

::: mozilla-tests/config.py
@@ +1971,5 @@
>          if slave_platform in BRANCHES['cedar']['platforms'][platform]:
>              BRANCHES['cedar']['platforms'][platform][slave_platform]['opt_unittest_suites'] += MOZBASE[:]
>              BRANCHES['cedar']['platforms'][platform][slave_platform]['debug_unittest_suites'] += MOZBASE[:]
>  
> +# Enable webapprt-chrome tests on gum

gum mentioned here

@@ +1977,5 @@
> +    for slave_platform in PLATFORMS[platform]['slave_platforms']:
> +        if slave_platform not in BRANCHES['gum']['platforms'][platform]:
> +            continue
> +        BRANCHES['cedar']['platforms'][platform][slave_platform]['opt_unittest_suites'] += MOCHITEST_JP[:]
> +        BRANCHES['cedar']['platforms'][platform][slave_platform]['debug_unittest_suites'] += MOCHITEST_JP[:]

cedar is used for the above two lines
Attachment #8486818 - Flags: review?(jlund) → review+
We will also need a tbpl patch so the job shows correctly. Without patch it might show up as some default like M instead of a custom one like MJP. This isn't such a big deal for Cedar but it will need to be done.

I will ping Sheriffs to see if they have docs for this.
(Assignee)

Comment 18

3 years ago
Landed:

https://hg.mozilla.org/build/buildbot-configs/rev/4092544e664c
https://hg.mozilla.org/build/mozharness/rev/4a63895bd8b2
In prod with reconfig on 2014-09-15 08:30 PT
(Assignee)

Updated

3 years ago
Depends on: 1068764
Depends on: 1068765
Assignee: nobody → dtownsend+bugmail
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Summary: Add support for mochitest-jetpack tests → Add support for mochitest-jetpack tests & enable on cedar
(Assignee)

Comment 20

3 years ago
All the appropriate things have landed now. How do I get cedar updated to the latest m-c so we can see how the tests are running?
Flags: needinfo?(jlund)
(In reply to Dave Townsend [:mossop] from comment #20)
> All the appropriate things have landed now. How do I get cedar updated to
> the latest m-c so we can see how the tests are running?

hg clone https://hg.mozilla.org/projects/cedar && cd cedar && hg pull https://hg.mozilla.org/mozilla-central && hg merge && hg commit -m "Merge mozilla-central into cedar" && hg push ssh://hg.mozilla.org/projects/cedar
(Assignee)

Comment 22

3 years ago
(In reply to Ed Morley [:edmorley] from comment #21)
> (In reply to Dave Townsend [:mossop] from comment #20)
> > All the appropriate things have landed now. How do I get cedar updated to
> > the latest m-c so we can see how the tests are running?
> 
> hg clone https://hg.mozilla.org/projects/cedar && cd cedar && hg pull
> https://hg.mozilla.org/mozilla-central && hg merge && hg commit -m "Merge
> mozilla-central into cedar" && hg push ssh://hg.mozilla.org/projects/cedar

Unfortunately there are merge conflicts in the web platform tests that it looks like are being experimented on on cedar right now so I'm not sure I know enough to resolve them.
(In reply to Dave Townsend [:mossop] from comment #22)
> Unfortunately there are merge conflicts in the web platform tests that it
> looks like are being experimented on on cedar right now so I'm not sure I
> know enough to resolve them.

Ah - James I don't suppose you could do an m-c import into Cedar for Dave? :-)
Flags: needinfo?(jlund) → needinfo?(james)
Something like https://hg.mozilla.org/projects/cedar/rev/649768be4000 ?
Flags: needinfo?(james)
(Assignee)

Comment 25

3 years ago
(In reply to James Graham [:jgraham] from comment #24)
> Something like https://hg.mozilla.org/projects/cedar/rev/649768be4000 ?

Wonderful, thanks
Blocks: 1117922
You need to log in before you can comment on or make changes to this bug.