523712 - Need mochitest-ipcplugins tests run on electrolysis tinderbox

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

There's a new make target in the e10s repo, mochitest-ipcpugins, which runs the plugin mochitests with out-of-process plugins enabled.  This needs to be added to the e10s tinderbox runs at http://tinderbox.mozilla.org/showbuilds.cgi?tree=Electrolysis.

Comment 1

[Benjamin Smedberg]

These are running packaged tests. The packaged tests don't use any makefile targets... instead, they call runtests.py directly. See http://hg.mozilla.org/build/buildbotcustom/file/4c0741a94bbe/steps/unittest.py#l498 for the buildbot step, and http://hg.mozilla.org/build/buildbotcustom/file/4c0741a94bbe/process/factory.py#l4256 for the factory that runs the various flavors of mochitest. 

The list of flavors is specified, I think, at http://hg.mozilla.org/build/buildbot-configs/file/c230f470c116/mozilla2/config.py#l64 which is not a per-branch config.

With a little guidance I'm happy to write the patch to these various pieces; I don't know how releng wants to deal with having a "new" testsuite that only applies to certain branches: currently electrolysis, and after a bit mozilla-central and its derivatives (places and tracemonkey).

Comment 2

[Benjamin Smedberg]

Forgot to cc the folks who can provide the guidance.

Comment 3

[[e-mail] (:bhearsum)]

(In reply to comment #1)
> These are running packaged tests. The packaged tests don't use any makefile
> targets... instead, they call runtests.py directly. See
> http://hg.mozilla.org/build/buildbotcustom/file/4c0741a94bbe/steps/unittest.py#l498
> for the buildbot step, and
> http://hg.mozilla.org/build/buildbotcustom/file/4c0741a94bbe/process/factory.py#l4256
> for the factory that runs the various flavors of mochitest. 
> 
> The list of flavors is specified, I think, at
> http://hg.mozilla.org/build/buildbot-configs/file/c230f470c116/mozilla2/config.py#l64
> which is not a per-branch config.

That dict is actually a set of defaults. You can override any of those per branch. mozilla-central does this for unittest_suites, even. So, you can override this for e10s, like m-c does here: http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla2/config.py#l161.

I think this is all that needs to be done.

Comment 4

[Jonathan Griffin (:jgriffin)]

Attachment: turn on mochitest-ipcplugins tests on e10s

Cool...is this the right patch to enable this on e10s tinderbox?

Comment 5

[[e-mail] (:bhearsum)]

(In reply to comment #4)
> Created an attachment (id=407804) [details]
> turn on mochitest-ipcplugins tests on e10s
> 
> Cool...is this the right patch to enable this on e10s tinderbox?

I think so. I'll test it in staging.

Comment 6

[Jonathan Griffin (:jgriffin)]

Attachment: patch v2

move mochitest-ipcplugins from mochitest to everything-else, per discussion with bhearsum

Comment 7

[[e-mail] (:bhearsum)]

Hit this problem with your patch:
Usage: Usage instructions for runtests.py.
All arguments are optional.
If --chrome is specified, chrome tests will be run instead of web content tests.
If --browser-chrome is specified, browser-chrome tests will be run instead of web content tests.
See <http://mochikit.com/doc/html/MochiKit/Logging.html> for details on the logging levels.

runtests.py: error: no such option: --ipcplugins

Sounds like you need to patch runtests.py, too.

Comment 8

[Jonathan Griffin (:jgriffin)]

Hmm, I guess I don't understand how this list works then.  Where did --ipcplugins come from?

I thought this config list was working off make targets, of which mochitest-ipcplugins is one, like mochitest-plain, etc.  There is no --ipcplugins option to runtests.py.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

mochitest-ipcplugins doesn't work like the other mochitest targets, see
http://hg.mozilla.org/projects/electrolysis/file/tip/testing/testsuite-targets.mk#l83
We probably have to hack something in at 
http://hg.mozilla.org/build/buildbotcustom/file/4c0741a94bbe/steps/unittest.py#l510

Comment 10

[Jonathan Griffin (:jgriffin)]

Attachment: unittest.py patch

It seems like something like this would work, as long as that --test-path is still valid in packaged tests

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Doing some builds in staging with attachments 407808 and 407887.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 407887 [details] [diff] [review]
unittest.py patch

This looks fine to me. 

In staging, mac succeeded (177/0/0), linux was busted because of bug 515436, and windows crashed. I'll upload the load with the stack in it shortly.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: mochitest-ipcplugins log for win32

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 407808 [details] [diff] [review]
patch v2

>diff --git a/mozilla2/config.py b/mozilla2/config.py
>+BRANCHES['electrolysis']['unittest_suites'] = [
>+    # Turn on mochitest-ipcplugins tests
>+    ('mochitests', ['mochitest-plain']),

When we start running mochitest-plain in 5 chunks on more than mozilla-central there's a chance we'll forget to update this. Maintenance-wise I think it would be better to add mochitest-ipcplugins to what comes from the global defaults, something like
 BRANCHES['electrolysis']['unittest_suites'][1][1].append('mochitest-ipcplugins')
only without assuming that everythingelse will always be the second tuple.

Comment 15

[Jonathan Griffin (:jgriffin)]

Attachment: config.py patch

How's this look?

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 408129 [details] [diff] [review]
config.py patch

WFM.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Thanks for the patches. I'll get his landed first thing next week.

Comment 18

[[e-mail] (:bhearsum)]

I'm not going to have a chance to try out the latest patches soon. Some urgent things have come up.

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 407887 [details] [diff] [review]
unittest.py patch

http://hg.mozilla.org/build/buildbotcustom/rev/2dbf5ffcb0f8

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 408129 [details] [diff] [review]
config.py patch

I added the matching change to mozilla2-staging/config.py on checkin.

http://hg.mozilla.org/build/buildbot-configs/rev/8875a802d026

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

pm and pm02 reconfig'd to enable mochitest-ipcplugins on
* Linux electrolysis test everythingelse
* OS X 10.5.2 electrolysis test everythingelse
* WINNT 5.2 electrolysis test everythingelse
on the Electrolysis tree. Triggered some builds too.