557921 - Disable running opt/debug packaged unittests on build machines for mozilla-central macosx machines

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

Now that the unit tests are running on talos machines it has come the time to disabled them.

At first, disabling only mozilla-central since that's the branch we are currently running concurrently the unit tests on talos slaves and builds slaves.

Comment 2

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: turns off opt/debug packaged unittests for staging & production

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 438181 [details] [diff] [review]
turns off opt/debug packaged unittests for staging & production

http://hg.mozilla.org/build/buildbot-configs/rev/910472137a45

Comment 4

[Armen [:armenzg]]

Comment on attachment 438181 [details] [diff] [review]
turns off opt/debug packaged unittests for staging & production

I had to back this out so we could have unit tests again.
Please refer to bug 558501.

Comment 5

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: turns off opt/debug packaged unittests for staging & production

this time enabling packageTests for the debug builds so we don't lose them

Comment 6

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: turns off opt/debug packaged unittests for staging & production macosx m-c

tested in staging.  amazing the difference a "d" can make between packageTests and packagedTests.

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: turns off opt/debug packaged unittests for staging & production macosx m-c

without my staging stuff.

Comment 8

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 438626 [details] [diff] [review]
turns off opt/debug packaged unittests for staging & production macosx m-c

http://hg.mozilla.org/build/buildbot-configs/rev/1793c27af2f9

reconfig'd pm01.

Comment 9

[Lukas Blakk [:lsblakk] use ?needinfo]

disabled Leopard unittests, they are now running on talos only - closing.

Comment 10

[Aki Sasaki (not active)]

Comment on attachment 438626 [details] [diff] [review]
turns off opt/debug packaged unittests for staging & production macosx m-c

http://hg.mozilla.org/build/buildbot-configs/rev/51ab063381b5

Comment 11

[Phil Ringnalda (:philor)]

Sorry, I didn't realize I should have pointed out that when this was in last Friday, it was causing a bunch of test failures, in opt mochitest 2 (test_bug427744.html | Should have seen the test plugin), 4 (test_flush_on_paint.html | [SimpleTest/SimpleTest.js, window.onerror] An error occurred - plugin.setColor is not a function plus several others), 5 (test_bug532208.html | Data delivery crashed plus several others), other (10 mochitest-chrome, 9 mochitest-browser-chrome, 20 mochitest-ipcplugins), and in opt xpcshell (test_csputils.js, et al.), but since they are mostly plugin-related, and there was a plugin-related push around the time this landed last Friday, we didn't realize that this was at fault, and when the errors went away when this was backed out again, we just thought a random clobber had cured what ailed us.

Comment 12

[Aki Sasaki (not active)]

s: talos-r3-leopard-040
3075 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testElementReplace - 6 assertions, 1 failures, 0 errors
3198 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testGetValue - 3 assertions, 1 failures, 0 errors
3199 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/prototype/test_Prototype.html | testSetValue - 12 assertions, 1 failures, 0 errors
4808 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/bugs/test_bug427744.html | Should have seen the test plugin

s: talos-r3-leopard-028
732 ERROR TEST-UNEXPECTED-FAIL | /tests/layout/base/tests/test_flush_on_paint.html | [SimpleTest/SimpleTest.js, window.onerror] An error occurred - plugin.setColor is not a function at http://mochi.test:8888/tests/layout/base/tests/test_flush_on_paint.html:27
2040 ERROR TEST-UNEXPECTED-FAIL | /tests/layout/generic/test/test_plugin_focus.html | Check event target - got "p1", expected "p2"
2043 ERROR TEST-UNEXPECTED-FAIL | /tests/layout/generic/test/test_plugin_mouse_coords.html | [SimpleTest/SimpleTest.js, window.onerror] An error occurred - p1.getLastMouseX is not a function at http://mochi.test:8888/tests/layout/generic/test/test_plugin_mouse_coords.html:33

s: talos-r3-leopard-006
6 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_bug532208.html | Data delivery crashed
9 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_cookies.html | [SimpleTest/SimpleTest.js, window.onerror] An error occurred - pluginElement.setCookie is not a function at http://mochi.test:8888/tests/modules/plugin/test/test_cookies.html:14
18 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_fullpage.html | Test timed out.
21 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_getauthenticationinfo.html | [SimpleTest/SimpleTest.js, window.onerror] An error occurred - plugin.getAuthInfo is not a function at http://mochi.test:8888/tests/modules/plugin/test/test_getauthenticationinfo.html:27
27 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_instantiation.html | Dynamic plugin instantiation.
30 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_multipleinstanceobjects.html | [SimpleTest/SimpleTest.js, window.onerror] An error occurred - p1.getObjectValue is not a function at http://mochi.test:8888/tests/modules/plugin/test/test_multipleinstanceobjects.html:17
33 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_newstreamondestroy.html | [SimpleTest/SimpleTest.js, window.onerror] An error occurred - p.callOnDestroy is not a function at http://mochi.test:8888/tests/modules/plugin/test/test_newstreamondestroy.html:20
36 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_npn_asynccall.html | [SimpleTest/SimpleTest.js, window.onerror] An error occurred - plugin.asyncCallbackTest is not a function at http://mochi.test:8888/tests/modules/plugin/test/test_npn_asynccall.html:28
39 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_npn_timers.html | [SimpleTest/SimpleTest.js, window.onerror] An error occurred - plugin.timerTest is not a function at http://mochi.test:8888/tests/modules/plugin/test/test_npn_timers.html:28
lastScript42 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_npobject_getters.html | plugin1.pluginFoundElement (NPNVPluginElementNPObject)
43 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_npobject_getters.html | window.pluginFoundWindow (NPNVWindowNPObject)
46 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_npruntime_npnevaluate.html | [SimpleTest/SimpleTest.js, window.onerror] An error occurred - plugin.npnEvaluateTest is not a function at http://mochi.test:8888/tests/modules/plugin/test/test_npruntime_npnevaluate.html:62
49 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_npruntime_npninvoke.html | [SimpleTest/SimpleTest.js, window.onerror] An error occurred - plugin.npnInvokeTest is not a function at http://mochi.test:8888/tests/modules/plugin/test/test_npruntime_npninvoke.html:126
52 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_npruntime_npninvokedefault.html | [SimpleTest/SimpleTest.js, window.onerror] An error occurred - plugin.npnInvokeDefaultTest is not a function at http://mochi.test:8888/tests/modules/plugin/test/test_npruntime_npninvokedefault.html:104
55 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_painting.html | [SimpleTest/SimpleTest.js, window.onerror] An error occurred - paint_waiter.getPaintCount is not a function at http://mochi.test:8888/tests/modules/plugin/test/test_painting.html:67
58 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_pluginstream_asfile.html | Test timed out.
61 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_pluginstream_asfileonly.html | Test timed out.
64 ERROR TEST-UNEXPECTED-FAIL | /tests/modules/plugin/test/test_pluginstream_err.html | Test timed out.
65 ERROR TEST-UNEXPECTED-FAIL | (SimpleTest/TestRunner.js) | 4 test timeouts, giving up.
66 ERROR TEST-UNEXPECTED-FAIL | (SimpleTest/TestRunner.js) | Skipping 174 remaining tests.

Comment 13

[Aki Sasaki (not active)]

[19:07]	<dolske> hmm. so, this is interesting. The CSP test fails while doing something with http://localhost:9000 (it makes an XMLHttpRequest and something goes awry)
	the next test that fails is test_error_codes.js, which is doing stuff with http://localhost:4444/test_error_code.xml
[19:08]	it's expecting a connection refused, but instead gets a generic failure.
	maybe an OS firewall issue?
<aki|buildduty>	that would be a good thing to investigate

Comment 14

[Aki Sasaki (not active)]

And then mochitest-plain-2 just went green on talos-r3-leopard-017 with unit tests re-enabled.

So:

* firewall
* turning off the tests has some side effect we're unaware of
* maybe because this build was a depend build? Maybe we should really look at how the nightly goes, and test a nightly in staging against staging leopard unittests-on-talos?
* something else?

Comment 15

[Justin Dolske [:Dolske]]

Looking through http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1271201842.1271202755.6954.gz&fulltext=1 (Mo2 failure on OSX)...

test_bug427744.html fails because the test plugin doesn't seem to be preset. Sounds like bug 552365, but that was fixed.

Comment 16

[Justin Dolske [:Dolske]]

The log shows Test.plugin being extracted, and aki sees it on the box, so it's odd it wasn't loaded.

test_flush_on_paint.htm (in Mo4) is also using the test plugin, and in fact the rest of those errors all seem to be related to the test plugin. 

So, it looks like there are 2 basic problems happening:

1) Something networkish, (comment 13, bug 559246)
2) Something with the test plugin

All the test failures seem to be one or the other.

Comment 17

[Armen [:armenzg]]

Lukas I have found why this stops working:
http://hg.mozilla.org/build/buildbotcustom/file/tip/misc.py#l603

elif pf.get('enable_opt_unittests'):
    packageTests = True

# Allow for test packages on platforms that can't be tested
# on the same master.
packageTests = pf.get('packageTests', packageTests)

Everytime we switch enable_opt_unittest to False we stop packaging the tests and therefore we stop uploading it and ya ra ya ra.

Have a look at Linux64:
http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla2/config.py#l148

Feel free to send me the review.

Comment 18

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Disable Leopard mozilla-central unittest opt/debug packaged builders

doing macosx/macosx64 as separate patches this time.  this is for staging and production.

Comment 19

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Disable Snow Leopard mozilla-central unittest opt/debug packaged builders

Comment 20

[Armen [:armenzg]]

Comment on attachment 439121 [details] [diff] [review]
Disable Snow Leopard mozilla-central unittest opt/debug packaged builders

10.6 has already disabled opt_unittests across the board (all branches):
http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla2/config.py#l216

I am working on it on bug 559182.

Comment 21

[Armen [:armenzg]]

Comment on attachment 439118 [details] [diff] [review]
Disable Leopard mozilla-central unittest opt/debug packaged builders

:)

Comment 22

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to comment #20)
> (From update of attachment 439121 [details] [diff] [review])
> 10.6 has already disabled opt_unittests across the board (all branches):
> http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla2/config.py#l216
> 
> I am working on it on bug 559182.

This patch is specific to mozilla-central because that's where we are planning to turn this on at first.  You setting global configs for macosx64 is separate and I think this patch should go in as is, with adjustments made later when it's for ALL branches.

Comment 23

[Armen [:armenzg]]

(In reply to comment #22)
> (In reply to comment #20)
> > (From update of attachment 439121 [details] [diff] [review] [details])
> > 10.6 has already disabled opt_unittests across the board (all branches):
> > http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla2/config.py#l216
> > 
> > I am working on it on bug 559182.
> 
> This patch is specific to mozilla-central because that's where we are planning
> to turn this on at first.  You setting global configs for macosx64 is separate
> and I think this patch should go in as is, with adjustments made later when
> it's for ALL branches.

As we open more branches for 10.6 (currently running builds only on m-c) they will pick up the default behaviour which is to:
- not enable opt_unittests or the old unittests
- generate packaged tests

The fix is already landed on bug 559182. Let me know if this does make sense.

Comment 24

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 439118 [details] [diff] [review]
Disable Leopard mozilla-central unittest opt/debug packaged builders

http://hg.mozilla.org/build/buildbot-configs/rev/9e92a1842eed

Comment 25

[Armen [:armenzg]]

Looking good!