Open Bug 688052 Opened 9 years ago Updated 7 years ago

Perma-orange Android TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityEventUtils.html | Test timed out. (followed by >100 more failures)

Categories

(Testing :: Mochitest, defect)

Product:
Testing
Component:
Mochitest
Platform:
ARM
Android
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

People

(Reporter: mbrubeck, Unassigned)

References

Details

(Whiteboard: [mobile_unittests][test disabled on Android])

Attachments

(1 file, 1 obsolete file)

Disable 3 broken Android tests.
1.02 KB, patch
mbrubeck
: review+
Details | Diff | Splinter Review 
https://tbpl.mozilla.org/php/getParsedLog.php?id=6474086&tree=Mozilla-Inbound
and
https://tbpl.mozilla.org/php/getParsedLog.php?id=6472144&tree=Mozilla-Inbound
Android Tegra 250 mozilla-inbound opt test mochitest-7 on 2011-09-20 13:33:19 PDT for push 1e41259daf67

35 INFO TEST-END | /tests/Harness_sanity/test_sanity.html | finished in 342ms
36 INFO TEST-START | /tests/Harness_sanity/test_sanityEventUtils.html
37 INFO TEST-INFO | /tests/Harness_sanity/test_sanityEventUtils.html | 
Profile::EventUtilsLoadTime: 18
38 ERROR TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityEventUtils.html | Test timed out.
39 INFO TEST-END | /tests/Harness_sanity/test_sanityEventUtils.html | finished in 325158ms
40 INFO TEST-START | /tests/Harness_sanity/test_sanityException.html
41 INFO TEST-PASS | /tests/Harness_sanity/test_sanityException.html | a call to ok
42 INFO TEST-PASS | /tests/Harness_sanity/test_sanityException.html | [SimpleTest/SimpleTest.js, window.onerror] - An error occurred: uncaught exception: an uncaught exception at :0
43 INFO TEST-END | /tests/Harness_sanity/test_sanityException.html | finished in 137ms
44 INFO TEST-START | /tests/Harness_sanity/test_sanityException2.html
45 INFO TEST-PASS | /tests/Harness_sanity/test_sanityException2.html | a call to ok
46 INFO TEST-PASS | /tests/Harness_sanity/test_sanityException2.html | [SimpleTest/SimpleTest.js, window.onerror] - An error occurred: uncaught exception: an uncaught exception at :0
47 INFO TEST-END | /tests/Harness_sanity/test_sanityException2.html | finished in 85ms
48 INFO TEST-START | /tests/Harness_sanity/test_sanityPluginUtils.html
49 INFO TEST-INFO | /tests/Harness_sanity/test_sanityPluginUtils.html | 
Profile::PluginUtilsLoadTime: 19

50 INFO TEST-KNOWN-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Need a test plugin on this platform
51 ERROR TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Test plugin should be found
52 INFO TEST-KNOWN-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Need a test plugin on this platform
53 ERROR TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Test plugin should be found
54 INFO TEST-KNOWN-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Need a test plugin on this platform
55 ERROR TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Test plugin should be found
56 INFO TEST-KNOWN-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Need a test plugin on this platform
57 ERROR TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Test plugin should be found
58 INFO TEST-KNOWN-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Need a test plugin on this platform
59 ERROR TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Test plugin should be found
60 INFO TEST-KNOWN-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Need a test plugin on this platform
61 ERROR TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Test plugin should be found
62 INFO TEST-KNOWN-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Need a test plugin on this platform
63 ERROR TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Test plugin should be found
64 INFO TEST-KNOWN-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Need a test plugin on this platform
65 ERROR TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Test plugin should be found
66 INFO TEST-KNOWN-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Need a test plugin on this platform
67 ERROR TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Test plugin should be found
68 INFO TEST-KNOWN-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Need a test plugin on this platform

[... continues for ~200 lines ...]

247 ERROR TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Test plugin should be found
248 INFO TEST-KNOWN-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Need a test plugin on this platform
249 ERROR TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityPluginUtils.html | Test plugin should be found
250 INFO TEST-INFO | /tests/Harness_sanity/test_sanityPluginUtils.html | 
Profile::PluginUtilsRunTime: 74

251 INFO TEST-END | /tests/Harness_sanity/test_sanityPluginUtils.html | finished in 220ms
252 INFO TEST-START | /tests/Harness_sanity/test_sanitySimpletest.html
253 INFO TEST-INFO | /tests/Harness_sanity/test_sanitySimpletest.html | Profile::SimpleTestLoadTime: 29
254 ERROR TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanitySimpletest.html | Test timed out.
255 INFO TEST-END | /tests/Harness_sanity/test_sanitySimpletest.html | finished in 329535ms
This is looking perma-orange on mozilla-inbound.  The first occurence of this that I can find is here:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=ab0a6fdf0144
https://tbpl.mozilla.org/php/getParsedLog.php?id=6454938&tree=Mozilla-Inbound

though the tests were not really running before that for other reasons, so it's hard to know exactly when it started.
Summary: Intermittent Android TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityEventUtils.html | Test timed out. (followed by >100 more failures) → Perma-orange Android TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityEventUtils.html | Test timed out. (followed by >100 more failures)
Here's a regression range between the last green changeset and first instance of this bug on mozilla-inbound:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=9ae43404500d&tochange=ab0a6fdf0144

It might be possible to narrow the range by looking at results from mozilla-central, and/or retriggering the tests a bunch of times.
Oh, it looks like these tests are new from bug 677626 and have never been green on Android.  We should disable them on Android and then fix them.
Blocks: 677626
Attached patch Disable test (obsolete) — — Splinter Review 
Requesting review since I didn't know whether your IRC comment counted as rs+, plus not quite sure if commenting out in the Makefile is what you wanted or else using todo_is()? Also not sure if test_sanityPluginUtils.html needed disabling at the same time, given comment 0.
Comment on attachment 561369 [details] [diff] [review]
Disable test

We should disable this on Android only (#ifndef ANDROID).  I'm also not sure exactly which tests we need to disable; we could push some different options to Try to figure it out.  I'm going to sleep soon, but I'll try this in the morning if no one else gets to it first.
Attachment #561369 - Flags: review-
(In reply to Matt Brubeck (:mbrubeck) from comment #7)
> We should disable this on Android only (#ifndef ANDROID).  I'm also not sure
> exactly which tests we need to disable; we could push some different options
> to Try to figure it out.  I'm going to sleep soon, but I'll try this in the
> morning if no one else gets to it first.

Whoops that was a bit retarded on my part; was somewhat hurrying to get to sleep :-)
This time with |ifneq ($(OS_TARGET),Android)|, sent to try to see what else needs disabling:
https://tbpl.mozilla.org/?tree=Try&usebuildbot=1&rev=6e1ff2427a04
(In reply to Ed Morley [:edmorley] from comment #6)
> Created attachment 561369 [details] [diff] [review]
> Disable test
> 
> Requesting review since I didn't know whether your IRC comment counted as
> rs+, plus not quite sure if commenting out in the Makefile is what you
> wanted or else using todo_is()? Also not sure if test_sanityPluginUtils.html
> needed disabling at the same time, given comment 0.

This is the right thing to do, but as Matt mentioned in comment 7 these tests should only be disabled for Android, so you can #ifdef them in the makefile.  Are you planning to get to this today?  Let me know if you need help, happy to jump in.  We can't let Android tests sit perma-orange for a long time.
(In reply to Clint Talbert ( :ctalbert ) from comment #16)
> This is the right thing to do, but as Matt mentioned in comment 7 these
> tests should only be disabled for Android, so you can #ifdef them in the
> makefile.  Are you planning to get to this today?  Let me know if you need
> help, happy to jump in.  We can't let Android tests sit perma-orange for a
> long time.

I pushed a try run with updated patch (see comment 12, somewhat lost in the TBPL robot replies), to work out which tests needed disabling in addition to test_sanityEventUtils.html, but results haven't come back yet (presume due to tree closure). Feel free to take over if you have a better idea of which to disable, I only started doing it after mbrubeck asked if I wanted to last night - but in retrospect maybe I should have declined, since I haven't touched Mochitests before, so I'm not going to be the quickest person to do this :-)
That sounds good.  Let us know what the try server comes back with. The code changes are simple, we can help walk you through that.
Next in the pending queue (with a wait time of 9:25:28 woot!)
https://tbpl.mozilla.org/?tree=Try&usebuildbot=1&rev=6e1ff2427a04
Thanks for your help, Ed! 

Looking at the errors and the code, we'll need to kick out test_sanitySimpletest.html and test_sanityPluginUtils.html from Android tests as well. test_sanitySimpletest.html and test_sanityEventUtils.html both make calls to netscape.security.PrivilegeManager.enablePrivilege("UniversalXPConnect"), instead of using SpecialPowers, which ends up breaking in Android. test_sanityPluginUtils.html won't work in Android since plugins aren't supported.

I just kicked off a try run with these tests removed for Android:

https://tbpl.mozilla.org/?tree=Try&usebuildbot=1&rev=2a76d3dffe11

I'll try to migrate the calls to SpecialPowers as soon as possible so we can re-enable the non-plugin tests.
Attached patch Disable 3 broken Android tests. — — Splinter Review 
Here's the patch being run
Comment on attachment 561608 [details] [diff] [review]
Disable 3 broken Android tests.

(In reply to Malini Das from comment #23)
> I just kicked off a try run with these tests removed for Android:
> 
> https://tbpl.mozilla.org/?tree=Try&usebuildbot=1&rev=2a76d3dffe11

Passed try. 

M7 isn't showing on TBPL for some reason, but you can see here:
https://build.mozilla.org/buildapi/self-serve/try/rev/2a76d3dffe11
Attachment #561608 - Flags: review?(mbrubeck)
Attachment #561608 - Flags: review?(mbrubeck) → review+
Unfortunately, I don't have permissions to submit patches to mozilla-central/mozilla-inbound.

Is it possible for one of you to land this patch?
Attachment #561369 - Attachment is obsolete: true
Comment on attachment 561608 [details] [diff] [review]
Disable 3 broken Android tests.

Thanks, pushed to inbound:
https://hg.mozilla.org/integration/mozilla-inbound/rev/376187bc07dc
Whiteboard: [orange][mobile_unittests] → [orange][mobile_unittests] [on merging, leave open for remaining work]
Whiteboard: [orange][mobile_unittests] [on merging, leave open for remaining work] → [orange][mobile_unittests][test disabled on Android]
Whiteboard: [orange][mobile_unittests][test disabled on Android] → [mobile_unittests][test disabled on Android]
OS: Android → All
Hardware: ARM → All
Summary: Perma-orange Android TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityEventUtils.html | Test timed out. (followed by >100 more failures) → Intermittent TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityEventUtils.html | Test timed out. (followed by >100 more failures)
Since this bug was very much about these tests being permaorange on Android when they were introduced and disabling them on Android at that time, in 2011, if someone has introduced a new Mac regression in them, please get a new bug for it.
OS: All → Android
Hardware: All → ARM
Summary: Intermittent TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityEventUtils.html | Test timed out. (followed by >100 more failures) → Perma-orange Android TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanityEventUtils.html | Test timed out. (followed by >100 more failures)
You need to log in before you can comment on or make changes to this bug.