Closed Bug 965358 Opened 9 years ago Closed 9 years ago

Intermittent testHomeBanner | banner text - correct text appeared in the home banner

Categories

(Firefox for Android Graveyard :: General, defect)

Product:
Firefox for Android Graveyard
Component:
General
Platform:
ARM
Android
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox27 unaffected, firefox28 unaffected, firefox29 fixed, firefox-esr24 unaffected)

Status:
RESOLVED FIXED
Milestone:
Firefox 29
Tracking Flags:
Tracking Status
firefox27 --- unaffected
firefox28 --- unaffected
firefox29 --- fixed
firefox-esr24 --- unaffected

People

(Reporter: emorley, Assigned: Margaret)

References

Details

(Keywords: intermittent-failure)

Attachments

(2 files)

disable snippets for mochitests
1.35 KB, patch
gbrown
: review+
Details | Diff | Splinter Review
disable snippets in default testing profile
873 bytes, patch
gbrown
: review+
Details | Diff | Splinter Review 
Android 4.0 Panda mozilla-inbound opt test robocop-2 on 2014-01-29 08:03:54 PST for push 523a1736de8f

slave: panda-0805

https://tbpl.mozilla.org/php/getParsedLog.php?id=33747048&tree=Mozilla-Inbound

{
08:25:11     INFO -  1 INFO TEST-START | testHomeBanner
08:25:11     INFO -  2 INFO TEST-PASS | testHomeBanner | Given message occurred for registered event: {"type":"Gecko:Ready"} - Gecko:Ready should equal Gecko:Ready
08:25:11     INFO -  EventExpecter: no longer listening for Gecko:Ready
08:25:11     INFO -  3 INFO TEST-PASS | testHomeBanner | waiting for urlbar text to gain focus - urlbar text gained focus
08:25:11     INFO -  4 INFO TEST-PASS | testHomeBanner | URL typed properly - chrome://roboextender/content/robocop_home_banner.html#addMessage should equal chrome://roboextender/content/robocop_home_banner.html#addMessage
08:25:11     INFO -  5 INFO TEST-PASS | testHomeBanner | Given message occurred for registered event: {"errorType":"","bgColor":"transparent","type":"DOMContentLoaded","tabID":0} - DOMContentLoaded should equal DOMContentLoaded
08:25:11     INFO -  EventExpecter: no longer listening for DOMContentLoaded
08:25:11     INFO -  6 INFO TEST-PASS | testHomeBanner | Given message occurred for registered event: {"type":"TestHomeBanner:MessageAdded"} - TestHomeBanner:MessageAdded should equal TestHomeBanner:MessageAdded
08:25:11     INFO -  7 INFO TEST-PASS | testHomeBanner | Given message occurred for registered event: {"type":"TestHomeBanner:MessageShown"} - TestHomeBanner:MessageShown should equal TestHomeBanner:MessageShown
08:25:11     INFO -  8 INFO TEST-PASS | testHomeBanner | waiting for urlbar text to gain focus - urlbar text gained focus
08:25:11     INFO -  9 INFO TEST-PASS | testHomeBanner | URL typed properly - about:home should equal about:home
08:25:11     INFO -  10 INFO TEST-PASS | testHomeBanner | Given message occurred for registered event: {"errorType":"","bgColor":"transparent","type":"DOMContentLoaded","tabID":0} - DOMContentLoaded should equal DOMContentLoaded
08:25:11     INFO -  EventExpecter: no longer listening for DOMContentLoaded
08:25:11     INFO -  waitForText timeout on The quick brown fox jumps over the lazy dog.
08:25:11     INFO -  11 INFO TEST-UNEXPECTED-FAIL | testHomeBanner | banner text - correct text appeared in the home banner
08:25:11     INFO -  Exception caught during test!
08:25:11     INFO -  junit.framework.AssertionFailedError: 11 INFO TEST-UNEXPECTED-FAIL | testHomeBanner | banner text - correct text appeared in the home banner
08:25:11     INFO -  	at junit.framework.Assert.fail(Assert.java:47)
08:25:11     INFO -  	at org.mozilla.gecko.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:110)
08:25:11     INFO -  	at org.mozilla.gecko.FennecMochitestAssert.ok(FennecMochitestAssert.java:139)
08:25:11     INFO -  	at org.mozilla.gecko.tests.testHomeBanner.testHomeBanner(testHomeBanner.java:30)
08:25:11     INFO -  	at java.lang.reflect.Method.invokeNative(Native Method)
08:25:11     INFO -  	at java.lang.reflect.Method.invoke(Method.java:511)
08:25:11     INFO -  	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
08:25:11     INFO -  	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
08:25:11     INFO -  	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
08:25:11     INFO -  	at org.mozilla.gecko.tests.BaseTest.runTest(BaseTest.java:150)
08:25:11     INFO -  	at junit.framework.TestCase.runBare(TestCase.java:127)
08:25:11     INFO -  	at junit.framework.TestResult$1.protect(TestResult.java:106)
08:25:11     INFO -  	at junit.framework.TestResult.runProtected(TestResult.java:124)
08:25:11     INFO -  	at junit.framework.TestResult.run(TestResult.java:109)
08:25:11     INFO -  	at junit.framework.TestCase.run(TestCase.java:118)
08:25:11     INFO -  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
08:25:11     INFO -  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
08:25:11     INFO -  	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:545)
08:25:11     INFO -  	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1551)
08:25:11     INFO -  12 INFO TEST-UNEXPECTED-FAIL | testHomeBanner | Exception caught - junit.framework.AssertionFailedError: 11 INFO TEST-UNEXPECTED-FAIL | testHomeBanner | banner text - correct text appeared in the home banner
08:25:11     INFO -  13 INFO TEST-END | testHomeBanner | finished in 32629ms
}
I suspect this may be caused by the fact that we recently pushed some test snippets to production, since I'm seeing snippets appear when running tests locally right now.

If that is the case, I'm not sure the best way to fix this. I don't want to totally disable snippets for testing, because we did find some performance regressions related to them. However, maybe we should change the update URL pref to point to some local test server rather than the snippets server.

I wonder if this is only intermittently green because the snippets fail to update sometimes.
Assignee: nobody → margaret.leibovic
I think we should try to just disable snippets for mochitests. I ran this locally, and it looks like it fixed the problem.
Attachment #8367741 - Flags: review?(gbrown)
My first try push accidentally included a patch that broke other things. Here's a clean try push:
https://tbpl.mozilla.org/?tree=Try&rev=b74251a1a7f4

I really think that the snippets must have caused this. I also think it would be good to disable them for mochitests in general, so that we avoid making network requests during test runs.
Comment on attachment 8367741 [details] [diff] [review]
disable snippets for mochitests

Review of attachment 8367741 [details] [diff] [review]:
-----------------------------------------------------------------

Disabling snippets definitely seems like a good idea.

This change only disables them for Robocop tests, while your comment says "for mochitests", and it might be best to disable for all test suites. I think you want to update testing/profiles/prefs_general.js instead...but I'm not 100% sure that will be picked up by Robocop.
Attachment #8367741 - Flags: review?(gbrown) → review+
(In reply to Geoff Brown [:gbrown] from comment #33)
> Comment on attachment 8367741 [details] [diff] [review]
> disable snippets for mochitests
> 
> Review of attachment 8367741 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Disabling snippets definitely seems like a good idea.
> 
> This change only disables them for Robocop tests, while your comment says
> "for mochitests", and it might be best to disable for all test suites. I
> think you want to update testing/profiles/prefs_general.js instead...but I'm
> not 100% sure that will be picked up by Robocop.

Oh, I misinterpreted what I was doing then :)

I can also make a try run flipping the pref in prefs_general.js.
Try run looking green so I pushed to fx-team:
https://hg.mozilla.org/integration/fx-team/rev/335c1799e1c0

Here's a try push that disables snippets in prefs_general.js:
https://tbpl.mozilla.org/?tree=Try&rev=145a0754c23c
Here's the patch I pushed to try.

Although I now realize I should probably do a try push that runs more than just robocop tests, since this affects more than that.
Attachment #8368062 - Flags: review?(gbrown)
Try run that runs all unit tests:
https://tbpl.mozilla.org/?tree=Try&rev=5be485ff07fd
Attachment #8368062 - Flags: review?(gbrown) → review+
https://hg.mozilla.org/mozilla-central/rev/335c1799e1c0
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 29
Try run was green, so I pushed the patch to disable snippets more globally:
https://hg.mozilla.org/integration/fx-team/rev/cbcf7d2e9cb3
Depends on: 1036028
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.