Closed Bug 862493 Opened 9 years ago Closed 9 years ago

Disable 7 rc tests that cause instability on Panda

Categories

(Firefox for Android Graveyard :: General, defect)

x86
Android
defect
Not set
normal

Tracking

(firefox22 fixed)

RESOLVED FIXED
Firefox 23
Tracking Status
firefox22 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

Attachments

(1 file)

See https://bugzilla.mozilla.org/show_bug.cgi?id=855406#c3.

This is a proposal to disable 7 robocop tests so that Panda rc will pass consistently, allowing us to un-hide Android 4.0 rc.

The tests are:

-[testAxisLocking]
-[testAboutPage]
-[testWebContentContextMenu]
-[testFormHistory]
-[testDoorHanger]
-[testClearPrivateData]
-[testSystemPages]

The obvious trade-off is that we won't be running these tests at all; but we will be running the remaining tests on both tegra and panda (Android 2.2 and 4.0).

Of course we intend to follow-up with investigations to re-enable the tests, but that doesn't look easy!
Blocks: 855406
Comment on attachment 738123 [details] [diff] [review]
disable 7 tests that cause panda instability

Review of attachment 738123 [details] [diff] [review]:
-----------------------------------------------------------------

awesome, lets get these green and reliable!
Attachment #738123 - Flags: review?(jmaher) → review+
https://hg.mozilla.org/mozilla-central/rev/0627e8608a97
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 23
It seems that the order of the tests can also influence the failures. With a bit of change to account for the loading problems of robocop_boxes.html after bloackForGeckoReady() and rearranging most of the disabled tests the run looks better. I still get restarts and connection timeouts but they are a lot less frequent and also the connection timeouts fails seem to not be influenced by the actual tests.
https://tbpl.mozilla.org/?tree=Try&rev=4be6711c762c

Is there a way we could run all the tests in 3 separate robocop runs instead of 2 like we do know and see if the actual number of tests run in a suite influence the errors? Were the lower number of tests with the 7 disabled tests the cause for the runs going green or did those tests actually influence the run? Running the full suite locally on different devices (Pandaboard (Android 4.1), LG Optimus 2X (Android 2.2), HTC Desire (Android 2.2) ) I am not seeing any issues.
(In reply to Adrian Tamas from comment #6)
> Were the lower number of tests with the 7 disabled
> tests the cause for the runs going green or did those tests actually
> influence the run? 

I am not 100% sure, but I believe it is not just the lower number of tests. I tried a number of experiments on try with different subsets of tests; they generally failed until I started looking at which test actually failed and eliminated each in turn.

There has been speculation that the failures might be related to our panda power supply issue. I intend to try re-introducing these tests once that is resolved.
Ported this to beta in https://hg.mozilla.org/releases/mozilla-beta/rev/3a42947b91b8 since someone decided to turn on panda tests there even though runs like https://tbpl.mozilla.org/?tree=Mozilla-Beta&rev=67b0e4327b11 taking a dozen tries to make it through robocop are pretty typical.
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.