862493 - Disable 7 rc tests that cause instability on Panda

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Geoff Brown [:gbrown]]

See https://bugzilla.mozilla.org/show_bug.cgi?id=855406#c3.

This is a proposal to disable 7 robocop tests so that Panda rc will pass consistently, allowing us to un-hide Android 4.0 rc.

The tests are:

-[testAxisLocking]
-[testAboutPage]
-[testWebContentContextMenu]
-[testFormHistory]
-[testDoorHanger]
-[testClearPrivateData]
-[testSystemPages]

The obvious trade-off is that we won't be running these tests at all; but we will be running the remaining tests on both tegra and panda (Android 2.2 and 4.0).

Of course we intend to follow-up with investigations to re-enable the tests, but that doesn't look easy!

Comment 1

[Geoff Brown [:gbrown]]

https://tbpl.mozilla.org/?tree=Try&rev=c9065600a3df

Comment 2

[Geoff Brown [:gbrown]]

Attachment: disable 7 tests that cause panda instability

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 738123 [details] [diff] [review]
disable 7 tests that cause panda instability

Review of attachment 738123 [details] [diff] [review]:
-----------------------------------------------------------------

awesome, lets get these green and reliable!

Comment 4

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/0627e8608a97

Comment 5

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/0627e8608a97

Comment 6

[Adrian Tamas (:AdrianT)]

It seems that the order of the tests can also influence the failures. With a bit of change to account for the loading problems of robocop_boxes.html after bloackForGeckoReady() and rearranging most of the disabled tests the run looks better. I still get restarts and connection timeouts but they are a lot less frequent and also the connection timeouts fails seem to not be influenced by the actual tests.
https://tbpl.mozilla.org/?tree=Try&rev=4be6711c762c

Is there a way we could run all the tests in 3 separate robocop runs instead of 2 like we do know and see if the actual number of tests run in a suite influence the errors? Were the lower number of tests with the 7 disabled tests the cause for the runs going green or did those tests actually influence the run? Running the full suite locally on different devices (Pandaboard (Android 4.1), LG Optimus 2X (Android 2.2), HTC Desire (Android 2.2) ) I am not seeing any issues.

Comment 7

[Geoff Brown [:gbrown]]

(In reply to Adrian Tamas from comment #6)
> Were the lower number of tests with the 7 disabled
> tests the cause for the runs going green or did those tests actually
> influence the run? 

I am not 100% sure, but I believe it is not just the lower number of tests. I tried a number of experiments on try with different subsets of tests; they generally failed until I started looking at which test actually failed and eliminated each in turn.

There has been speculation that the failures might be related to our panda power supply issue. I intend to try re-introducing these tests once that is resolved.

Comment 8

[Phil Ringnalda (:philor)]

Ported this to beta in https://hg.mozilla.org/releases/mozilla-beta/rev/3a42947b91b8 since someone decided to turn on panda tests there even though runs like https://tbpl.mozilla.org/?tree=Mozilla-Beta&rev=67b0e4327b11 taking a dozen tries to make it through robocop are pretty typical.