845767 - Create an Autophone specific version of the robocop manifest.

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Bob Clary [:bc] (inactive)]

Attachment: patch

Testing real phones with autophone+robocop results in a number of failures. I ran tests with:

Droid Pro        2.3.3
Nexus One      2.3.6
Nexus S          2.3.6
Samsung GS2 2.3.6
Samsung GS2 4.0.3
LG Revolution  2.3.4

I commented out the tests which fail consistently across the phones.

Nexus One and Samsung GS2 2.3.6 pass all remaining tests consistently.

The remaining issues are:

Nexus S failed 1 of 6 tries with:
testAwesomebar
Shutdown
PROCESS-CRASH | application crashed [@ libpvrANDROID_WSEGL.so + 0x73c]

Droid Pro fails at least 1 of 3 tries with:
testBookmarksTab
TEST-UNEXPECTED-FAIL | Exception caught - junit.framework.AssertionFailedError: View is null and can therefore not be clicked!

LG Revolution fails 2 of 3 tries with:
java-exception
PROCESS-CRASH | android.database.sqlite.SQLiteDiskIOException: error code 10: disk I/O error at android.database.sqlite.SQLiteStatement.native_execute(Native Method)

Samsung GS2 4.0.3 fails consistently with:
testBookmark
TEST-UNEXPECTED-FAIL | Exception caught - junit.framework.AssertionFailedError: The text: Bookmark is not found!

testBookmarksTab
TEST-UNEXPECTED-FAIL | Exception caught - junit.framework.AssertionFailedError: The text: Cancel is not found!

testBrowserProvider - TestInsertBookmarks
TEST-UNEXPECTED-FAIL | Should not be able to insert bookmark with invalid parent - got 11, expected -1
TEST-UNEXPECTED-FAIL | Exception caught - junit.framework.AssertionFailedError: 3
TEST-UNEXPECTED-FAIL | Should not be able to insert bookmark with invalid parent - got 11, expected -1

I think this manifest is sufficient since:

LG Revolution can't be used for the JS tests due to its lack of timezone setting.

We can put the GS2 4.0.3 phone on s1s2 and not run it on the UnitTests. If we disable the tests which fail in GS2 4.0.3 we lose testBrowserProvider and its 4353 tests.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 718921 [details] [diff] [review]
patch

Review of attachment 718921 [details] [diff] [review]:
-----------------------------------------------------------------

Can you put a comment after the test case name as to which platforms it fails on.  I am not sure if exactly what the comments indicate.

One problem I foresee is that as we add new tests to robocop for panda/tegra we will not add them to autophone.  Can we leave a bug open for finding a more compatible solution?

Comment 2

[Bob Clary [:bc] (inactive)]

(In reply to Joel Maher (:jmaher) from comment #1)
> Comment on attachment 718921 [details] [diff] [review]

> Can you put a comment after the test case name as to which platforms it
> fails on.  I am not sure if exactly what the comments indicate.
> 

I tried # comments after uncommented test names and it caused the tests to fail with no tests found in the manifest. I put the # comments under each test to indicate the phones the test failed upon and some indication if it was random and that worked ok. That allowed me to comment on a test which was not commented out in a consistent fashion to the others. Is that not sufficient?

> One problem I foresee is that as we add new tests to robocop for panda/tegra
> we will not add them to autophone.  Can we leave a bug open for finding a
> more compatible solution?

Yeah, without some conditional means of marking the tests or identifying the phones it will be a problem but I don't know how we might approach that. It does appear from my results that the Robocop tests are Android specific at least with respect to the gs2's. Shall I file a bug on conditional Robocop manifests?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

can you add whitespace so it is easy to see which comment belongs with which test case?

Please file a bug for us to figure out way to add better conditional logic into the manifests.  I suspect a handful of your tests could be fixed to work on those environments, Personally I think getting robocop tests running 100% on all phones would be a higher priority than mochitest/reftest.  I have found that some tests fail on panda or tegra, but not on both.  Small adjustments need to be made and robocop has some logic to detect different versions of the browser, os, and layout.

Comment 4

[Bob Clary [:bc] (inactive)]

http://hg.mozilla.org/integration/mozilla-inbound/rev/c87c6b23794e

Comment 5

[Paul Adenot (:padenot)]

Attachment: Initialize mPauseForPlaybackRateNull in MediaDecoder constructor to prevent an assertion. r=

This fixes both assertions.

Sorry about that, I'll try to keep an eye open for new assertions in the future.

Comment 6

[Bob Clary [:bc] (inactive)]

Paul, I think you have the wrong bug.

Comment 7

[Geoff Brown [:gbrown]]

fwiw, there is current work on-going in bug 784242 to fix the PixelTest tests on Galaxy Nexus and other devices: testLoad, testAxisLocking, testOverscroll, testPanCorrectness.

I would like to see bugs opened - hopefully with logs! - for the other Robocop failures.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/c87c6b23794e