865006 - Run more xpcshell tests on Android

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Geoff Brown [:gbrown]]

Android xpcshell is running reliably now, but only runs one test. We can run many more with just a little effort.

Comment 1

[Geoff Brown [:gbrown]]

Attachment: First patch: adds chrome, locale, netwerk, zipwriter, xml, image, xpcshell example, xpcom, rdf

These all pass locally. Just need to get the modules dir set correctly in automated runs before checking this in.

Comment 2

[Geoff Brown [:gbrown]]

Comment on attachment 741035 [details] [diff] [review]
First patch: adds chrome, locale, netwerk, zipwriter, xml, image, xpcshell example, xpcom, rdf

This patch updates xpcshell manifests to run more android tests.

Runs fine on try:

https://tbpl.mozilla.org/?tree=Try&rev=9ff36c69d1c4

There is nothing special about this particular subset of tests: It's just the ones I started with. I will try enabling more tests in batches in the coming weeks.

I intend to update the bugs associated with modified skip-if/fail-if's (eg, bug 675039) once this lands.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 741035 [details] [diff] [review]
First patch: adds chrome, locale, netwerk, zipwriter, xml, image, xpcshell example, xpcom, rdf

Review of attachment 741035 [details] [diff] [review]:
-----------------------------------------------------------------

I would like to see the try server run for this and see 10 greens in a row if possible.

Comment 4

[Geoff Brown [:gbrown]]

First patch: https://hg.mozilla.org/integration/mozilla-inbound/rev/a0e3982ba3fb

Typical output:

INFO | Passed: 261
INFO | Failed: 0
INFO | Todo: 15
program finished with exit code 0
elapsedTime=913.717827

Comment 5

[Geoff Brown [:gbrown]]

That started to fail on inbound, for android armv6 in the netwerk tests -- I disabled all of the netwerk tests again: 

https://hg.mozilla.org/integration/mozilla-inbound/rev/0c6b9bedfa65

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/a0e3982ba3fb

Comment 7

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/mozilla-central/rev/0c6b9bedfa65 (landed with wrong bug #)

Comment 8

[Geoff Brown [:gbrown]]

Attachment: (2) Add more xpcshell tests - mostly dom

https://tbpl.mozilla.org/?tree=Try&rev=a75e2d8ca68f

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 745730 [details] [diff] [review]
(2) Add more xpcshell tests - mostly dom

Review of attachment 745730 [details] [diff] [review]:
-----------------------------------------------------------------

wow, we are about half way there.

Comment 10

[Geoff Brown [:gbrown]]

(2) https://hg.mozilla.org/integration/mozilla-inbound/rev/9fedd4a2170e

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/9fedd4a2170e

Comment 12

[Geoff Brown [:gbrown]]

Attachment: (3) Add more xpcshell tests

https://tbpl.mozilla.org/?tree=Try&rev=cc3ba7368a27

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 747951 [details] [diff] [review]
(3) Add more xpcshell tests

Review of attachment 747951 [details] [diff] [review]:
-----------------------------------------------------------------

wow, are we close to the end here?

Comment 14

[Geoff Brown [:gbrown]]

We are making progress, but there's still another 60 or so manifests listed in xpcshell.ini not yet in xpcshell_android.ini.


(3) https://hg.mozilla.org/integration/mozilla-inbound/rev/1c84eeff5a68

Comment 15

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

https://hg.mozilla.org/mozilla-central/rev/1c84eeff5a68

Comment 16

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

This should not have landed with DONTBUILD...

Comment 17

[Geoff Brown [:gbrown]]

Quick status update: I am holding off on enabling more tests until xpcshell tests are running on Android 4.0/panda -- bug 874965. Progress on that bug is waiting on the ability to run Android 4.0 X on try -- bug 881293. AFAIK, none of this is a high priority; if that's not the case, please comment.

Comment 18

[Geoff Brown [:gbrown]]

Attachment: (4) Add more xpcshell tests; skip a few for panda and x86

xpcshell tests for Panda/Android 4.0 can be run, hidden, on try now. 

This patch adds a few more manifests and skips various tests for failures on the various Android platforms. The result runs clean on Android 2.2, Android armv6, Android 4.0, and Android x86 (coming soon!). 

https://tbpl.mozilla.org/?tree=Try&rev=8acbac0913ad&showall=1

I hope to wrap up this bug with 2 or 3 more patches.

Comment 19

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 794979 [details] [diff] [review]
(4) Add more xpcshell tests; skip a few for panda and x86

Review of attachment 794979 [details] [diff] [review]:
-----------------------------------------------------------------

this looks like a nice addition to our xpcshell tests!

Comment 20

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/f0efe070e0b4

Comment 21

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/f0efe070e0b4

Comment 22

[Geoff Brown [:gbrown]]

Attachment: wip (5) sort existing includes for easier maintenance

Comment 23

[Geoff Brown [:gbrown]]

Attachment: wip (6) add all manifests (find -name xpcshell.ini), remove references to files that don't exist, massage until green

Comment 24

[Geoff Brown [:gbrown]]

I resurrected the wip patches, updated as necessary, and verified on try:

https://tbpl.mozilla.org/?tree=Try&rev=5bef6486d53f

This feels like a good place to close this bug. About half of all xpcshell tests now run on Android. Many of the remaining manifests require substantial skip-if/fail-if annotations for Android, and often tests fail on Android simply because the feature under test is not implemented, or is implemented differently, on Android. We've established a good framework here and shown that xpcshell tests can be run; I'll leave it to developers to add more tests as desired. 

https://hg.mozilla.org/integration/mozilla-inbound/rev/ddda7acf3739

Comment 25

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/ddda7acf3739