637660 - add sandbox.Android, and skip a few crashtests so we can run green on Android

Status: RESOLVED FIXED

(Testing :: Reftest, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: skip-if(xulRuntime.OS=="Android") for crashtests.list that hang/fail (1.0)

this patch ends up doing a skip-if(xulRuntime.OS=="Android") on all hangs and failures.  There are 5 failures that don't need a skip-if, but a "fails-if(..) load filename.html" results in a manifest parsing error.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 515913 [details] [diff] [review]
skip-if(xulRuntime.OS=="Android") for crashtests.list that hang/fail (1.0)

for remote testing, we don't really support .xul (it doesn't load over http).  That is the majority of the changes here for crashtests.

Comment 2

[(not currently active) Ted Mielczarek]

For pedantry's sake, you should use skip-if for tests that hang or crash, and fails-if for tests that fail. (skipped tests don't get run at all, fail tests will still run, which is nice).

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

yes, I use skip-if for all the .xul files which hang and tried to use fails-if, but I ran into manifest errors when using a 'fails-if(...) load filename'.  So there are some comments in those instances indicating it should be a fails-if instead of a skip-if.

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Here:
http://hg.mozilla.org/mozilla-central/file/372eca6704c2/layout/tools/reftest/reftest.js#l404
you could add a line saying:
  sandbox.Android = xr.OS == "Android";

and then you could just write all of this as skip-if(Android), fails-if(Android), etc., which would be preferable.


Don't we flip the remote XUL pref while running reftest and crashtest?  Is that insufficient for loading over HTTP?  Or is the mobile harness just not doing that correctly?

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: skip-if(Android) for tests that hang (2.0)

updated to include a sandbox.Android as suggested in previous comment.  Also updated patch to register a remoteXUL instance for a webserver when we are accessing a manifest via http[s].  This works great and only 2 crashtests need to be skip-if to run clean on android tegra.

Comment 6

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Comment on attachment 516131 [details] [diff] [review]
skip-if(Android) for tests that hang (2.0)

I'd somewhat prefer not to have the permission manager stuff in reftest.js, which I'd prefer not to mess with profiles or mess with them as little as possible.

We already have stuff to set up the appropriate permissions in automation.py and runreftest.py:
http://mxr.mozilla.org/mozilla-central/search?string=setuppermissions
Maybe remotereftest.py needs similar code?

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: skip-if(Android) for tests that hang (3.0)

thanks for the pointer to how this is done in automation.py.  I wasn't aware we had added support for permissions.sqlite entries.  I was able to do this and have a working patch that does the trick.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

this patch is still blocking us from running crashtests on android.

Comment 9

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Comment on attachment 516283 [details] [diff] [review]
skip-if(Android) for tests that hang (3.0)

r=dbaron.  Sorry for the delay.

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

http://hg.mozilla.org/mozilla-central/pushloghtml?changeset=2269cb1136f9

Comment 11

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Have bugs been filed to re-enable these test? If not, please do. In any case, the manifests should be annotated.

Comment 12

[[e-mail] (:bhearsum)]

bug 638844 tracks it.