869694 - enable layout/reftests/bugs on android

Status: RESOLVED FIXED

(Core :: Layout, defect)

Description

[Timothy Nikkel (:tnikkel)]

I think this is an important omission.

It's enabled on b2g.

Comment 1

[Timothy Nikkel (:tnikkel)]

Here is a try run with about 60 failures
https://tbpl.mozilla.org/?tree=Try&rev=7f601f035eac
A good chunk are unexpected passes, I think most of the failures are annotated as skip or fails on b2g already, and there are some after that to look at still.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

this might be a quick fix.  Our bar for turning on new android tests is to run each test suite (chunk in this case) 10 times and have consistent results.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

first pass at fixing the manifest:
https://tbpl.mozilla.org/?tree=Try&rev=3d878ead1491

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: fix the bugs/reftest.list manifest to support android (1.0)

this is my second attempt to fix this.  My first attempt missed a couple instances.  This patch is up on try server right now:
https://tbpl.mozilla.org/?tree=Try&rev=76298dc34770

If this looks good we can review and turn on hundreds of additional reftests for android.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: fix the bugs/reftest.list manifest to support android (2.0)

Looking good on try server:
https://tbpl.mozilla.org/?tree=Try&rev=4ea6ddeeeaa1

Comment 6

[Timothy Nikkel (:tnikkel)]

Comment on attachment 747199 [details] [diff] [review]
fix the bugs/reftest.list manifest to support android (2.0)

Yay! Thanks Joel for picking this up so quickly that I didn't even have a chance to get to it.

It's great that we'll have coverage for one of the most important reftest directories on android.

Any particular reason that the first few annotations are skip-if(Android) and the rest are fails-if(Android)? I think fails-if is better, but that's a nit.

I'm going to try to go through the tests that fail and see if any of them point out any major problems that we should have bugs filed on.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: fix the bugs/reftest.list manifest to support android (2.1)

Updated to change most of the skip-if to fails-if and tested on try!  In all my testing I have seen a couple random failures crop up from this, we are looking at those being really random orange, not some 20%+ of the time thing.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/75d1409ae090

Comment 9

[Timothy Nikkel (:tnikkel)]

I marked two fails-if as random-if because they were passing sometimes.
https://hg.mozilla.org/integration/mozilla-inbound/rev/c3626167f6d8

Comment 10

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/75d1409ae090
https://hg.mozilla.org/mozilla-central/rev/c3626167f6d8

Comment 11

[Ryan VanderMeulen [:RyanVM]]

753329-1.html also likes to pass from time to time.
https://hg.mozilla.org/integration/mozilla-inbound/rev/d72307b5972b

Comment 12

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

https://hg.mozilla.org/mozilla-central/rev/d72307b5972b