1550895 - Green up geckoview wpt so they can be upgraded to tier-2 visibility.

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, task, P3)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Now that the wpt suite is running against geckoview and expectation metadata has been updated to account for that platform's current state, there are still several tests/dirs that are frequently intermittently failing in unexpected ways. Frequent enough that it prevents me from moving these tests from tier-3 to tier-2 (where they get shown by default in treeherder and sheriffs actively file bugs about intermittent failures).

I'm going to disable my way to victory on these tests so I can bump them to tier-2 in the next week or so. (Can't do tier-1, we don't have the capacity at the moment to run these per-push on integration branches, so they only run on mozilla-central and try at the moment. Treeherder's Add New Jobs (Search) feature CAN run these jobs on inbound and autoland if you filter for android x86_64 web-platform.)

Setting the bug as leave-open for now because I expect I'll have to play whackamole on the tests over a few rounds of disablings to get things truly green. I'll resolve the bug when I bump the jobs to tier-2.

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1550895 - Disable some additional frequently failing geckoview wpt tests

This disables a few additional tests and updates the annotations for other disabled tests to point to this bug instead of the general geckoview-wpt bug.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cd771a0b9c07
Disable some additional frequently failing geckoview wpt tests

Comment 3

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/cd771a0b9c07

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1550895 - Disable some additional frequently failing geckoview wpt tests

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c23eb772cf2d
Disable some additional frequently failing geckoview wpt tests

Comment 6

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/c23eb772cf2d

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1550895 - disable some additional frequently failing geckoview wpt tests

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/64bfa1117b35
disable some additional frequently failing geckoview wpt tests

Comment 9

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/64bfa1117b35

Comment 10

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1550895 - Disable some more frequently failing wpt tests on geckoview

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e1080b62268c
Disable some more frequently failing wpt tests on geckoview

Comment 12

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/e1080b62268c

Comment 13

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1550895 - Run geckoview wpt as tier-2 jobs r?jgraham

This patch runs wpt against opt and debug builds of geckoview as tier-2 jobs on mozilla-central and try.

Comment 14

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1550895 - Run the android web-platform tests on all built-projects r?gbrown

This takes the android wpt jobs that have previously been constrained to running against mozilla-central and try, and will begin also running them on integration branches.

Depends on D31581

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a4c80ac02e50
Run geckoview wpt as tier-2 jobs r=jgraham

Comment 16

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/a4c80ac02e50

Comment 17

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

They're tier-2 now, so I think we can resolve this bug. I might still use this as a meta bug for tracking all of the gv wpt failures that sheriffs file, at least for a while until they get greened up more.

Comment 18

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1550895 - Bump android gv wpt back up to tier-3

Bug 1553584 was filed and shows a race condition in the test harness that could make fixing bugs difficult. Bumping these back to tier-3 until that can be fixed.

Comment 19

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

We need to rehide these until bug 1553584 can be fixed.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/217038bea1c7
Bump android gv wpt back up to tier-3

Comment 21

[James Graham [:jgraham]]

I think we should have put the reftests down to Tier 3 and left the other tests as Tier 2.

Comment 22

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/217038bea1c7

Comment 23

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1550895 - Clear up ambiguities in android wpt tier definitions so they apply correctly.

Taskcluster didn't correctly apply the change back to tier-3 in the previous patch because the wildcard setting the tier to 2 by default beat out the specific line setting tier-3 for the geckoview tests. This patch specifically changes the tier for non-geckoview reftests (fennec runs on x86 emulators) to tier-2, while geckoview reftests (TRA runs on x86_64 emulators) to tier-3.

This change also explicitly sets the non-reftest tests to tier-2 for all android platforms, because they didn't have the permafails that the reftests seem to have.

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/61e78fd6c946
Clear up ambiguities in android wpt tier definitions so they apply correctly.

Comment 25

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/61e78fd6c946

Comment 26

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Attachment: Bug 1550895 - Bump web-platform-tests-reftests back to Tier-2 on Android; r=jgraham!

Now that Bug 1553584 is fixed, we can unhide these tests again.

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/59ed9ee0bd98
Bump web-platform-tests-reftests back to Tier-2 on Android; r=jgraham

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/253fa3bd2d74
Run the android web-platform tests on all built-projects r=gbrown,coop

Comment 29

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/59ed9ee0bd98
https://hg.mozilla.org/mozilla-central/rev/253fa3bd2d74

Comment 30

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

I'm going to update expectation metadata for a few more tests, then resolve this bug.

Comment 31

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Attachment: Bug 1550895 - Disable a few more tests to green up wpt on GeckoView; r=jgraham

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7b99cdff2976
Disable a few more tests to green up wpt on GeckoView; r=jgraham

Comment 33

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/7b99cdff2976

Comment 34

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

These have been Tier-2 for a while so let's close this bug.

A number of tests have been disabled in the past months: I'll organize the work to be done under Bug 1563334.