1558334 - Turn off web-platform-tests on Fennec

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, task, P3)

Description

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

The current status of web-platform-tests on Android is:

• web-platform-tests (including reftests) run on Fennec on mozilla-central and try. (Tier-2)
• web-platform-tests (Tier-2) and web-platform-tests-reftests (Tier-3) run on GeckoView on mozilla-central and try. These are being greened up and will soon be Tier-2.

Now that we're only interested in Fennec on ESR68, we can turn the corresponding wpt tests off, at least on mozilla-central.

To be clear, there are no web-platform-tests running on Android on mozilla-esr68.

We originally stood up wpt on Fennec last year to validate that wpt can run in an Android environment in CI. This was a bit before GeckoView/TestRunnerActivity had all the prerequisites ready for wpt.

Comment 1

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Attachment: Bug 1558334 - Turn off web-platform-tests on Fennec; r=gbrown!

Comment 2

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

James, wpt-sync is the main consumer of wpt results for Fennec on try. Given that Fennec is part of ESR68, do you still want to include Fennec results in the wpt-sync metadata update?

Comment 3

[Kevin Brosnan [Ex-Mozilla]]

We wanted to wait for Firefox for Android to ship the first ESR of no return which is ESR 71 before disabling Fennec tests. See bug 1548659 comment 0.

Comment 4

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

(In reply to Kevin Brosnan [:kbrosnan] from comment #3)

We wanted to wait for Firefox for Android to ship the first ESR of no return which is ESR 71 before disabling Fennec tests. See bug 1548659 comment 0.

No rush, happy to hold off.

Comment 5

[James Graham [:jgraham]]

Fromt he wpt-sync point of view the requirements are purely functional; we need to have results on try for anything that's running on integration branches or central. I don't have any other specific opinions on the value of fennec tests so I defer to the relevant engineering teams.

Comment 6

[Geoff Brown [:gbrown]]

(In reply to Kevin Brosnan [:kbrosnan] from comment #3)

We wanted to wait for Firefox for Android to ship the first ESR of no return which is ESR 71 before disabling Fennec tests. See bug 1548659 comment 0.

Can we reconsider? These run on packet.net, which has limited capacity. The Fennec wpt tests on Android 7.0 are not part of the "traditional" Android 4.3 range of test suites, run as tier 2 and have several perma-fails -- I'm not sure they really provide much coverage in this state.

Comment 7

[Kevin Brosnan [Ex-Mozilla]]

If there are well thought out reasons for disabling then I am happy with that decision. I just wanted to make sure that we thought about the reasons before disabling tests wholesale before 71.

Dropping a ni to Ryan for his info. Feel free to clear without comment if there is nothing to add.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

If it's not a test suite we care about for Fennec (and it sounds like that's the case), I don't see the harm in turning them off. Mainly we just want to avoid breaking any Fennec test suites we do care about until we're past the point of no return for Fennec shipping from anywhere other than ESR68.

Looking at it from another perspective, it sounds like this is something we would have wanted to do even if Fennec were still riding the trains as usual, further making me think we're fine to proceed here.

Thanks for checking, though!

Comment 9

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #8)

Looking at it from another perspective, it sounds like this is something we would have wanted to do even if Fennec were still riding the trains as usual, further making me think we're fine to proceed here.

Well, if Fennec were riding the trains we'd make an effort to green these up properly. It makes more sense to focus on wpt on GeckoView now.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f2e0194bb006
Turn off web-platform-tests on Fennec; r=gbrown

Comment 11

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/f2e0194bb006