Closed Bug 1558334 Opened 4 months ago Closed 4 months ago

Turn off web-platform-tests on Fennec

Categories

(Firefox for Android :: Testing, task, P3)

task

Tracking

()

RESOLVED FIXED
Firefox 69
Tracking Status
firefox69 --- fixed

People

(Reporter: maja_zf, Assigned: maja_zf)

References

Details

Attachments

(1 file)

The current status of web-platform-tests on Android is:

  • web-platform-tests (including reftests) run on Fennec on mozilla-central and try. (Tier-2)
  • web-platform-tests (Tier-2) and web-platform-tests-reftests (Tier-3) run on GeckoView on mozilla-central and try. These are being greened up and will soon be Tier-2.

Now that we're only interested in Fennec on ESR68, we can turn the corresponding wpt tests off, at least on mozilla-central.

To be clear, there are no web-platform-tests running on Android on mozilla-esr68.

We originally stood up wpt on Fennec last year to validate that wpt can run in an Android environment in CI. This was a bit before GeckoView/TestRunnerActivity had all the prerequisites ready for wpt.

James, wpt-sync is the main consumer of wpt results for Fennec on try. Given that Fennec is part of ESR68, do you still want to include Fennec results in the wpt-sync metadata update?

Flags: needinfo?(james)

We wanted to wait for Firefox for Android to ship the first ESR of no return which is ESR 71 before disabling Fennec tests. See bug 1548659 comment 0.

(In reply to Kevin Brosnan [:kbrosnan] from comment #3)

We wanted to wait for Firefox for Android to ship the first ESR of no return which is ESR 71 before disabling Fennec tests. See bug 1548659 comment 0.

No rush, happy to hold off.

Fromt he wpt-sync point of view the requirements are purely functional; we need to have results on try for anything that's running on integration branches or central. I don't have any other specific opinions on the value of fennec tests so I defer to the relevant engineering teams.

Flags: needinfo?(james)
Priority: -- → P3

(In reply to Kevin Brosnan [:kbrosnan] from comment #3)

We wanted to wait for Firefox for Android to ship the first ESR of no return which is ESR 71 before disabling Fennec tests. See bug 1548659 comment 0.

Can we reconsider? These run on packet.net, which has limited capacity. The Fennec wpt tests on Android 7.0 are not part of the "traditional" Android 4.3 range of test suites, run as tier 2 and have several perma-fails -- I'm not sure they really provide much coverage in this state.

Flags: needinfo?(kbrosnan)

If there are well thought out reasons for disabling then I am happy with that decision. I just wanted to make sure that we thought about the reasons before disabling tests wholesale before 71.

Dropping a ni to Ryan for his info. Feel free to clear without comment if there is nothing to add.

Flags: needinfo?(kbrosnan) → needinfo?(ryanvm)

If it's not a test suite we care about for Fennec (and it sounds like that's the case), I don't see the harm in turning them off. Mainly we just want to avoid breaking any Fennec test suites we do care about until we're past the point of no return for Fennec shipping from anywhere other than ESR68.

Looking at it from another perspective, it sounds like this is something we would have wanted to do even if Fennec were still riding the trains as usual, further making me think we're fine to proceed here.

Thanks for checking, though!

Flags: needinfo?(ryanvm)

(In reply to Ryan VanderMeulen [:RyanVM] from comment #8)

Looking at it from another perspective, it sounds like this is something we would have wanted to do even if Fennec were still riding the trains as usual, further making me think we're fine to proceed here.

Well, if Fennec were riding the trains we'd make an effort to green these up properly. It makes more sense to focus on wpt on GeckoView now.

Pushed by mjzffr@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/f2e0194bb006
Turn off web-platform-tests on Fennec; r=gbrown
Status: NEW → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 69
You need to log in before you can comment on or make changes to this bug.