Closed Bug 1546452 Opened 5 months ago Closed 5 months ago

reduce headless mochitests

Categories

(Testing :: General, task)

Version 3
task
Not set

Tracking

(firefox68 fixed)

RESOLVED FIXED
mozilla68
Tracking Status
firefox68 --- fixed

People

(Reporter: jmaher, Assigned: gbrown)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

I see we duplicate our mochitests linux64 and windows10 to be run in headless mode. These do run faster than normal, which is great- these also give us the option to run things in parallel (see bug 1419135) with some additional work.

Unfortunately we haven't put a lot of effort into advancing the effort of headless testing. Until there is a drive and agreement that we could replace browser tests with headless tests for the 98% of the time, then we should consider reducing our headless testing, but not get rid of it.

I propose:

  • removing debug headless tests
  • reducing headless to one config: either linux64 or windows10

we might want to consider:

  • making remaining headless tests as tier-2 that run on try/m-c
    ** only do this if we can ensure adequate sheriffing and that the tests won't be broken all the time.

:aryx - do you think that tier-2 tests can be adequately sheriffed (I know not real time, but within 2 business days) to find perma failures, regressions, and ask for backouts or fixes to the tests?

Flags: needinfo?(aryx.bugmail)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #0)

I propose:

  • reducing headless to one config: either linux64 or windows10
    If the goal is to ensure that headless testing still works, please go with linux64 as it's supposed to be cheaper and scale better.

we might want to consider:

  • making remaining headless tests as tier-2 that run on try/m-c
    ** only do this if we can ensure adequate sheriffing and that the tests won't be broken all the time.

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #1)

:aryx - do you think that tier-2 tests can be adequately sheriffed (I know not real time, but within 2 business days) to find perma failures, regressions, and ask for backouts or fixes to the tests?
Yes. Effectively it should reduce the number of test failures and most of them should be shared with normal mochitest plain chunks.

Flags: needinfo?(aryx.bugmail)

action:

  • stop mochitest-headless on windows10
  • stop mochitest-headless on linux64/debug
  • make mochitest-headless tier-2
  • make mochitest-headless run on m-c/try

:gbrown, want to pick this up? I think this should be relatively easy to do.

This still leaves marionette-headless running on most if not all platforms.

:whimboo, should we consider reducing marionette headless to a few configs instead of all?

Flags: needinfo?(hskupin)
Flags: needinfo?(gbrown)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #3)

:whimboo, should we consider reducing marionette headless to a few configs instead of all?

Can you please file a separate bug for that under Testing:Marionette? Thanks.

Flags: needinfo?(hskupin)
Type: defect → task
Assignee: nobody → gbrown
Flags: needinfo?(gbrown)
  • stop mochitest-headless on windows10
  • stop mochitest-headless on linux64/debug
  • make mochitest-headless tier-2
  • make mochitest-headless run on m-c/try

This should eliminate mochitest-headless on integration branches, 47 fewer tasks, generally run on every 5th push. It will also eliminate 47 mochitest-headless tasks per push on mozilla-central, and a variable number of tasks on try.

Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/18ba664cbccf
Run fewer headless mochitests; r=jmaher
Status: NEW → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68
You need to log in before you can comment on or make changes to this bug.