Disable non-e10s tests for OSX debug only

RESOLVED FIXED

Status

task
RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: jgriffin, Assigned: spacurar)

Tracking

Details

Attachments

(4 attachments, 2 obsolete attachments)

(Reporter)

Description

2 years ago
We've hit a wall with the number of tests we can run on OSX hardware and neet to reduce coverage, especially as we take segments of the OSX hardware pool offline to prep for the next datacenter move.

Let's turn off non-e10s versions of suites on OSX debug:

mochitest-plain
mochitest-browser-chrome
mochitest-clipboard
mochitest-devtools
mochitest-gpu
mochitest-media
mochitest-webgl
marionette
reftest
crashtest
jsreftest

The e10s versions of all these will remain running, and the non-e10s versions will continue to run alongside them on OSX opt.

Shell, let us know if there are any concerns here.
web-platform-tests
Component: General Automation → Buildduty
QA Contact: catlee → bugspam.Callek
jgriffin: is this a change that should ride the trains or should the tests be enabled on all branches?
Flags: needinfo?(jgriffin)

Comment 2

2 years ago
adding andy and ryanVM who would have more insight into the specific tests importance.
Flags: needinfo?(ryanvm)
Flags: needinfo?(amckay)
(Reporter)

Comment 3

2 years ago
(In reply to Kim Moir [:kmoir] from comment #1)
> jgriffin: is this a change that should ride the trains or should the tests
> be enabled on all branches

Ride the trains, I think.
Flags: needinfo?(jgriffin)
Some of these suites concern me because of missed debug coverage of the OSX graphics pipeline, but I'm also uncertain as to how different the e10s and non-e10s codepaths are in that respect. Would be good to have Milan or someone from his team weigh in on the potential risks here.
Flags: needinfo?(ryanvm)
Either way, *all* branches isn't an option since esr45 doesn't do e10s, so whether your gecko_version is at_least 46 or 54 you still need a loop.
Yes, we always check if they exist on the branch before removing them.  I was just asking for clarity for the buildduty folks when they arrive in the European morning.
I don't think we have the green light on this yet, lets see what :milan has to say about graphics support.  Possibly we keep mochitest-webgl/mochitest-gpu running- although there could be concerns to keep other suites running.

Comment 8

2 years ago
Concerned too given that non-e10s paths will happen until we can remove all legacy add-on support in 57 and everything is e10s support. Unfortunately I'm not familiar with exactly what those versions actually test.
Flags: needinfo?(amckay)
I think what we are proposing here is that we stop running debug specific tests on osx (only) for non-e10s.  we will still run opt tests (what we ship).  Debug tests catch leaks, assertions and have more logging.  We would run these debug tests for e10s tests and non-e10s on linux and windows 7, so we would still get coverage, especially on windows where our highest population of users exist.

Some downsides of keeping this around:
* We are at a tipping point in terms of available machine bandwidth for OSX- so new tests which are written end up causing more delays for developers waiting for try results
* We are spending time sheriff and developer time looking at failures in non-e10s mode, and in almost all the cases we fix the failures with changes to the test case, or disabling the test case.

While it might make sense to run specific tests in non-e10s debug for osx, I would really like to know that certain code paths are only seen on debug and osx.  RyanVM specified there might be graphics concerns, I would like to know what those are specifically and what other features have specific osx/debug code.
looking at the last 4 months of failures that we catch in automation (integration branches, not try), I see only a few commits that have only osx/debug failures, and narrowing that down to ones which we would have missed if only running e10s, we end up with xpcshell.  Otherwise we have a failure caught in automation and we have e10s tests running that would have caught the failure.

so based on historical data, we could assume all are ok to turn off except xpcshell.
Comment hidden (obsolete)
:cpeterson this bug actually is will not free up capacity on Linux, it's references capacity on Macosx only.  My apologies I added the bug to the quantum meeting notes.  I think jgriffin is following up with shell wrt disabling all non-e10s talos which will free up linux64 hardware capacity if implemented
No longer blocks: 1338871
What's the relative cost/computing time spent on different pieces?

mochitest-plain
mochitest-browser-chrome
mochitest-clipboard
mochitest-devtools
mochitest-gpu
mochitest-media
mochitest-webgl
marionette
reftest
crashtest
jsreftest

If it's critical to get rid of all, yes, keeping xpcshell because it catches things is good.  I don't know how difficult it would be to put together a rotating schedule and only running one of these (instead of all of them) at a time.  Once our e10s percentage on OS X gets to 80% we can probably turn them all off.
Gathered some data for the last week to get an idea on how many chunks/how much time is needed for each test. I only considered the OS X 10.10.5 debug non-e10s tests.

mochitest-plain: no tests running on OS X
mochitest-browser-chrome: 7 chunks, ~20 minutes/test
mochitest-clipboard: 1 chunk, ~20 minutes/test 
mochitest-devtools:8 chunks, duration varies between 26 and 38 minutes/test (depending on chunk)   
mochitest-gpu: 1 chunk, ~ 4 minutes/test
mochitest-media: 1 chunk, ~32 minutes
mochitest-gl: 3 chunks
    - mochitest-gl-1: ~26 minutes/test
    - mochitest-gl-2: ~16 minutes/test
    - mochitest-gl-3: ~ 15 minutes/test
marionette: 1 chunk, ~14 minutes/test
reftest: 2 chunks
    - reftest-1: 27 minutes/test
    - reftest-2: 36 minutes/test
crashtest: 1 chunk, ~10 minutes/test
jsreftest: 2 chunks
    - jsreftest-1: ~32 minutes/test
    - jsreftest-2: ~29 minutes/test
I found mochitest-plain-e10s in 5 chunks (1 data set, not an average):
1: 19 minutes
2: 20 minutes
3: 27 minutes
4: 29 minutes
5: 26 minutes
:milan, we don't currently have the capability to run the tests randomly.  This functionality could be implemented in the future as part of bug 1339133.  

So to summarize all the comments above, are we ready to disable all non-e10s tests on Macosx debug with the exception of xpcshell to ride the trains?

The buildduty folks are looking forward to writing these patches so we can get our pending counts down and deliver faster test results to developers.
aselagea I haven't heard any opposition to the approach in comment 16 so please proceed with someone on the buildduty team to write the patches.
(Assignee)

Updated

2 years ago
Assignee: nobody → spacurar
(Assignee)

Comment 18

2 years ago
Posted patch Bug 1339185_patch (obsolete) — Splinter Review
Attachment #8839531 - Flags: review?(kmoir)
Attachment #8839531 - Flags: review?(aselagea)
(Assignee)

Comment 19

2 years ago
Posted file diff regarding first patch (obsolete) —
(Assignee)

Comment 20

2 years ago
There are some problems with ash here. I believe I should write down this loop below the "Ash-specific branch config", because there are tests (non e10s) introduced on that branch for yosemite_r7. But as stated in the comment, "Please add any new buildbot test scheduling changes above this block", should I leave it like this?
Ash shouldn't ever be running non-e10s jobs. Which ones is it trying to schedule?
(Assignee)

Comment 22

2 years ago
I just double checked, and you're right, there aren't any non-e10s jobs. Sorry for the confusion.
Attachment #8839531 - Flags: review?(kmoir) → review+
Attachment #8839531 - Flags: review?(aselagea)
Comment on attachment 8839531 [details] [diff] [review]
Bug 1339185_patch

My understanding is that for OS X we'd want to remove all the non-e10s debug tests (minus xpcshell) from the trunk branches. 

From your diff I can see lots of e10s tests removed as well, so the patch will need some adjustments.
(Assignee)

Comment 24

2 years ago
Attachment #8839531 - Attachment is obsolete: true
Attachment #8839851 - Flags: review?(kmoir)
(Assignee)

Comment 25

2 years ago
Attachment #8839532 - Attachment is obsolete: true
(Assignee)

Comment 26

2 years ago
The script and diff file have been updated now. There are no e10s tests which are removed anymore. I omitted those few e10s tests which were removed in the first diff file. Thank you for your observation
Attachment #8839851 - Flags: review?(kmoir) → review+
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
This turned off mochitest-a11y and mochitest-chrome which only run on non-e10s. Please re-enable them.
Flags: needinfo?(spacurar)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Flags: needinfo?(spacurar)
Attachment #8840441 - Flags: review?(kmoir)
Attachment #8840441 - Flags: review?(kmoir) → review+
Attachment #8840441 - Flags: checked-in+
Checked some recent trunk pushes and can confirm that 'mochitest-a11y' and 'mochitest-chrome*' are enabled again.
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED
Depends on: 1342155
see bug 1342155
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
To sum this up, we only removed the OS X non-e10s debug tests that *do* have an e10s equivalent.
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED

Updated

a year ago
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.