Closed Bug 1569832 Opened 5 years ago Closed 5 years ago

Enable M-fis on tier2

Categories

(Testing :: General, task)

Version 3
task
Not set
normal

Tracking

(Fission Milestone:M4, firefox70 fixed)

RESOLVED FIXED
mozilla70
Fission Milestone M4
Tracking Status
firefox70 --- fixed

People

(Reporter: neha, Assigned: ahal)

References

(Blocks 1 open bug)

Details

Attachments

(4 files, 1 obsolete file)

This request is for enabling mochitests (except for remote and gpu) with fission on tier2 for Linux opt and debug builds and for windows opt platform.

Assignee: nobody → ahal
Fission Milestone: --- → M4

Aryx, we will still have one intermittent shutdown crash for the Linux debug builds. Is that okay or should we skip Linux debug builds until this gets fixed?

Flags: needinfo?(aryx.bugmail)
Blocks: fission
  1. If the intermittent shutdown crash is only after the execution of one test directory, you can disable that one and keep fission for Linux debug enabled.
  2. Are we talking about Linux 32-bit and 64-bit or only 64-bit? For Windows, it will run on both platforms?
  3. Is there a Try push available?
  4. Fission will only run on mozilla-central?
  5. It won't ride the trains for now?
  6. If a test fails only with fission, it won't get backed out because it's tier 2. If the patch author doesn't fix it, who should be needinfoed from the fission team to get it fixed? Else the test will get disabled for fission.
Flags: needinfo?(aryx.bugmail)

Hey Kris, could you verify this list is good for a first pass? I decided to just exclude all debug tests for now until Aryx's questions are answered (will be trivial to enable in a follow-up patch).

I also stuck to Windows 10 64 bit and Linux64. Let me know if you need other Windows platforms as well. Once you approve this list I'll do some testing on try and get review.

Flags: needinfo?(kmaglione+bmo)
Depends on: 1570097

(In reply to Andrew Halberstadt [:ahal] from comment #4)

Created attachment 9081726 [details]
List of new tasks that will run on central

Hey Kris, could you verify this list is good for a first pass? I decided to just exclude all debug tests for now until Aryx's questions are answered (will be trivial to enable in a follow-up patch).

Bug 1570097 fixes the intermittent shutdown hangs, so there's no reason not to enable debug at this point. Having debug runs is especially important, given that a lot of our new failures come from debug-only assertions that we really need to see.

Aside from that, the list looks good.

I also stuck to Windows 10 64 bit and Linux64. Let me know if you need other Windows platforms as well. Once you approve this list I'll do some testing on try and get review.

That should be fine, I think.

Flags: needinfo?(kmaglione+bmo)

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #2)

  1. If the intermittent shutdown crash is only after the execution of one test directory, you can disable that one and keep fission for Linux debug enabled.

Yes, but it's already been fixed.

  1. Are we talking about Linux 32-bit and 64-bit or only 64-bit? For Windows, it will run on both platforms?

For now, 64-bit only, I think.

  1. Is there a Try push available?

https://treeherder.mozilla.org/#/jobs?repo=try&revision=642cc6b4f6ead7fbb9fb092f25cda0a03e5e1cbb

  1. Fission will only run on mozilla-central?

It will also run on integration branches.

  1. It won't ride the trains for now?
    Correct.
  1. If a test fails only with fission, it won't get backed out because it's tier 2. If the patch author doesn't fix it, who should be needinfoed from the fission team to get it fixed? Else the test will get disabled for fission.

You can needinfo me or Andrew McCreight. However, if the test doesn't get fixed, I think the first recourse will probably be backing out the offending patch rather than disabling the test. We need to get as close as possible to 100% of tests passing under Fission as soon as possible, so patches which make the situation worse are not desirable.

Thank you for providing the information.

There are 2 permanent failures on the Linux/Windows x64 platforms in the Try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=642cc6b4f6ead7fbb9fb092f25cda0a03e5e1cbb
M(4) Assertion failure: !mIsBeingDestroyed, at z:/build/build/src/docshell/base/nsDocShell.cpp:5263
M(bcX) browser_thumbnails_bg_crash_during_capture.js

These should be gone when M-fis starts to run as tier 2, e.g. by tracking each issue in its own bug and disabling the test until it's fixed.

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #7)

There are 2 permanent failures on the Linux/Windows x64 platforms in the Try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=642cc6b4f6ead7fbb9fb092f25cda0a03e5e1cbb
M(4) Assertion failure: !mIsBeingDestroyed, at z:/build/build/src/docshell/base/nsDocShell.cpp:5263
M(bcX) browser_thumbnails_bg_crash_during_capture.js

Those are only on Windows debug, which we aren't enabling yet.

Attachment #9081718 - Attachment description: Bug 1569832 - Enable opt fission mochitests on mozilla-central (excluding gpu and remote) → Bug 1569832 - [ci] Enable fission mochitests on mozilla-central, r?jmaher
Attachment #9081726 - Attachment is obsolete: true

Here's my try push scheduling these tasks:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=de525b92b64adc6ea64af5c8f09921d885cc4ba8

A couple other things to note:

  1. This only enables it on central now (which is consistent with tier 2). I'd say we give it a week or so to settle and if it has been relatively stable, we can consider bumping it to tier 1 + running on integration branches.

  2. This enables fission tasks with the quantum render (qr) configuration. Just pointing it out in case this is unnecessary.

  3. This will be easy to tweak as we go. Just file a bug and needinfo me if you'd like any other adjustments down the road.

There are few issues in the try run:
https://treeherder.mozilla.org/#/jobs?repo=try&duplicate_jobs=visible&resultStatus=testfailed%2Cbusted%2Cexception%2Csuccess%2Crunning%2Cpending%2Crunnable&revision=de525b92b64adc6ea64af5c8f09921d885cc4ba8

  1. browser-screenshots is failing. I excluded this in the latest incarnation of the patch.
  2. dom/security/test/general/browser_test_referrer_loadInOtherProcess.js is permafailing across all platforms. Should be able to just disable this, can even add it as a parent revision in phabricator and land them together.
  3. There may be some new intermittents in the Linux qr platform. Should I skip QuantumRender on Linux for now? Or I guess we could leave them for now and let them go through the normal intermittent triage process.

Kris, would you mind adding a patch for 2? How do you want to proceed with 3?

Flags: needinfo?(kmaglione+bmo)

The attached patch skips the new intermittent failures in QR, so I think we should be fine enabling it.

Flags: needinfo?(kmaglione+bmo)

Looks like there's still at least one permafail and some intermittents:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=2630c3de18a5ef268f62e0a0ac92eea6890edf4f&duplicate_jobs=visible

But in the interest of not playing whack-a-mole forever I'll go ahead and land this anyway. Since it's tier-2 it's ok if it fails on central for a little bit, but we'll need to watch the retriggers and be ready to disable more tests.

See the try push above. Note the changes are in the autoland queue already, the tree has just been closed all morning.

Flags: needinfo?(kmaglione+bmo)

Hm. I'm a bit worried that if we only run on mozilla-central, it will be too hard to narrow the regressing changes if they happen too often. I suppose we can try for a while and see if regressions occur often enough for it to be a problem, though.

It should be possible to manually add the failing task to the first push in the range with Add New Jobs (or Add New Jobs (Search)). Then I think you can use the Backfill action to automatically trigger it on all tasks in-between.

It is a bit of a pain though.. maybe we should get it running on integration sooner rather than later.

Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c2b4a01f554b
Part 0 - Update crash annotations one last time. r=mccr8
https://hg.mozilla.org/integration/autoland/rev/20c6bad5230d
[ci] Enable fission mochitests on mozilla-central, r=jmaher
Pushed by maglione.k@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/1c22b590bc38
Part 2 - Skip another crashing test. r=mccr8
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla70
Flags: needinfo?(kmaglione+bmo)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: