Closed Bug 1303006 Opened 3 years ago Closed 2 years ago

ensure win7-ix jobs are not run on try for unittests which are running on AWS vm

Categories

(Infrastructure & Operations :: CIDuty, task, P5)

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: jmaher, Assigned: aselagea)

References

Details

Attachments

(4 files)

I see instances of win7-ix machines on try running reftest, and reftest-unaccel.  we should ensure we reduce our load on the win7-ix machines as much as reasonably possible- and make sure jobs are equal between production and try.
I see jittest and jsreftest as well
Blocks: 1303152
No longer blocks: 1303152
Component: General Automation → Buildduty
QA Contact: catlee → bugspam.Callek
Assignee: nobody → aselagea
Attached file win7_ix
Tests that are currently running on Windows 7 hardware machines.
Attached file win7_VM
Tests that are currently running on Windows 7 VMs.
It follows that we have 21 tests that are running both on HW and VMs (all of them on try):

Windows 7 32-bit try opt test mochitest-other
Windows 7 32-bit try opt test jsreftest
Windows 7 32-bit try opt test crashtest
Windows 7 32-bit try opt test jittest
Windows 7 32-bit try opt test reftest-no-accel
Windows 7 32-bit try opt test reftest
Windows 7 32-bit try opt test mochitest-media-e10s
Windows 7 32-bit try opt test reftest-e10s
Windows 7 32-bit try opt test mochitest-browser-screenshots
Windows 7 32-bit try opt test mochitest-clipboard
Windows 7 32-bit try opt test mochitest-clipboard-e10s
Windows 7 32-bit try debug test mochitest-other
Windows 7 32-bit try debug test jsreftest
Windows 7 32-bit try debug test crashtest
Windows 7 32-bit try debug test jittest
Windows 7 32-bit try debug test reftest-no-accel
Windows 7 32-bit try debug test reftest
Windows 7 32-bit try debug test reftest-e10s
Windows 7 32-bit try debug test mochitest-media-e10s
Windows 7 32-bit try debug test mochitest-clipboard
Windows 7 32-bit try debug test mochitest-clipboard-e10s
Patch to remove Win 7 HW try tests that also run on VMs.
Attachment #8798383 - Flags: review?(kmoir)
Attached file diff.txt
Tested the patch on dev-master2 and worked without issues. Attached the tests diff file.
Duplicate of this bug: 1308098
Duplicate of this bug: 1308095
Attachment #8798383 - Flags: review?(kmoir) → review+
Clipboard is the exception in that list, because although it runs on both the jobs on VM do not work and are hidden (and are the reason the suite exists, because those tests don't work on VMs).
Oh, not the only exception. The four things we're still running on hardware on integration branches and mozilla-central, and that must still run on hardware on try as well, are mochitest-clipboard, mochitest-clipboard-e10s, mochitest-other and mochitest-media-e10s.
Regarding those 4 tests, we have the following:
    - mochitest-clipboard, mochitest-clipboard-e10s: VM tests are generally failing, hidden in TH
    - mochitest-other: VM tests are mostly green, yet they are also hidden in TH
    - mochitest-media-e10s: VM tests are generally failing, but they are not hidden in TH

I also checked the rest of the VM tests and they all look fine, but some of them will *not* run by default (as their correspondent HW ones do) if 'Windows 7' is not explicitly specified as platform:
    - reftest-no-accel
    - reftest
    - reftest-e10s

@jmaher: any thoughts on this?
Flags: needinfo?(jmaher)
the reason to keep jobs duplicated is for theoretical work to green them up.  ideally we will fix the issues with clipboard in firefox and the mochitest-clipboard* jobs will be green;  there are 2 tests which need attention in mochitest-other, and mochitest-media-e10s needs some attention for a few tests before moving it over.  I am fine with all of those tests hidden by default on try (we should make mochitest-media-e10s hidden by default for win7-vm on try only).

we really should have reftest* jobs running by default without requiring 'windows 7'.  thanks for bringing that up.
Flags: needinfo?(jmaher)
Depends on: 1309812
(In reply to Alin Selagea [:aselagea][:buildduty] from comment #11)
> Regarding those 4 tests, we have the following:
>     - mochitest-clipboard, mochitest-clipboard-e10s: VM tests are generally
> failing, hidden in TH
>     - mochitest-other: VM tests are mostly green, yet they are also hidden
> in TH
>     - mochitest-media-e10s: VM tests are generally failing, but they are not
> hidden in TH

Update on this:
    - mochitest-clipboard, mochitest-clipboard-e10s: nothing changed - VM tests are generally failing, hidden in TH
    - mochitest-other: it was split in [a11y, chrome] on all platforms in bug 1338610
        - a11y: only running on VMs now (bug 1342202), not hidden in TH
        - chrome: VM tests are green, but hidden in TH 
    - mochitest-media-e10s: VM tests are green this time, not hidden in TH

Should the green ones run on AWS only?
Flags: needinfo?(jmaher)
all fully green tests should be AWS only- right now that is not mochitest-chrome, but it is:
mochitest-a11y, and from the comment mochitest-media-e10s.

I would like to ensure that mochitest-media-e10s has 50 retriggers with <=3 failures total, otherwise it might be too intermittent for running on AWS.
Flags: needinfo?(jmaher)
(In reply to Joel Maher ( :jmaher) from comment #14)

> I would like to ensure that mochitest-media-e10s has 50 retriggers with <=3
> failures total, otherwise it might be too intermittent for running on AWS.

I did a push to try to see how things go for multiple runs of mochitest-media-e10s on Windows 7 VMs:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4c861bc93933e51a6abe88eb9c4c26dd4868230d&exclusion_profile=false&filter-searchStr=Windows%207%20VM

As it can be noticed, there's a single failure for "Windows 7 VM opt" (out of 50 runs) and none for "Windows 7 VM debug" (again, out of 50 runs). That sounds like we're ready to stop running this test on hardware :-)
Found in triaging.
@aselagea: is there anything else to do here or can we close this?
Priority: -- → P5
Most Windows 7 unittests are already in TC, so I don't think this is something to worry about anymore.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → WONTFIX
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.