ensure win7-ix jobs are not run on try for unittests which are running on AWS vm

RESOLVED WONTFIX

Status

Infrastructure & Operations
CIDuty
P5
normal
RESOLVED WONTFIX
2 years ago
a month ago

People

(Reporter: jmaher, Assigned: aselagea)

Tracking

Details

Attachments

(4 attachments)

(Reporter)

Description

2 years ago
I see instances of win7-ix machines on try running reftest, and reftest-unaccel.  we should ensure we reduce our load on the win7-ix machines as much as reasonably possible- and make sure jobs are equal between production and try.
(Reporter)

Comment 1

2 years ago
I see jittest and jsreftest as well

Updated

2 years ago
Blocks: 1303152

Updated

2 years ago
No longer blocks: 1303152
Component: General Automation → Buildduty
QA Contact: catlee → bugspam.Callek
(Assignee)

Updated

2 years ago
Assignee: nobody → aselagea
(Assignee)

Comment 2

2 years ago
Created attachment 8798381 [details]
win7_ix

Tests that are currently running on Windows 7 hardware machines.
(Assignee)

Comment 3

2 years ago
Created attachment 8798382 [details]
win7_VM

Tests that are currently running on Windows 7 VMs.
(Assignee)

Comment 4

2 years ago
It follows that we have 21 tests that are running both on HW and VMs (all of them on try):

Windows 7 32-bit try opt test mochitest-other
Windows 7 32-bit try opt test jsreftest
Windows 7 32-bit try opt test crashtest
Windows 7 32-bit try opt test jittest
Windows 7 32-bit try opt test reftest-no-accel
Windows 7 32-bit try opt test reftest
Windows 7 32-bit try opt test mochitest-media-e10s
Windows 7 32-bit try opt test reftest-e10s
Windows 7 32-bit try opt test mochitest-browser-screenshots
Windows 7 32-bit try opt test mochitest-clipboard
Windows 7 32-bit try opt test mochitest-clipboard-e10s
Windows 7 32-bit try debug test mochitest-other
Windows 7 32-bit try debug test jsreftest
Windows 7 32-bit try debug test crashtest
Windows 7 32-bit try debug test jittest
Windows 7 32-bit try debug test reftest-no-accel
Windows 7 32-bit try debug test reftest
Windows 7 32-bit try debug test reftest-e10s
Windows 7 32-bit try debug test mochitest-media-e10s
Windows 7 32-bit try debug test mochitest-clipboard
Windows 7 32-bit try debug test mochitest-clipboard-e10s
(Assignee)

Comment 5

2 years ago
Created attachment 8798383 [details] [diff] [review]
bug_1303006.patch

Patch to remove Win 7 HW try tests that also run on VMs.
Attachment #8798383 - Flags: review?(kmoir)
(Assignee)

Comment 6

2 years ago
Created attachment 8798384 [details]
diff.txt

Tested the patch on dev-master2 and worked without issues. Attached the tests diff file.
(Assignee)

Updated

2 years ago
Duplicate of this bug: 1308098
(Assignee)

Updated

2 years ago
Duplicate of this bug: 1308095

Updated

2 years ago
Attachment #8798383 - Flags: review?(kmoir) → review+
Clipboard is the exception in that list, because although it runs on both the jobs on VM do not work and are hidden (and are the reason the suite exists, because those tests don't work on VMs).
Oh, not the only exception. The four things we're still running on hardware on integration branches and mozilla-central, and that must still run on hardware on try as well, are mochitest-clipboard, mochitest-clipboard-e10s, mochitest-other and mochitest-media-e10s.
(Assignee)

Comment 11

2 years ago
Regarding those 4 tests, we have the following:
    - mochitest-clipboard, mochitest-clipboard-e10s: VM tests are generally failing, hidden in TH
    - mochitest-other: VM tests are mostly green, yet they are also hidden in TH
    - mochitest-media-e10s: VM tests are generally failing, but they are not hidden in TH

I also checked the rest of the VM tests and they all look fine, but some of them will *not* run by default (as their correspondent HW ones do) if 'Windows 7' is not explicitly specified as platform:
    - reftest-no-accel
    - reftest
    - reftest-e10s

@jmaher: any thoughts on this?
Flags: needinfo?(jmaher)
(Reporter)

Comment 12

2 years ago
the reason to keep jobs duplicated is for theoretical work to green them up.  ideally we will fix the issues with clipboard in firefox and the mochitest-clipboard* jobs will be green;  there are 2 tests which need attention in mochitest-other, and mochitest-media-e10s needs some attention for a few tests before moving it over.  I am fine with all of those tests hidden by default on try (we should make mochitest-media-e10s hidden by default for win7-vm on try only).

we really should have reftest* jobs running by default without requiring 'windows 7'.  thanks for bringing that up.
Flags: needinfo?(jmaher)
(Assignee)

Updated

2 years ago
Depends on: 1309812
(Assignee)

Comment 13

a year ago
(In reply to Alin Selagea [:aselagea][:buildduty] from comment #11)
> Regarding those 4 tests, we have the following:
>     - mochitest-clipboard, mochitest-clipboard-e10s: VM tests are generally
> failing, hidden in TH
>     - mochitest-other: VM tests are mostly green, yet they are also hidden
> in TH
>     - mochitest-media-e10s: VM tests are generally failing, but they are not
> hidden in TH

Update on this:
    - mochitest-clipboard, mochitest-clipboard-e10s: nothing changed - VM tests are generally failing, hidden in TH
    - mochitest-other: it was split in [a11y, chrome] on all platforms in bug 1338610
        - a11y: only running on VMs now (bug 1342202), not hidden in TH
        - chrome: VM tests are green, but hidden in TH 
    - mochitest-media-e10s: VM tests are green this time, not hidden in TH

Should the green ones run on AWS only?
Flags: needinfo?(jmaher)
(Reporter)

Comment 14

a year ago
all fully green tests should be AWS only- right now that is not mochitest-chrome, but it is:
mochitest-a11y, and from the comment mochitest-media-e10s.

I would like to ensure that mochitest-media-e10s has 50 retriggers with <=3 failures total, otherwise it might be too intermittent for running on AWS.
Flags: needinfo?(jmaher)
(Assignee)

Comment 15

a year ago
(In reply to Joel Maher ( :jmaher) from comment #14)

> I would like to ensure that mochitest-media-e10s has 50 retriggers with <=3
> failures total, otherwise it might be too intermittent for running on AWS.

I did a push to try to see how things go for multiple runs of mochitest-media-e10s on Windows 7 VMs:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4c861bc93933e51a6abe88eb9c4c26dd4868230d&exclusion_profile=false&filter-searchStr=Windows%207%20VM

As it can be noticed, there's a single failure for "Windows 7 VM opt" (out of 50 runs) and none for "Windows 7 VM debug" (again, out of 50 runs). That sounds like we're ready to stop running this test on hardware :-)
Found in triaging.
@aselagea: is there anything else to do here or can we close this?
Priority: -- → P5
(Assignee)

Comment 17

8 months ago
Most Windows 7 unittests are already in TC, so I don't think this is something to worry about anymore.
Status: NEW → RESOLVED
Last Resolved: 8 months ago
Resolution: --- → WONTFIX

Updated

a month ago
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.