1727943 - Migrate remaining Windows 10 x64 CCov suites from AWS to Azure

Status: RESOLVED FIXED

(Testing :: General, task)

Comment 1

[Michelle Goossens [:masterwayz]]

Attachment: Bug 1727943 - Part 1: Migrate Windows 10 ccov from AWS to Azure r=jmaher

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8339eb731281
Part 1: Migrate Windows 10 ccov from AWS to Azure r=jmaher

Comment 3

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/8339eb731281

Comment 4

[Michelle Goossens [:masterwayz]]

Attachment: Bug 1727943 - Part 2: Migrate mochitest, reftest and crashtest Windows 10 ccov suites from AWS to Azure r=jmaher

Comment 5

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr91/rev/fd97421f88c4

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9f89e5ae8b4b
Part 2: Migrate mochitest, reftest and crashtest Windows 10 ccov suites from AWS to Azure r=jmaher

Comment 7

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/9f89e5ae8b4b

Comment 8

[Michelle Goossens [:masterwayz]]

Attachment: Bug 1727943 - Part 3: Migrate wpt Windows 10 CCov suites from AWS to Azure r=jmaher

Comment 9

[Michelle Goossens [:masterwayz]]

As requested in Matrix, NI'ing :marco.
You can see the "blocking" meta bug for more info and also Bug 1718290.

Comment 10

[Michelle Goossens [:masterwayz]]

Attachment: Bug 1727943 - Part 4: Migrate jittest and xpcshell Windows 10 CCov suites from AWS to Azure r=jmaher

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cc2c8ad78660
Part 3: Migrate wpt Windows 10 CCov suites from AWS to Azure r=jmaher,webdriver-reviewers,whimboo
https://hg.mozilla.org/integration/autoland/rev/77db0112e100
Part 4: Migrate jittest and xpcshell Windows 10 CCov suites from AWS to Azure r=jmaher

Comment 12

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/cc2c8ad78660
https://hg.mozilla.org/mozilla-central/rev/77db0112e100

Comment 13

[Marco Castelluccio [:marco]]

Joel, Michelle, I see jittest and xpcshell tests are no longer running under coverage on Windows.
Looking at https://hg.mozilla.org/mozilla-central/rev/77db0112e100, it seems they were disabled.
Why were they disabled? Can we revert that patch and keep them running on AWS for now until we figure out a solution?

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

As part of a migration, we need to not be on the hook to investigate everything and work to migrate things in a timely fashion, hence we filed this bug. Often there are edge cases which can take weeks of time (we already spent almost 2 weeks on ccov already longer than planned). With that said, I know Mark has a try push with the jit/ccov running on an upgraded disk instance:
https://treeherder.mozilla.org/jobs?repo=try&revision=3c1d45b320cfff3c7580d696587b5dd2980b0e81

this looks production ready and we should have the coverage soon.

Comment 15

[Marco Castelluccio [:marco]]

I understand the need for a timely resolution, but I was only contacted about this on Thursday night, and the suite disablings landed on Friday night, earlier than 24 hours later.

Next time, if you hit a code coverage-related problem, could you contact me or Calixte earlier so we can help you from the beginning? Our diagnosis was pretty quick, so we could have spared you the 2 weeks of investigation.

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

I didn't know who to talk to, I had to ask around for a few days after things were looking problematic. We need a clear ownership map for build variants. The failures were not showing up in code coverage code, they were showing up in test suites timing out, so we didn't think it was really code coverage related- I am refining this process and in the future all stakeholders who will have jobs they care about migrating will be aware of a migration affecting them (i.e. you and calixte would have known we were migrating windows and a rough timeline on when we would start working with a bug filed so you both would know when the work was starting).

Thanks for the quick diagnosis, I am excited to see this all up and running this week.

Comment 17

[Marco Castelluccio [:marco]]

(In reply to Joel Maher ( :jmaher ) (UTC -0800) from comment #16)

I didn't know who to talk to, I had to ask around for a few days after things were looking problematic. We need a clear ownership map for build variants. The failures were not showing up in code coverage code, they were showing up in test suites timing out, so we didn't think it was really code coverage related- I am refining this process and in the future all stakeholders who will have jobs they care about migrating will be aware of a migration affecting them (i.e. you and calixte would have known we were migrating windows and a rough timeline on when we would start working with a bug filed so you both would know when the work was starting).

This sounds like a great idea in general. If jobs weren't all defined together in the same files we could piggyback on the module system Zeid is working on.

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1727943 - run win10 ccov on -ssd instances. r=masterwayz!

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d5597b7c7a5b
run win10 ccov on -ssd instances. r=MasterWayZ

Comment 20

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/d5597b7c7a5b

Comment 21

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1727943 - turn on mochitest* tests that were skipped on win10-ccov and now pass. r=masterwayz!

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e93f80ea6d58
turn on mochitest* tests that were skipped on win10-ccov and now pass. r=MasterWayZ

Comment 23

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/e93f80ea6d58

Comment 24

[Marco Castelluccio [:marco]]

https://hg.mozilla.org/mozilla-central/rev/8339eb731281 seems to have disabled a few suites on CCOV (we discovered it in bug 1605650), was it intentional?

Comment 25

[Joel Maher ( :jmaher ) (UTC -8)]

it wasn't well documented if it was intentional.

I see these missing:
test-coverage-wpt
desktop-screenshot-capture
firefox-ui-functional-local
firefox-ui-functional-remote
mochitest-a11y
mochitest-chrome
mochitest-chrome-gpu
mochitest-media
mochitest-plain-gpu
mochitest-webgpu
mochitest-webgl1-core
mochitest-webgl1-ext
mochitest-webgl2-core
mochitest-webgl2-ext
reftest
telemetry-tests-client
test-verify-wpt

:masterwayz, want to give those test suites a try on win10-ccov?

Comment 26

[Michelle Goossens [:masterwayz]]

Whoops, that was not intentional!
Try has push been made, it may run more jobs than it should though, will contact you through Matrix once it is up again.

Comment 27

[Michelle Goossens [:masterwayz]]

Attachment: Bug 1727943 - Re-enable Windows 10 x64 2004 CCov tests that were forgotten r=jmaher

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5324daf54f51
Re-enable Windows 10 x64 2004 CCov tests that were forgotten r=jmaher

Comment 29

[Michelle Goossens [:masterwayz]]

Attachment: Bug 1727943 - Fix mistake in manifest edit r=jmaher

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8030fda0796e
Fix mistake in manifest edit r=jmaher a=fix CLOSED TREE

Comment 31

[Michelle Goossens [:masterwayz]]

Attachment: Bug 1727943 - Fix refest failure on Windows

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/77c6e311075f
Fix refest failure on Windows

Comment 33

[Michelle Goossens [:masterwayz]]

Attachment: Bug 1727943 - Fix another refest failure on Windows

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d00a942b7b05
Fix another refest failure on Windows

Comment 35

[Cristina Cozmuta (:CrissCozmuta)]

https://hg.mozilla.org/mozilla-central/rev/5324daf54f51
https://hg.mozilla.org/mozilla-central/rev/8030fda0796e
https://hg.mozilla.org/mozilla-central/rev/77c6e311075f
https://hg.mozilla.org/mozilla-central/rev/d00a942b7b05