1806073 - Migrate Windows 2012 workloads to Azure

Status: RESOLVED FIXED

(Testing :: General, task)

Comment 1

[Michelle Goossens [:masterwayz]]

Attachment: Bug 1806073 - Migrate Windows 2012 workloads to Azure r=ahal!,jmaher!

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/00838fbbc755
Migrate Windows 2012 workloads to Azure r=ahal,jmaher

Comment 3

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/00838fbbc755

Comment 4

[Natalia Csoregi [:nataliaCs]]

Backed out for causing clang bustage.

• backout: https://hg.mozilla.org/integration/autoland/rev/857b1d139d115fa38ae78dd4d024a053dfd374ad
• push: https://treeherder.mozilla.org/jobs?repo=autoland&revision=00838fbbc755c9c520bfe58154059f8e20fe337d&searchStr=windows%2Ctoolchains&selectedTaskRun=f6Yw_pjrQ-Wysypp5TsW1w.0
• failure log: https://treeherder.mozilla.org/logviewer?job_id=400183311&repo=autoland&lineNumber=1493

Comment 5

[Mike Hommey [:glandium]]

Relevant part of the log: https://treeherder.mozilla.org/logviewer?job_id=400183311&repo=autoland&lineNumber=658

Also, mozmake bustage:
https://firefoxci.taskcluster-artifacts.net/CJPzdo4FQ22bMWtzTX-0jg/0/public/logs/live_backing.log

Comment 6

[Sandor Molnar[:smolnar]]

Backout merged to central : https://hg.mozilla.org/mozilla-central/rev/00838fbbc755

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

:glandium, I see you added the clang-15 toolchain job, not sure if you are the owner/expert of that job- if you are, can you fix it? If not, can you redirect to someone who can?

Comment 8

[Michelle Goossens [:masterwayz]]

I think we missed those jobs during our initial testing.

Comment 9

[Mike Hommey [:glandium]]

See the log, for some reason find resolves to the DOS find, not the POSIX find when running shell scripts. POSIX find is probably missing, which is unexpected. For mozmake, my wild guess is that there used to be an install of MSVC and now there isn't. I'm not sure whether we'd want to add it just for that one task, so we should probably change it to build differently.

Comment 10

[Mark Cornmesser [:markco] OOO 2024/04/15]

This is PATH issue. find is being resolved to C:\Windows\system32\find and not C:\mozilla-build\msys2\usr\bin\find because C:\Windows\system32 is the PATH variable before the msys2. I verified on an AWS 2012 worker the PATH has msys before syetm32.

Because of how the Azure images get built I am hesitant on changing the PATH, so I am going to remove C:\Windows\system32\find from the image.

Comment 11

[Mike Hommey [:glandium]]

See bug 1806634 for yet another regression caused by this change. It seems there is a version of Visual Studio installed, but it's more recent than the one in the AWS image, and causes problems. I'd rather not have VS installed at all in the image and deal with the fallout from that, than figure out what individual things are subtly or less subtly broken by a new version of VS being in the image.

Comment 12

[Mike Hommey [:glandium]]

For mozmake, nothing from the build_w32.bat script runs at all. I modified build-mozmake.sh to replace all occurrences of echo off by echo on in build_w32.bat, and that shows that nothing happens when the script is run on azure, while it shows all the commands executed in the build_w32.bat script when run on AWS.

Comment 13

[Mark Cornmesser [:markco] OOO 2024/04/15]

(In reply to Mike Hommey [:glandium] from comment #11)

See bug 1806634 for yet another regression caused by this change. It seems there is a version of Visual Studio installed, but it's more recent than the one in the AWS image, and causes problems. I'd rather not have VS installed at all in the image and deal with the fallout from that, than figure out what individual things are subtly or less subtly broken by a new version of VS being in the image.

I am building an image test without VS installed. It might be tricky not to have it because there maybe be some dependencies there for the newer Python. However, looking into an AWS worker, it has a very limited installation of Microsoft Studio that only has an one directory C:\Program Files (x86)\Microsoft Visual Studio 14.0\XML\Schemas\1033 . In that directory there are 2 .xml files and a .xsd file. Hopefully those are no longer needed and there are no Python dependencies missing, so we can drop this VS install.

(In reply to Mike Hommey [:glandium] from comment #12)

For mozmake, nothing from the build_w32.bat script runs at all. I modified build-mozmake.sh to replace all occurrences of echo off by echo on in build_w32.bat, and that shows that nothing happens when the script is run on azure, while it shows all the commands executed in the build_w32.bat script when run on AWS.

Where can I find the build_w32.bat script ? I was unable to find it in searchfox and it is not on the AWS worker.

Comment 14

[Mark Cornmesser [:markco] OOO 2024/04/15]

The first go without VS Python packages like psutil failed to install. The error message includes:
istutils.errors.DistutilsPlatformError: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/

Which is the package we are installing. It's not a full VS install, It's a stand alone compiler with library and scripts. I am pretty sure that the Python packages are requiring the libraries that come with it. I was able to find a 2015 version of it. Next step will be to test the image build with it and see if the Python packages install.

Currently it is unclear how this was working on the AWS workers.

Comment 15

[Mike Hommey [:glandium]]

Where can I find the build_w32.bat script ? I was unable to find it in searchfox and it is not on the AWS worker.

It's in the artifact of the fetch-mozmake dependency.

Comment 16

[Mike Hommey [:glandium]]

(In reply to Mark Cornmesser [:markco] (PTO till Dec 5th) from comment #14)

The first go without VS Python packages like psutil failed to install. The error message includes:
istutils.errors.DistutilsPlatformError: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/

The latest version of psutil has wheels, build tools shouldn't be needed for it.

Comment 17

[Mark Cornmesser [:markco] OOO 2024/04/15]

The latest version of psutil has wheels, build tools shouldn't be needed for it.

Oh, nice. Hopefully that is the case for other packages that were dependent on build tools. I will test this afternoon.

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

the build_win32.bat can be found in make.tar.zst which is an artifact here: https://firefox-ci-tc.services.mozilla.com/tasks/TVmhNWTGSXqm-JzHgNm8wg

Comment 19

[Mark Cornmesser [:markco] OOO 2024/04/15]

Went with the latest psutil and zstandard and the image successful built with out the build tools. I am still trying to figure out how to deal with the incorrect find being resolved from the PATH. I will continue to look into that and the build_win32.bat tomorrow.

Comment 20

[Mike Hommey [:glandium]]

(In reply to Mark Cornmesser [:markco] from comment #19)

Went with the latest psutil and zstandard and the image successful built with out the build tools. I am still trying to figure out how to deal with the incorrect find being resolved from the PATH. I will continue to look into that and the build_win32.bat tomorrow.

Does that mean that workers are up-to-date with this new image yet?

Comment 21

[Mike Hommey [:glandium]]

Bug 1806959 will work around the mozmake problem.

Comment 22

[Mike Hommey [:glandium]]

And after bug 1806960, there will only be a limited number of toolchain tasks running on Windows workers (clang*, gn).

Comment 23

[Mark Cornmesser [:markco] OOO 2024/04/15]

I have a fix for the find issue.

Currently the gecko-1/b-win2012-azure-alpha worker pool has the find fix and do not have the VS build tools installed. If it tests out well, I will update the Azure production pools.

glandium, jmaher: What steps should we take to validate these workers before kicking off the migration?

Comment 24

[Mike Hommey [:glandium]]

I did a try push triggering all the tasks that run on b-win2012 workers except updatebot-cron-win, which requires permissions try doesn't have, using your pool: https://treeherder.mozilla.org/jobs?repo=try&revision=470232c159f739a6b0c787d78d3be66a11f2ce11
mozmake surprisingly worked fine, as well as the others... except:

• condprof, broken because multidict 4.7 doesn't have wheels for python 3.9 (it has for 3.7 and 3.8). More recent versions of multidict do have wheels for python 3.9. It's not a direct requirement, but a requirement from aiohttp 3.5.4. aiohttp 3.7 (from 2 years ago) allows to use multidict 5.0.
• openh264 broken because make is not installed in the msys environment apparently. It might be worth switching those to use mozmake.

Comment 25

[Mike Hommey [:glandium]]

Bug 1807203 fixes the latter.

Comment 26

[Joel Maher ( :jmaher ) (UTC -8)]

I could get the condprof jobs running by installing:

                python3.exe -m pip install pyyaml &&
                python3.exe -m pip install arsenic==19.1 &&
                python3.exe -m pip install mozversion &&
                python3.exe -m pip install mozprofile &&
                python3.exe -m pip install mozdevice &&
                python3.exe -m pip install requests==2.22.0

Comment 27

[Joel Maher ( :jmaher ) (UTC -8)]

we need the specific version of arsenic and requests (see: https://searchfox.org/mozilla-central/source/testing/condprofile/requirements/base.txt ), I assume we could install all of these on the system beforehand and just call it good, or install some of the testing/condprofile/requirements/* files. that seems like the right thing todo, but more work.

Comment 28

[Mike Hommey [:glandium]]

condprof itself is installing the requirements from some of testing/condprofile/requirements/*.txt per testing/condprofile/condprof/check_install.py, which is called during condprof startup in testing/condprofile/confprof/main.py. So all you really need is to adjust the right requirements files.

Comment 29

[Joel Maher ( :jmaher ) (UTC -8)]

for some reason the ccov builds are failing, specifically unable to find the grcov tool:
https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=2414dde0ac071bcace94c41bb9e3df64dd3cb923

everything else seems good- not sure if I am missing something like a specific update, etc.

Comment 30

[Joel Maher ( :jmaher ) (UTC -8)]

oh that has been broken since Dec 29th (bug 1807908):
https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=ccov%2Cwin

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f8b651d2acf9
Migrate Windows 2012 workloads to Azure r=ahal,jmaher

Comment 32

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/f8b651d2acf9

Comment 33

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr102/rev/4113506d856cf7a5bd8c87573d73a6945e3cffb8
https://hg.mozilla.org/releases/mozilla-esr102/rev/1f102a72f2f4fa6d01a68b4a182b0b875ea970c7