1831552 - The Windows OS version of the NSS CI is outdated (2012)

Status: RESOLVED FIXED

(NSS :: Libraries, defect, P2)

Description

[Anna Weine]

The NSS CI currently runs on Windows 2012 (https://treeherder.mozilla.org/jobs?repo=nss-try). As we are currently reaching the End of Support for this OS (https://learn.microsoft.com/en-us/lifecycle/announcements/windows-server-2012-r2-end-of-support), we would like to update the Continuous Integration to a more recent version of the Windows OS.

Comment 1

[Julien Cristau [:jcristau]]

I started looking at this, switching to the win2022 pool that relops has set up for nss.

Currently hitting an issue where nspr's "make install" doesn't seem to deal well with windows path oddities:

nsinstall -t -m 644 ../../../../pr/include/md/_winnt.cfg /d/task_169832164303801/dist/Release/include/nspr
mv -f /d/task_169832164303801/dist/Release/include/nspr/_winnt.cfg /d/task_169832164303801/dist/Release/include/nspr/prcpucfg.h
mv: cannot stat '/d/task_169832164303801/dist/Release/include/nspr/_winnt.cfg': No such file or directory

It looks like mv expects D:\task_...\dist\Release\include\... type paths instead of /d/task_.../dist/... (unlike nsinstall).

Try push: https://treeherder.mozilla.org/jobs?repo=nss-try&revision=1c2381ee291fc474127e97217fe3dd02979e2d99

Comment 2

[Benjamin Beurdouche [:beurdouche]]

Julien, are you running the gyp build script from NSS to build NSPR?
build.sh does build NSPR on linux/mac automatically, if there is WSL on windows, maybe using it to run build.sh will work ?

FYI to Kai about the NSPR part...

Comment 3

[Julien Cristau [:jcristau]]

(In reply to Benjamin Beurdouche [:beurdouche] from comment #2)

Julien, are you running the gyp build script from NSS to build NSPR?

I'm runing what your automation is set up to run (automation/taskcluster/windows/build_gyp.sh).

Comment 4

[Lucas Mülling [:lmulling]]

It looks like mv expects D:\task_...\dist\Release\include... type paths instead of /d/task_.../dist/...
I have the same problem, though, it seems nsinstall is installing under /c/c/, the correct dist folder has nothing for me.

Comment 5

[Julien Cristau [:jcristau]]

Thanks lmulling. It's possible the same is happening for me and I just mis-read the log.

Comment 6

[Kai Engert [:KaiE:]]

I'm not a good person to help with the build system issues.

Comment 7

[Lars Eggert [:lars]]

In my case, the error '.../_winnt.cfg': No such file or directory was due to mixing nsinstall from https://ftp.mozilla.org/pub/mozilla/libraries/win32/moztools-static.zip with the MSYS2 environment the Windows GitHub CI runner is otherwise using. Installing nsinstall via the MSYS2 pacman tool resolved this.

I'm still puzzled why the moztools version of nsinstall would fail to put the files in place without error...

Comment 8

[Lars Eggert [:lars]]

I'll also note that on other platforms (at least Linux and MacOS), nsinstall is built as part of NSPR so no separate install is required. But it's not built on Windows. Building nsinstall as part of NSPR on Windows would bypass the issue, too.

Comment 9

[Julien Cristau [:jcristau]]

Presumably we could replace nsinstall.exe in https://hg.mozilla.org/mozilla-build/ with a version that works with msys2. Alex, is this something you can help with?

Comment 10

[Alex Hochheiden [:ahochheiden] (PTO until Feb 3)]

Is there any process/docs for updating moztools-static.zip? At first I thought it was part of https://ftp.mozilla.org/pub/mozilla/libraries/win32/src/, but it's just sitting at the top level, and I don't think I've ever interacted with it. Looking in my current mozilla-build-stage, it doesn't appear to be generated when creating a MozillaBuild installer either.

Either way, I should be able to do that once I get my access to gcloud sorted. It appears I have general access, but not to the specific mozilla-build directory.

What version of nsinstall.exe do you want there?

Comment 11

[Benjamin Beurdouche [:beurdouche]]

That would be my guess: https://packages.msys2.org/package/nsinstall

Comment 12

[Julien Cristau [:jcristau]]

I don't know anything about moztools-static.zip, I was asking about MozillaBuild.

As an alternative :glandium pointed me at https://searchfox.org/mozilla-central/source/config/nsinstall.py, we could use something like that for nss instead of the MozillaBuild version.

Comment 13

[Alex Hochheiden [:ahochheiden] (PTO until Feb 3)]

That's doable. I'll update the nsinstall.exe that's bundled with MozillaBuild to the linked version and it'll be included in the next release.

I did have release in the works, but I can start over and include that and some other stuff now that I have access to write files to the MozillaBuild FTP again.

I can probably have the release ready to go within a week, but I'm hesitant to push it out the door then since I'll be on Holiday for a week afterwards. I'd say 2-3 weeks from now is when the new MozillaBuild (that has the updated nsinstaller.exe that you need) will be released.

Comment 14

[Alex Hochheiden [:ahochheiden] (PTO until Feb 3)]

Attachment: Bug 1831552 - Update `nsinstall.exe` to version `4.35-1` r?#build

https://packages.msys2.org/package/nsinstall

Comment 15

[Alex Hochheiden [:ahochheiden] (PTO until Feb 3)]

Current pre-release version: https://ftp.mozilla.org/pub/mozilla/libraries/win32/MozillaBuildSetup-4.1pre.exe

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-build/rev/c97f43ffa08d
Update nsinstall.exe to version 4.35-1 r=firefox-build-system-reviewers,glandium

Comment 17

[Alex Hochheiden [:ahochheiden] (PTO until Feb 3)]

Can this be resolved now that MozillaBuild 4.1 has been released?

Comment 18

[Julien Cristau [:jcristau]]

This bug is about updating the NSS windows workers in CI. A new MozillaBuild was probably a prerequisite, but there's more to this.

Comment 19

[:jmoss]

The worker pool nss-1/b-win2022-alpha is ready to test.

Comment 20

[Julien Cristau [:jcristau]]

Thanks Jonathan, this is looking promising. I've filed a couple of bugs to track issues found so far.

Comment 21

[Julien Cristau [:jcristau]]

Attachment: Bug 1831552 - ci: update windows workers to win2022

• switch to win2022 pool
• stop overriding PATH, the value set on the workers is fine (and ours would
  need updating e.g. for the switch to msys2)
• use venv module instead of virtualenv (now part of stdlib)
• stop using build/tools for tooltool_wrapper; point openssl at the
  certifi cert bundle
• clone gyp instead of using the old version in tooltool
• pip install six, required by gyp
• use mozmake.exe instead of make (which isn't there)
• prepend VS paths to $PATH instead of appending so link.exe is the
  linker and not GNU link

Comment 22

[Julien Cristau [:jcristau]]

Attachment: Bug 1831552 - ci: update windows task names and labels for switch to win2022

Comment 23

[Julien Cristau [:jcristau]]

Attachment: Bug 1831552 - ci: use mozmake instead of make

make is not available on the win2022 workers.

Comment 24

[Julien Cristau [:jcristau]]

Attachment: Bug 1831552 - ci: let the windows setup script work without $m

The $m variable is set by the build_gyp.sh and build.sh scripts, but not
by gen_certs.sh and run_tests.sh, which also source setup.sh. On the
other hand, they don't appear to need virtual studio environment
variables.

For some reason this didn't use to fail on the win2012 workers, even
though the script uses set -e.

Comment 25

[Julien Cristau [:jcristau]]

Attachment: Bug 1831552 - strip trailing carriage returns in tools tests

pp's output has windows (CRLF) line endings, but msys2's bash expects
unix (LF), which leads to mismatches when comparing parameters.

Comment 26

[Julien Cristau [:jcristau]]

ci: let the windows setup script work without $m r=nss-reviewers,nkulatova
https://hg.mozilla.org/projects/nss/rev/366d592017074260365b4bcd7966effa858cbea4
strip trailing carriage returns in tools tests r=nss-reviewers,nkulatova
https://hg.mozilla.org/projects/nss/rev/c6021535537aa1be81f39022f38a51d4cc25777c
ci: update windows workers to win2022 r=nss-reviewers,nkulatova
https://hg.mozilla.org/projects/nss/rev/6b240bfa4108cd79935b2863b6de51e1c6fc90d2