1434943 - Unable to build with mozilla-build on Windows 10 x64

Status: RESOLVED FIXED

(NSS :: Build, defect, P1)

Description

[afrowuk]

Attachment: config.log

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0
Build ID: 20180128191252

Steps to reproduce:

Windows 10 x64
> Installed Visual Studio 2015 Community Edition.
> Installed latest mozilla-build (3.1.1) to default install directory (C:\mozilla-build)
> Downloaded NSS/NSPR source (3.35) to C:\src
> Ran C:\mozilla-build\start-shell.bat
> cd c:/src/nss
> make nss_build_all


Actual results:

Initially I was getting "configure: error: C compiler cannot create executables" because it couldn't find the location of cl.exe. To fix this I had to add both "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin" followed by "C:\Program Files (x86)\Windows Kits\10\bin\x64" (for mt.exe) to my PATH environment variable. There is nothing about having to do this in the build documentation.

Now I'm getting the following:
c:/src/nspr/WIN954.0_OPT.OBJ/config/../../config/now.c(6): fatal error C1034: stdio.h: no include path set
make[2]: *** [now.obj] Error 2
make[2]: Leaving directory `/c/src/nspr/WIN954.0_OPT.OBJ/config'
make[1]: *** [export] Error 2
make[1]: Leaving directory `/c/src/nspr/WIN954.0_OPT.OBJ'
make: *** [build_nspr] Error 2

I have tried running vsvars32.bat first and confirmed that INCLUDE and LIB are set up correctly.

I have attached C:\src\nspr\WIN954.0_OPT.OBJ\config.log

I also tried with Visual Studio 2017 (Community Edition and Professional) with the same result.


Expected results:

Build was successful!

Comment 1

[nankezhou]

The same happens with NNS programs.
Computer configuration：  WIN10 * 64 WINNT

../../config/now.c(6) : fatal error C1034: stdio.h: 不包括路径集
make[2]: *** [now.obj] Error 2
make[2]: Leaving directory `/c/m/nsprpub/WINNT6.2_x86_64_64_DBG.OBJD/config'
make[1]: *** [export] Error 2
make[1]: Leaving directory `/c/m/nsprpub/WINNT6.2_x86_64_64_DBG.OBJD'
make: *** [build_nspr] Error 2

Comment 2

[nankezhou]

My situation is exactly the same with you.
Computer configuration WIN10 * 64 / winnt

nsprpub/WINNT6.2_x86_64_64_DBG.OBJD/config/../../config/now.c(6) : fatal error C1034: stdio.h:

Comment 3

[Xidorn Quan [:xidorn] UTC+10]

Yeah, it seems it is pretty hard to get NSS build on Windows nowadays...

Comment 4

[Martin Thomson [:mt:]]

We don't need Bug 1439442 if we are willing to limit ourselves to MSVC 15.2 installs (that's VS 2017).

https://github.com/Microsoft/vswhere/wiki/Find-VC provides a useful command for finding the VS installation.  From there there's a bit of tail chasing involved to get things together (see https://blogs.msdn.microsoft.com/vcblog/2017/03/06/finding-the-visual-c-compiler-tools-in-visual-studio-2017/).  Why vswhere can't produce the default include path or binary directory for a given host architecture is beyond me.  Maybe it can, but the documentation is pretty poor.

Comment 5

[J.C. Jones [:jcj] (he/they)]

I'll work on this patch in the next couple of weeks.

Comment 6

[Mike]

Is a patch still in the works?  3.36.1 was released today and this bug was not listed in the release notes.  I don't believe there is a workaround for this bug, so I'm stuck until this gets fixed.

Comment 7

[rommelot]

Good day. 

Any update on this issue? 
I am having the same problem also. 

I cannot build NSS 3.35 using the C Compiler that came with Visual Studio 2017. I Tried MingW which has the gcc Compiler, but i also get the C Compiler error. I have added the compiler PATH to the environment variables. but it still did not work. 


PATH: /local/bin
PATH: /c/mozilla-build/bin
PATH: /c/mozilla-build/kdiff3
PATH: /c/mozilla-build/node-v8.11.1-win-x64
PATH: /c/mozilla-build/nsis-3.01
PATH: /c/mozilla-build/python
PATH: /c/mozilla-build/python/Scripts
PATH: /c/mozilla-build/python3
PATH: /c/mozilla-build/python3/Scripts
PATH: .
PATH: /usr/local/bin
PATH: /mingw/bin
PATH: /bin
PATH: /c/WINDOWS/System32
PATH: /c/WINDOWS
PATH: /c/WINDOWS/System32/Wbem
PATH: /c/Users/ROMMELDVO/.cargo/bin


## ----------- ##
## Core tests. ##
## ----------- ##

configure:2383: checking build system type
configure:2397: result: i686-pc-mingw32
configure:2417: checking host system type
configure:2430: result: i686-pc-mingw32
configure:2450: checking target system type
configure:2463: result: i686-pc-mingw32
configure:2543: checking for cl
configure:2570: result: gcc
configure:3054: checking for whoami
configure:3072: found /c/WINDOWS/System32/whoami
configure:3085: result: /c/WINDOWS/System32/whoami
configure:3481: checking for C compiler version
configure:3490: gcc --version >&5
../configure: line 3492: gcc: command not found
configure:3501: $? = 127
configure:3490: gcc -v >&5
../configure: line 3492: gcc: command not found
configure:3501: $? = 127
configure:3490: gcc -V >&5
../configure: line 3492: gcc: command not found
configure:3501: $? = 127
configure:3490: gcc -qversion >&5
../configure: line 3492: gcc: command not found
configure:3501: $? = 127
configure:3521: checking whether the C compiler works
configure:3543: gcc    conftest.c  >&5
../configure: line 3545: gcc: command not found
configure:3547: $? = 127
configure:3585: result: no
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME ""
| #define PACKAGE_TARNAME ""
| #define PACKAGE_VERSION ""
| #define PACKAGE_STRING ""
| #define PACKAGE_BUGREPORT ""
| #define PACKAGE_URL ""
| #define DEBUG 1
| /* end confdefs.h.  */

I am now stuck on this problem, as i need to use CertUtil to programmatically install our custom Certificate into Firefox. 
Please advise. 

Thank you,

Comment 8

[Franziskus Kiefer [:franziskus]]

There's no good way to do this right now. I recommend setting the VS path manually. I use this [1] for VS2017. This works fine with the gyp build (you need to have ninja and gyp in your PATH as well).

[1] https://gist.github.com/franziskuskiefer/d73426ab875dec991328c812d5965278

Comment 9

[Xidorn Quan [:xidorn] UTC+10]

MozillaBuild now includes vswhere, so it should be possible for the build system to query a path for it.

Comment 10

[Martin Thomson [:mt:]]

Moving to vswhere is definitely planned, but I don't know its priority.

Comment 12

[Martin Thomson [:mt:]]

Attachment: windows-msvc-auto.patch

Hey JC, I know that you said you would do this, but I had to build with MSVC today (well I didn't in the end, but I hope it wasn't wasted effort), and I decided that I didn't want to just use the workaround that Franziskus uses.  So I built this.  It's not a complete solution, but it got me going.

Run `./build.sh --msvc`

It does what the gecko build does in terms of using a combination of vswhere, the windows registry, and grovelling around on the filesystem.  I have to say, dealing with VS is a massive pain.

I tried to make it generic, but it is probably only going to work on my machine for reasons I didn't anticipate.  Ultimately we probably want to run this code when we detect that we are on windows, but I didn't go that far.  If we made it that far, we might be able to switch to using this for CI, which would be ideal, but those machines probably have strange MSVC configurations.

Gotta say, it's a bit weird running shell scripts to process windows filenames.

Comment 13

[J.C. Jones [:jcj] (he/they)]

Yeah, there's zero chance of me getting to this before my leave. Unassigning myself.

That is a weird script, mt. ;) But maybe all such scripts look that way on Windows for VS these days.

Comment 14

[Elio Maldonado]

(In reply to Martin Thomson [:mt:] from comment #12)
> 
> Run `./build.sh --msvc`

I had to do it like this: export PATH=$PATH:${PATH_TO_WHER_I_HAVE_GYP}; ./build.sh --msvc
otherewise I would tell me I have to Install gyp
Is there away to make the mozilla build sytemm to know where gyp is?
> 
> Gotta say, it's a bit weird running shell scripts to process windows
> filenames.

Oh yes, it's much harder to set things up in order to build on Windows compared to Linux and macOS. Again, I am an occasional user of windows. I had to read a lot and I suspect the build documentation for NSS will need some work once this is done.

Comment 15

[Martin Thomson [:mt:]]

OK, at least it worked.  Yes, you have to have gyp and ninja on the path.

Comment 16

[Jacob]

Has anyone made any progress toward's a fix for this?

I am getting the following whenever I try to build:

c:/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows/../../../../../pr/src/md/windows/ntmisc.c(1095): error C2065: 'eax': undeclared identifier
c:/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows/../../../../../pr/src/md/windows/ntmisc.c(1095): error C2146: syntax error: missing ';' before identifier 'lock'
c:/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows/../../../../../pr/src/md/windows/ntmisc.c(1095): error C2065: 'lock': undeclared identifier
c:/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows/../../../../../pr/src/md/windows/ntmisc.c(1095): error C2146: syntax error: missing ';' before identifier 'xadd'
c:/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows/../../../../../pr/src/md/windows/ntmisc.c(1095): error C2065: 'xadd': undeclared identifier
c:/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows/../../../../../pr/src/md/windows/ntmisc.c(1095): error C2146: syntax error: missing ';' before identifier 'dword'
c:/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows/../../../../../pr/src/md/windows/ntmisc.c(1095): error C2065: 'dword': undeclared identifier
c:/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows/../../../../../pr/src/md/windows/ntmisc.c(1095): error C2146: syntax error: missing ';' before identifier 'ptr'
c:/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows/../../../../../pr/src/md/windows/ntmisc.c(1095): error C2065: 'ptr': undeclared identifier
c:/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows/../../../../../pr/src/md/windows/ntmisc.c(1095): error C2065: 'ecx': undeclared identifier
c:/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows/../../../../../pr/src/md/windows/ntmisc.c(1095): error C2109: subscript requires array or pointer type
c:/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows/../../../../../pr/src/md/windows/ntmisc.c(1096): error C2065: 'eax': undeclared identifier
c:/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows/../../../../../pr/src/md/windows/ntmisc.c(1096): error C2146: syntax error: missing ';' before identifier 'add'
c:/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows/../../../../../pr/src/md/windows/ntmisc.c(1096): error C2065: 'add': undeclared identifier
c:/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows/../../../../../pr/src/md/windows/ntmisc.c(1096): error C2146: syntax error: missing ';' before identifier 'eax'
c:/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows/../../../../../pr/src/md/windows/ntmisc.c(1097): error C2065: 'edx': undeclared identifier
c:/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows/../../../../../pr/src/md/windows/ntmisc.c(1097): error C2143: syntax error: missing ';' before '}'
make[5]: *** [ntmisc.obj] Error 2
make[5]: Leaving directory `/c/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md/windows'
make[4]: *** [export] Error 2
make[4]: Leaving directory `/c/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src/md'
make[3]: *** [export] Error 2
make[3]: Leaving directory `/c/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr/src'
make[2]: *** [export] Error 2
make[2]: Leaving directory `/c/src/nss-3.38/nspr/WIN954.0_DBG.OBJ/pr'
make[1]: *** [export] Error 2
make[1]: Leaving directory `/c/src/nss-3.38/nspr/WIN954.0_DBG.OBJ'
make: *** [build_nspr] Error 2

Comment 17

[Martin Thomson [:mt:]]

Did you try the attachment?  Elio reports that it mostly works.

Comment 18

[Jacob]

I had some execution errors that I was trying to work out with it.  Not sure if my path is incorrect or my system config is different.

Comment 19

[mouton.math]

I finally made it.
I stop try to use deliver.sh because gyp doesn't support VS 2017 (have a look at mozilla-build\node-v8.11.1-win-x64\node_modules\node-gyp\gyp\pylib\gyp\MSVSVersion.py). I try to add missing version but the compilation still failed on "Environment variable "SYSTEMROOT" required to be set to valid path" ...

So, starting from the proposition of Martin Thomson (setup_msvc.sh) to create this build script:

./setup_msvc.sh
export PATH=$PATH:/c/mozilla-build/node-v8.11.1-win-x64/node_modules/node-gyp:/c/mozilla-build/node-v8.11.1-win-x64/node_modules/node-gyp/gyp
export OS_TARGET=WIN95
export BUILD_OPT=1
export USE_STATIC_RTL=1
make nss_build_all

Just before running you have to modify ../nspr/configure, there is a test on compiler version:
CC_VERSION=`${CC} -v 2>&1 | sed -ne "$_MSVC_VER_FILTER"`
The thing is cl.exe doesn't take -v as argument. I remove it.

Compilation still exit on error while compiling gtests but I have the binaries I needed ...

Comment 20

[Martin Thomson [:mt:]]

Attachment: Bug 1434943 - Support for MSVC in build.sh, r?jcj

This adds basic support for MSVC to build.sh.  It uses the registry and vswhere
(which is part of the standard mozilla-build setup now) to work out paths and
set them properly.  It's probably a little fragile, but it's better than the
shoestring and tape we have in builds right now.

I took the liberty of sanitizing the command-line options a little here.  Mostly
that is sorting them, but I also deprecated the -m32 option in favour of
specifying target architecture with -t.  That turned out to be a lot cleaner.

Comment 21

[Martin Thomson [:mt:]]

:pmoore, I am close to having this ready, but it appears as though the configuration we have on taskcluster is a little different than what I was expecting.  I think that it's just a matter of picking up the latest MozillaBuild (3.2 added vswhere and that seems to be the problem).

Can you help me out there?  I'm not familiar enough with the taskcluster setup to be confident that I'd hit everything necessary.  (https://raw.githubusercontent.com/taskcluster/generic-worker/665d793bf303c514692d9ec21f4cd105882a4e86/worker_types/nss-win2012r2/userdata shows that you have MozillaBuild 3.1.1, but I'm not sure what it would take to update that.)

Comment 22

[Pete Moore [:pmoore][:pete]]

(In reply to Martin Thomson [:mt:] from comment #21)
> :pmoore, I am close to having this ready, but it appears as though the
> configuration we have on taskcluster is a little different than what I was
> expecting.  I think that it's just a matter of picking up the latest
> MozillaBuild (3.2 added vswhere and that seems to be the problem).
> 
> Can you help me out there?  I'm not familiar enough with the taskcluster
> setup to be confident that I'd hit everything necessary. 
> (https://raw.githubusercontent.com/taskcluster/generic-worker/
> 665d793bf303c514692d9ec21f4cd105882a4e86/worker_types/nss-win2012r2/userdata
> shows that you have MozillaBuild 3.1.1, but I'm not sure what it would take
> to update that.)

Hi Martin,

You've found exactly the right line that needs updating. We just need to change the version here, and redeploy the worker:

https://github.com/taskcluster/generic-worker/blob/fc610c547908a843f5cf922a164e0d406de9d3f2/worker_types/nss-win2012r2/userdata#L67

Before we do that though, we I need to check, because we pinned explicitly to version 3.1.1 here due to a problem, which unfortunately I don't seem to have linked to in my commit message, so I'll need to dig up IRC / bugzilla / email history on this to remind myself what the problem was (or maybe Franziskus remembers).

Comment 23

[Pete Moore [:pmoore][:pete]]

(In reply to Pete Moore [:pmoore][:pete] from comment #22)

> Before we do that though, we I need to check, because we pinned explicitly
> to version 3.1.1 here due to a problem, which unfortunately I don't seem to
> have linked to in my commit message, so I'll need to dig up IRC / bugzilla /
> email history on this to remind myself what the problem was (or maybe
> Franziskus remembers).

... commit was: https://github.com/taskcluster/generic-worker/commit/665d793bf303c514692d9ec21f4cd105882a4e86

Comment 24

[Pete Moore [:pmoore][:pete]]

I found the details in #nss scrollback from 6 June 2018.


pmoore> Peter Moore franziskus: hi! any idea what might be up with this try push? this is with an upgraded worker, wondering if this is related to the upgrade or not: https://treeherder.mozilla.org/#/jobs?repo=nss-try&revision=74248f094f1b47602bb742e623fa9d032cad2629
14:26 
<franziskus> pmoore: well it says `nsinstall: Command not found`. looks like that's no available on the system?
14:26 
<pmoore> Peter Moore is the root cause this? "make[3]: nsinstall: Command not found"
14:26 
<franziskus> ;) I think so
14:26 
<pmoore> Peter Moore do you know what nsinstall is?
14:27 i mean, is it something that should be built/installed as part of the task, or a system dependency?
14:27 
<@kaie> nsinstall is something built as part of nss
14:28 
<franziskus> yeah, it's part of nspr actually
14:28 
<@kaie> nss also has something in coreconf/nsinstall
14:29 
<pmoore> Peter Moore ah, so it might be a PATH issue, that it doesn't find it?
14:29 
<franziskus> yep, but looking at the build it's the NSPR version that fails
14:29 not sure. seems to be the first thing it tries. looks to me like the build environment (mozilla-build) isn't setup properly
14:30 
<pmoore> Peter Moore franziskus: kaie: many thanks - that sounds like a good place for me to start digging
14:30 
<@kaie> pmoore, you could test a build without any changes
14:32 
<pmoore> Peter Moore kaie: i think my task is based on https://treeherder.mozilla.org/#/jobs?repo=nss&revision=798e88281cef81baeb6a445aa8288b2fc752a30e
14:33 but the delta i applied is just to switch the version of generic-worker which i'm testing the upgrade for
14:33 i.e. https://hg.mozilla.org/projects/nss-try/rev/74248f094f1b47602bb742e623fa9d032cad2629
14:34 i'll get a loaner and see if i can find any problems with the mozilla-build environment....
14:35 (btw if that is useful for you guys, the instructions for getting a windows loaner are here: https://wiki.mozilla.org/index.php?title=ReleaseEngineering/How_To/Self_Provision_a_TaskCluster_Windows_Instance&oldid=1194794#For_generic-worker_10.5.0_onwards)
14:40 
<franziskus> 👍
14:58 
<pmoore> Peter Moore franziskus: i can't see anything wrong with the mozilla-build installation - do you know which step should have created nsinstall or placed it in the PATH?
15:00 
<franziskus> pmoore: this is where my nsinstall is in mozilla-build https://www.irccloud.com/pastebin/3p3sc8OF/
15:03 
<pmoore> Peter Moore interesting - so it looks like of those two paths, i have it in one of them, and not the other, and i guess the PATH isn't set correctly to find it...
15:03 franziskus: thanks!
15:03 https://irccloud.mozilla.com/pastebin/mui0R7UI/
 Plain Text • 6 lines raw | line numbers 
15:04 i'll check which version of mozilla-build is being installed
15:04 
<franziskus> hm, might be some changes to mozilla-build again that broke NSS? I wouldn't be surprised 😞
15:07 
<pmoore> Peter Moore it looks like i install from https://ftp.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/MozillaBuildSetup-Latest.exe
15:08 
<franziskus> I think I'm still on 3.1..
15:08 
<pmoore> Peter Moore and it looks like the path gets set to
15:09 set PATH=c:\mozilla-build\python;c:\mozilla-build\msys\local\bin;c:\mozilla-build\7zip;c:\mozilla-build\info-zip;c:\mozilla-build\python\Scripts;c:\mozilla-build\yasm;c:\mozilla-build\msys\bin;c:\Windows\system32;c:\mozilla-build\upx391w;c:\mozilla-build\moztools-x64\bin;c:\mozilla-build\wget
15:09 i'll check what is on the nss-win2012r2 workers...
15:16 → JFSimon joined ([e-mail])
15:32 
<pmoore> Peter Moore interesting, on a live worker, nsinstall.exe is found under C:\mozilla-build\msys\local\bin\nsinstall.exe, even though that seems to be from the same mozilla-build release :/
15:32 i wonder if that directory is writable to all users, and it gets generated as part of the task
15:36 ahhhhh it isn't an absolute link
15:36 https://ftp.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/MozillaBuildSetup-Latest.exe
15:37 i scanned the url and saw "32" and thought it was mozilla-build 3.2 ... lol
15:37 so probably mozilla-build version has been upgraded, and that same url now returns the newer version, which broke things
15:38 
— pmoore confirms suspicion
15:42 
<franziskus> :D
15:44 
<pmoore> Peter Moore indeed!
15:44 https://irccloud.mozilla.com/pastebin/CJpIwAgM/
 Plain Text • 19 lines raw | line numbers 
15:44 version 3.1.1
15:45 versus
15:45 https://irccloud.mozilla.com/pastebin/d4ypjyv9/
 Plain Text • 19 lines raw | line numbers 
15:46 
<franziskus> :/ let me see if they say saomething in the release notes
15:47 of course they don't....
15:47 I guess we have to keep using 3.1.1 for now
15:50 
<pmoore> Peter Moore franziskus: i don't mind either way - i can alter PATH in my try push or i can revert to using mozilla-build 3.1.1
15:50 maybe we should have a separate bug for the mozilla-build upgrade though
15:50 ok, let me freeze on 3.1.1
15:51 
<franziskus> ok, I don't mind either as long as it works ;)
15:54 
<pmoore> Peter Moore franziskus: i've taken the easy/safe option to avoid scope creep, and us getting bitten later: https://github.com/taskcluster/generic-worker/commit/665d793bf303c514692d9ec21f4cd105882a4e86
15:56 
<franziskus> 👍 thanks
15:59 pmoore → pmoore|meeting
17:13 → ntome joined ([e-mail])
17:13 
<ntome> hi
17:13 i have an issue when using pr_recv with a sub-second timeout, it seems the timeout is ignored
18:04 pmoore|meeting → pmoore|afk
18:49 pmoore|afk → pmoore
18:50 
<pmoore> Peter Moore Heads up folks - I'm upgrading taskcluster windows machine image for nss-win2012r2 in https://bugzilla.mozilla.org/show_bug.cgi?id=1467192#c2 to use latest generic-worker release....
18:50 
<nss-tc> https://bugzil.la/1467192 — NEW, [e-mail] — Upgrade generic-worker to 10.8.4 on nss-win2012r2 / win2012r2 / win2012r2-cu
20:54 ⇐ hkario quit ([e-mail]) Quit: Konversation terminated!

Thursday, June 7th, 2018
03:20 ↔ emaldona and KingGhost popped in  ↔ nss-tc nipped out  
08:16 
<franziskus> 👌 thanks pmoore

Comment 25

[Pete Moore [:pmoore][:pete]]

Martin,

So it looks like if we want to upgrade to MozillaBuild 3.2 we'll need to update the PATH in the generated tasks. Currently, I believe the PATH is set to:

> c:\mozilla-build\python;c:\mozilla-build\msys\local\bin;c:\mozilla-build\7zip;c:\mozilla-build\info-zip;c:\mozilla-build\python\Scripts;c:\mozilla-build\yasm;c:\mozilla-build\msys\bin;c:\Windows\system32;c:\mozilla-build\upx391w;c:\mozilla-build\moztools-x64\bin;c:\mozilla-build\wget

But it needs to include the following directory to pick up nsinstall.exe when using MozillaBuild 3.2.

> c:\mozilla-build\bin

... otherwise nsinstall.exe won't be found.

I think you can do that here:

https://hg.mozilla.org/projects/nss/file/8f6014565b91d90ffe67f28a74c4e5cb0d4ab0ec/automation/taskcluster/graph/src/extend.js#l793

So in summary, I think you need to patch these two things:

1) Add 'c:\mozilla-build\bin' to https://hg.mozilla.org/projects/nss/file/8f6014565b91d90ffe67f28a74c4e5cb0d4ab0ec/automation/taskcluster/graph/src/extend.js#l793 (where nsinstall.exe was moved to in MozillaBuild 3.2)
2) Update version number from 3.1.1 to 3.2 in https://github.com/taskcluster/generic-worker/blob/fc610c547908a843f5cf922a164e0d406de9d3f2/worker_types/nss-win2012r2/userdata#L67

Good luck!

Comment 26

[Martin Thomson [:mt:]]

Attachment: Bug 1434943 - Update paths on Windows workers, r?pmoore

We should get this in place on the trunk, so that we can do this properly.

Ideally we would be referencing the worker version by a git tag or commit hash,
I'll raise that issue separately.

Comment 27

[Martin Thomson [:mt:]]

Thanks Pete!

I've made the NSS change on my branch (review: https://phabricator.services.mozilla.com/D5483), and a PR on taskcluster (here: https://github.com/taskcluster/generic-worker/pull/121), but it appears as though you need to merge that PR before we'll pick up the changes.

In other words, is there a process for testing this without just landing code?

Comment 28

[Pete Moore [:pmoore][:pete]]

Comment on attachment 9007938 [details]
Bug 1434943 - Update paths on Windows workers, r?pmoore

Pete Moore [:pmoore][:pete] has approved the revision.

Comment 29

[Pete Moore [:pmoore][:pete]]

(In reply to Martin Thomson [:mt:] from comment #27)
> Thanks Pete!
> 
> I've made the NSS change on my branch (review:
> https://phabricator.services.mozilla.com/D5483), and a PR on taskcluster
> (here: https://github.com/taskcluster/generic-worker/pull/121), but it
> appears as though you need to merge that PR before we'll pick up the changes.
> 
> In other words, is there a process for testing this without just landing
> code?

Yes, we have a worker type called nss-win2012r2-new which is a staging worker type for nss-win2012r2 for exactly this purpose.

nss-win2012r2-new and nss-win2012r2 are in sync, and you can freely land your changes to nss-win2012r2-new without impacting anybody (nobody is currently using this staging worker type). Once that is landed, we can roll it out to production, and you can modify your try push to run against that worker type. If all is well, then we mirror over the worker type userdata file to nss-win2012r2 and deploy that to production.

This patch should be enough to switch the worker type in your try push, once nss-win2012r2-new has been updated:

  * https://raw.githubusercontent.com/taskcluster/generic-worker/master/mozilla-try-scripts/nss.patch

Comment 30

[[github robot]]

Commit pushed to master at https://github.com/taskcluster/generic-worker

https://github.com/taskcluster/generic-worker/commit/654bc76d24a2ccd81673696d52cf61a9755a3970
Bug 1434943 - roll out Mozilla Build 3.2 to nss staging worker type (nss-win2012r2-new)

Comment 31

[Pete Moore [:pmoore][:pete]]

The machine image for nss-win2012r2-new is currently building, and should be propagated to production in around an hour or less. At that point, try pushes can be made for MozillaBuild 3.2 using the patch in comment 29. If that goes well, we can land the PR you made. I also put these details in that PR.

Many thanks Martin!

Comment 32

[Pete Moore [:pmoore][:pete]]

Attachment: Github Pull Request for generic-worker

Comment 33

[Pete Moore [:pmoore][:pete]]

To avoid further delays (as it took me a while to get back to you) I took the liberty of making a try push:

https://treeherder.mozilla.org/#/jobs?repo=nss-try&revision=dac62b5c197cf55f3b8b1c18c9b695c6439e8ad1

If this passes, I think we should be safe to land your nss-win2012r2/userdata patch, roll out to production, and then land your nss patch...

Fingers crossed!

Comment 34

[Martin Thomson [:mt:]]

Thanks Pete, that saved me lots of time!

https://hg.mozilla.org/projects/nss/rev/168510d45e3112cc7d9764799da5a72be130cc06 is the path change landed.  Thanks for the try run (the reds and oranges there are why I'm working on bug 1488148, so ignore those).  Can you make the changes to nss-win2012r2 now?  (At your leisure, I'm still awaiting review on the larger patch, so it's not urgent.)

Comment 35

[Martin Thomson [:mt:]]

Small problem: I'm getting https://raw.githubusercontent.com/taskcluster/generic-worker/5e4ea9732014c04e65a6b2a313ddae28b7fabc30/worker_types/nss-win2012r2-new/userdata which still lists MozillaBuild 3.1.1.  The same shows for the try push in comment 33.  Pete, I think that something hasn't propagated fully (the file is fine on taskcluster/generic-worker master, but it's picking up an older copy).

Comment 36

[Pete Moore [:pmoore][:pete]]

(In reply to Martin Thomson [:mt:] from comment #35)
> Small problem: I'm getting
> https://raw.githubusercontent.com/taskcluster/generic-worker/
> 5e4ea9732014c04e65a6b2a313ddae28b7fabc30/worker_types/nss-win2012r2-new/
> userdata which still lists MozillaBuild 3.1.1.  The same shows for the try
> push in comment 33.  Pete, I think that something hasn't propagated fully
> (the file is fine on taskcluster/generic-worker master, but it's picking up
> an older copy).

Hi Martin,

I'm so sorry, it seems I rolled out nss-win2012r2 rather than nss-win2012r2-new, you are quite right the changes had not propagated! :(

To fix this I'm now deploying to nss-win2012r2-new, and that should be rolled out in about an hour. I will then make a new try push, this time just for Windows, and double check that a) we picked up the new userdata and b) that the version of MozillaBuild is really 3.2.

For the sake of transparency, here is some information about how the worker types get updated:

  https://github.com/taskcluster/generic-worker/blob/master/worker_types/README.md

Comment 37

[Pete Moore [:pmoore][:pete]]

New try push:

  https://treeherder.mozilla.org/#/jobs?repo=nss-try&revision=85a4da743171ef94e81126d02013017e2b5c8fcc&group_state=expanded

I hope all will be ok this time, and again sorry for messing up the previous roll out! :-(

Comment 38

[[github robot]]

Commits pushed to master at https://github.com/taskcluster/generic-worker

https://github.com/taskcluster/generic-worker/commit/a42b71ee58c50cfe8944c5e99d44c27669456a5b
Bug 1434943 - Update MozillaBuild version to 3.2, r?pmoore

https://github.com/taskcluster/generic-worker/commit/7661d65c8f7e0673e29334f7a9d480d232ef91f9
Merge pull request #121 from martinthomson/update-mozilla-build

Bug 1434943 - Update MozillaBuild version to 3.2 for NSS

Comment 39

[Pete Moore [:pmoore][:pete]]

(In reply to Pete Moore [:pmoore][:pete] from comment #37)
> New try push:
> 
>  
> https://treeherder.mozilla.org/#/jobs?repo=nss-
> try&revision=85a4da743171ef94e81126d02013017e2b5c8fcc&group_state=expanded
> 
> I hope all will be ok this time, and again sorry for messing up the previous
> roll out! :-(

Since the try run against nss-win2012r2-new was ok, I've merged the nss-win2012r2 patch (comment 38) and will roll out nss-win2012r2 to production.

Comment 40

[Pete Moore [:pmoore][:pete]]

nss-win2012r2 updates are now in production.

In addition, I've added some information about the process for making changes to nss-win2012r2 here:

https://github.com/taskcluster/generic-worker/commit/3816493808e5d2d32d15ce5187e314d6cac22eda

I realised I hadn't documented this anywhere; hopefully, this will make it easier for the next person that wishes to make changes.

Comment 41

[J.C. Jones [:jcj] (he/they)]

Comment on attachment 9006814 [details]
Bug 1434943 - Support for MSVC in build.sh, r?jcj

J.C. Jones [:jcj] (he/him) has approved the revision.

Comment 42

[Martin Thomson [:mt:]]

https://hg.mozilla.org/projects/nss/rev/8bea05067a0ba403f4066860ae8a59d4410e2e19

Comment 43

[Martin Thomson [:mt:]]

The second landing should stick better:

https://hg.mozilla.org/projects/nss/rev/45f894ebabdb085f1428b748ff28c659e5c309c5