1248963 - Buildbot factory jobs on try have stale hardcoded values (win build on try-comm-central can't find nsis 3.0b1)

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[ISHIKAWA, Chiaki]

(Sorry I am not sure which component to choose.)

The failure is obvious.

https://treeherder.mozilla.org/logviewer.html#?job_id=16322&repo=try-comm-central

You get in the raw log, the following error that caused the configure failure.


checking for jarsigner... :
checking for keytool... :
Found D3D compiler in Windows SDK.
Found MOZ_D3DCOMPILER_VISTA_DLL_PATH: /c/Program Files (x86)/Windows Kits/8.1//Redist/D3D/x86/d3dcompiler_47.dll
Found DirectX SDK via registry, using C:/Program Files (x86)/Microsoft DirectX SDK (June 2010)
Found d3dcompiler DLL for Vista+: d3dcompiler_47.dll
Found d3dcompiler DLL for XP: D3DCompiler_43.dll
checking for makensis-3.0b3.exe... no
checking for makensis-3.0b1.exe... no
checking for makensis... no
------ config.log ------
configure: error: To build the installer you must have the latest MozillaBuild or NSIS version 3.0b1 or greater in your path.
#line 22625 "configure"
#include "confdefs.h"

int main() {
static char c __attribute__ ((aligned(16))) = 0; return c;
; return 0; }
configure:22632: cl -c  -TC -nologo -D_HAS_EXCEPTIONS=0 -W3 -Gy -arch:IA32 -FS -wd4244 -wd4267 -wd4819 -we4553 -Werror  conftest.c 1>&5
cl : Command line error D8021 : invalid numeric argument '/Werror'
configure: failed program was:
#line 22625 "configure"
#include "confdefs.h"

int main() {
static char c __attribute__ ((aligned(8))) = 0; return c;
; return 0; }
configure:23830: checking for java
configure:23879: checking for javac
configure:23928: checking for javah
configure:23977: checking for jar
configure:24026: checking for jarsigner
configure:24075: checking for keytool
configure:24957: checking for makensis-3.0b3.exe
configure:24957: checking for makensis-3.0b1.exe
configure:24957: checking for makensis
configure: error: To build the installer you must have the latest MozillaBuild or NSIS version 3.0b1 or greater in your path.
*** Fix above errors and then restart with\
               "c:/builds/moz2_slave/tb-try-c-cen-w32-0000000000000/build/mozmake.exe -f client.mk build"
c:/builds/moz2_slave/tb-try-c-cen-w32-0000000000000/build/client.mk:361: recipe for target 'configure' failed
mozmake.exe[2]: Leaving directory 'c:/builds/moz2_slave/tb-try-c-cen-w32-0000000000000/build'
c:/builds/moz2_slave/tb-try-c-cen-w32-0000000000000/build/client.mk:375: recipe for target 'c:/builds/moz2_slave/tb-try-c-cen-w32-0000000000000/build/objdir-tb/Makefile' failed
mozmake.exe[1]: Leaving directory 'c:/builds/moz2_slave/tb-try-c-cen-w32-0000000000000/build'
mozmake.exe[2]: *** [configure] Error 1
mozmake.exe[1]: *** [c:/builds/moz2_slave/tb-try-c-cen-w32-0000000000000/build/objdir-tb/Makefile] Error 2
client.mk:219: recipe for target 'build' failed
mozmake.exe: *** [build] Error 2
program finished with exit code 2
elapsedTime=986.904000
========= master_lag: -0.19 =========
========= Finished compile failed (results: 2, elapsed: 16 mins, 26 secs) (at 2016-02-17 04:24:11.668522) =========

========= Skipped  (results: not started, elapsed: not started) =========
========= Skipped  (results: not started, elapsed: not started) =========
========= Skipped  (results: not started, elapsed: not started) ==

Strange that the DEBUG version works and can be build in the same job submission.

Anyway, BOTH the debug build and optimize build seem to suffer from these problems.

The error messages is scattered in the command output, but please note that there are
(about three dozens before the quoted error message appear in
the optimization build raw log, lines that state

--- error in configure line 5910:

c:/builds/moz2_slave/tb-try-c-cen-w32-0000000000000/build/mozilla/configure: line 5910: test: /c/mozilla-build/vim/vim72/c: binary operator expected
c:/builds/moz2_slave/tb-try-c-cen-w32-0000000000000/build/mozilla/configure: line 5910: test: /c/mozilla-build/python27/c: binary operator expected
c:/builds/moz2_slave/tb-try-c-cen-w32-0000000000000/build/mozilla/configure: line 5910: test: /c/mozilla-build/hg/c: binary operator expected
c:/builds/moz2_slave/tb-try-c-cen-w32-0000000000000/build/mozilla/configure: line 5910: test: /c/ProgramData/chocolatey/bin/c: binary operator expected
--- quote end.

Actually there are MANY such errors.
I look in mozilla/configure on my local PC and found that
on 5905  (five lines shift
5901  IFS="${IFS= 	}"; ac_save_ifs="$IFS"; IFS=":"
5902   ac_dummy="$PATH"
5903  for ac_dir in $ac_dummy; do 
5904    test -z "$ac_dir" && ac_dir=.
5905    if test -f $ac_dir/$ac_word; then
5906      ac_cv_path_GMAKE="$ac_dir/$ac_word"
5907      break
5908    fi
5909  done

I think in the following line,

5905    if test -f $ac_dir/$ac_word; then

$ac_dir/$ac_word probably need to be quoted as in
        if test -f "$ac_dir/$ac_word" ; then

Maybe for some reason, due to the failure to set proper search path,
optimization binary could not be built (I have no idea why the debug version worked somehow).

There is also ANOTHER similar configure issue.

When mozilla/js/src is configured under windows, we get the
following errors:

js\src> c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/mozilla/js/src/configure: line 5012: test: /c/tools/vs2013/Common7/IDE/c: binary operator expected
js\src> c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/mozilla/js/src/configure: line 5012: test: /c/tools/vs2013/VC/BIN/amd64_x86/c: binary operator expected
js\src> c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/mozilla/js/src/configure: line 5012: test: /c/tools/vs2013/VC/BIN/amd64/c: binary operator expected
js\src> c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/mozilla/js/src/configure: line 5012: test: /c/tools/vs2013/Common7/Tools/c: binary operator expected
js\src> c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/mozilla/js/src/configure: line 5012: test: /c/tools/vs2013/VC/VCPackages/c: binary operator expected
 ... many more lines ...

I also notice the similar error in js/src/configure:

 5006	  for ac_dir in $ac_dummy; do 
 5007	    test -z "$ac_dir" && ac_dir=.
 5008	    if test -f $ac_dir/$ac_word; then
 5009	      ac_cv_path_GMAKE="$ac_dir/$ac_word"
 5010	      break
 5011	    fi
 5012	  done

in line 5008, probably $ac_dir/$ac_word needs to be quoted.

TIA

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

We have compiled on Windows once or twice in the past, so I doubt the issue comes down to a quoting error in autoconf.

Can you identify the differences between this try job and, say,
  https://treeherder.mozilla.org/#/jobs?repo=try&filter-searchStr=Windows%20XP%20opt
which appear to be substantially green?  In particular, does comm-central lack some patch present in mozilla-central to use a different version of NSIS?

Comment 2

[ISHIKAWA, Chiaki]

(In reply to Dustin J. Mitchell [:dustin] from comment #1)
> We have compiled on Windows once or twice in the past, so I doubt the issue
> comes down to a quoting error in autoconf.
> 
> Can you identify the differences between this try job and, say,
>  
> https://treeherder.mozilla.org/#/jobs?repo=try&filter-
> searchStr=Windows%20XP%20opt
> which appear to be substantially green?  In particular, does comm-central
> lack some patch present in mozilla-central to use a different version of
> NSIS?

I will try. I noticed that even the successful mozilla m-c x86 opt job
suffers from the same "binary operator" bug in configures (two of them),
see, for example, 
http://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-688ef5e39fd6fe6dc0b0df5b9263664a0fe09fee/try-win32/try-win32-bm78-try1-build3130.txt.gz

and it makes comparison rather difficult.
I will first try to eliminate this bug in configure and see if this improves the situation, i.e. makes x86 optimization build succeed on try-comm-central, and
then if not, check there are nsis-related  differences.

Comment 3

[ISHIKAWA, Chiaki]

Hi, come to think of it, I use Debian GNU/Linux 64-bit for my local development and
I am discussing the Windows build on try-comm-central.
Obviously the "configure" scripts after it is created from "configure.in" will
be very different under two OSes.

Is there any handy manner I can take a look at the content of the configure on try-comm-central from within the submission?
(I don't think the content is visible inside the raw log.)

TIA

Comment 4

[Justin Wood (:Callek)]

(In reply to ISHIKAWA, Chiaki from comment #3)
> Hi, come to think of it, I use Debian GNU/Linux 64-bit for my local
> development and
> I am discussing the Windows build on try-comm-central.
> Obviously the "configure" scripts after it is created from "configure.in"
> will
> be very different under two OSes.

The autoconf version in use will generate the same output.  M4 has no OS consideration in its use (and thats what autoconf uses, assuming same version of autoconf in both places)

configure.in-->configure is basically just expanding a large set of macros.  And then the *run* of configure is done with OS specifics in mind.

Comment 5

[Stefan Sitter [:ssitter]]

On my local Windows 7 system with MozillaBuild 2.1.0 PATH contains
c:\mozilla-build\nsis-3.0b1;c:\mozilla-build\nsis-2.46u

The try-comm-central Windows XP opt PATH contains
c:\mozilla-build\nsis-2.46u;c:\mozilla-build\nsis-3.0a2

Bug 1237040 added support for NSIS 3.0b3 and removed support for NSIS 2.46u and NSIS 3.0a1
http://hg.mozilla.org/mozilla-central/rev/01a7297a24d0

Maybe the build slave just requires that a more recent MozillaBuild package is deployed?

Comment 6

[ISHIKAWA, Chiaki]

(In reply to Stefan Sitter from comment #5)
> On my local Windows 7 system with MozillaBuild 2.1.0 PATH contains
> c:\mozilla-build\nsis-3.0b1;c:\mozilla-build\nsis-2.46u
> 
> The try-comm-central Windows XP opt PATH contains
> c:\mozilla-build\nsis-2.46u;c:\mozilla-build\nsis-3.0a2
> 
> Bug 1237040 added support for NSIS 3.0b3 and removed support for NSIS 2.46u
> and NSIS 3.0a1
> http://hg.mozilla.org/mozilla-central/rev/01a7297a24d0
> 
> Maybe the build slave just requires that a more recent MozillaBuild package
> is deployed?

It looks like that:
Bug 1237040 comment 15, and 
Bug 1237040 comment 19 are exactly the errors I see in failed Windows optimized build
on try-comm-central.
There is antoher bug 
Bug 1236624 - Please add NSIS 3.0b3 to the Windows build slaves
which is referenced by bug 1237040.

Lt us hope the new NSIS is available on try-comm-central build machines soon.

Comment 7

[ISHIKAWA, Chiaki]

(In reply to Justin Wood (:Callek) from comment #4)
> (In reply to ISHIKAWA, Chiaki from comment #3)
> > Hi, come to think of it, I use Debian GNU/Linux 64-bit for my local
> > development and
> > I am discussing the Windows build on try-comm-central.
> > Obviously the "configure" scripts after it is created from "configure.in"
> > will
> > be very different under two OSes.
> 
> The autoconf version in use will generate the same output.  M4 has no OS
> consideration in its use (and thats what autoconf uses, assuming same
> version of autoconf in both places)
> 
> configure.in-->configure is basically just expanding a large set of macros. 
> And then the *run* of configure is done with OS specifics in mind.

Hi, thank you for your comment.
M4 itself does not have much OS dependency (maybe aside from line ending string issue).

But configure.in can be written in very OS-dependent manner so that the output
will change drastically depending on OS. No?

I re-read configure.in below ./mozilla subdirectory of C-C tree.
You are right. The configure.in is written in such a manner that
any run-time OS check, etc. is exposed and done at the invocation time of the
resulting configure and NOT at the time configure.in is transformed into configure using M4 (unless some M4 macros try to optimize the output by assuming the target OS in advance, but this is unlikely when I think about it).

Also, I noticed that the minimum version of NSIS supported  is hard-coded in configure.in, and
as Stefan Sitter said in comment 5, if windows XP OPT build slave does not have
the upgraded NSIS (I suspect any build machines), 
it will fail to build a binary there.

I think it is clear that the build slave needs to have the new NSIS binaries, etc.

cf. From configure.in: For windows, NSIS 3.0b1 seems to be 
the minimum version required for build now.

dnl ========================================================
dnl Installer
dnl ========================================================
dnl Abort Windows build if the required major version and
dnl minimum minor version of Unicode NSIS isn't in the path
dnl (unless in case of cross compiling, for which Unicode
dnl is not yet sufficient).
if test "$OS_ARCH" = "WINNT"; then
    MIN_NSIS_MAJOR_VER=3
    MIN_NSIS_MINOR_VER=0
    MIN_NSIS_PRERELEASE_TYPE=b
    MIN_NSIS_PRERELEASE_VER=1
    MOZ_PATH_PROGS(MAKENSISU, $MAKENSISU makensis-3.0b3.exe makensis-3.0b1.exe makensis)
    ...

Comment 8

[ISHIKAWA, Chiaki]

(In reply to Dustin J. Mitchell [:dustin] from comment #1)
> We have compiled on Windows once or twice in the past, so I doubt the issue
> comes down to a quoting error in autoconf.
> 
> Can you identify the differences between this try job and, say,
>  
> https://treeherder.mozilla.org/#/jobs?repo=try&filter-
> searchStr=Windows%20XP%20opt
> which appear to be substantially green?  In particular, does comm-central
> lack some patch present in mozilla-central to use a different version of
> NSIS?


From the observation in comment 5 and comment 6, and bug 1237040 and bug 1236624, not it is clear that on XP OPT build slave for try-comm-central,

 - build script has been updated to expect at least 3.0b1 version of NSIS

but somehow

 - build slave does not have the necessary newer NSIS binaries.

So as in bug 1236624, please add new NSIS binaries such as NSIS3.0b3, etc.

TIA

Comment 9

[ISHIKAWA, Chiaki]

https://bugzilla.mozilla.org/show_bug.cgi?id=1236624#c5
now has an outline of what one can do (rewrite a path in a script) on XP OPT buildbot
to fix the issue.

Anyone?

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-configs] Remove try hardcodes

I happened to be looking at this code for a similar bug, so this is a drive-by fix.

It affects these jobs:
 Firefox try win32 l10n
 TB WINNT 5.2 try-comm-central build
 TB WINNT 5.2 try-comm-central leak test build
 WINNT 5.2 try pgo-build
 WINNT 5.2 try spidermonkey_try-compacting
 WINNT 5.2 try spidermonkey_try-plain build

It changes the env vars like so:
'PATH': '${MOZILLABUILD}\\python27;${MOZILLABUILD}\\buildbotve\\scripts;${PATH}' -->
'PATH': '${MOZILLABUILD}\\nsis-3.0b1;${MOZILLABUILD}\\nsis-2.46u;${MOZILLABUILD}\\python27;${MOZILLABUILD}\\buildbotve\\scripts;${PATH}'
'BINSCOPE': 'C:\\Program Files\\Microsoft\\SDL BinScope\\Binscope.exe' -->
'BINSCOPE': 'C:\\Program Files (x86)\\Microsoft\\SDL BinScope\\BinScope.exe'

No longer defined
'SYMBOL_SERVER_HOST': 'relengweb1.dmz.scl3.mozilla.com'

New variables
'MOZ_AUTOMATION': '1'

The key thing is prepending nsis-3.0b1 directory to PATH.

Comment 12

[Rail Aliiev [:rail]]

Comment on attachment 8742615 [details] [diff] [review]
[buildbot-configs] Remove try hardcodes

DIAF!

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8742615 [details] [diff] [review]
[buildbot-configs] Remove try hardcodes

https://hg.mozilla.org/build/buildbot-configs/rev/4098e57a23a3
https://hg.mozilla.org/build/buildbot-configs/rev/5b7f5d392608

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Green rebuild on https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=5e7c1ec836d6424913a2678d8bfae34a6b5c3337.

Comment 15

[aleth [:aleth]]

Thanks!