1034594 - Trunk Build fails with Makefile:723: recipe for target 'out/build/icudt52l/root.res' failed (genrb.exe)

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Carsten Book [:Tomcat]]

Attachment: log output

Tried to build a Windows 7 Trunk Debug Build today and it failed with

15:18.42 c:\Users\mozilla\debug-builds\mozilla-central\firefox-debug\intl\icu\target\bin\genrb.exe: error in command line argument "-kC:/mozilla-build/msys"

15:18.45 Makefile:723: recipe for target 'out/build/icudt52l/root.res' failed

not sure what changed, was fine yesterday ?

Comment 1

[:Gijs (he/him)]

My opt builds are also dead.

Comment 2

[:Gijs (he/him)]

Building with --verbose shows:

 0:20.16 PATH=../lib:../stubdata:../tools/ctestfw:$PATH  ../bin/genrb --usePoolBundle -k/ -i ./out/build/icudt52l -s e:/dev/fx-team/intl/icu/source/data/locales -d ./out/build/icudt52l root.txt


The forward slash after the -k gets resolved to the msys root, which is what ends up being shown in the error, it looks like...

Comment 3

[:Gijs (he/him)]

Confirmed by local backout that this is due to bug 1033958.

Comment 4

[:Gijs (he/him)]

Mike / Greg, any idea what's up here?

Comment 5

[:Gijs (he/him)]

Fixed by backing out bug 1033958:

remote:   https://hg.mozilla.org/mozilla-central/rev/e268bb51e084

Comment 6

[Mike Hommey [:glandium]]

O_O you'll have to tell me where this / comes from because here's what the source tree says:
http://mxr.mozilla.org/mozilla-central/search?string=usePoolBundle
The ones in makedata.mak have "-k" with nothing.
The ones in Makefile.in don't have a visiblt "-k", but they have $(GENRBOPTS) after --usePoolBundle, and that comes from there:
http://mxr.mozilla.org/mozilla-central/source/intl/icu/Makefile.in#57
where, as you can see, GENRBOPTS is set to "-k -R"

Comment 7

[Mike Hommey [:glandium]]

I'll add this: bug 1033958 changed nothing about how icu is built.

Comment 8

[Mike Hommey [:glandium]]

And of course, I can't reproduce on 74d9bc471cc5, which is the changeset before the backout. The only reason the backout worked for you is that it triggered a reconfigure, which fixed up whatever broken state you were in. I still don't see how that could ever happen, considering the build rules. But since you're not giving any more detail about your build environment... (mozconfig, paths, etc)

Comment 9

[Mike Hommey [:glandium]]

I seriously doubt whatever problem was there was actually fixed by the backout. I backed out the backout anyways.

Comment 10

[Bill Gianopoulos [:WG9s]]

My original build that failed was from a full source clobber built on the same changeset as today's nightly.  Backing out the fix for bug 1033958 fixed the issue for me.

my .mozconfig looks as follows:

BUILD_OFFICIAL=1
export BUILD_OFFICIAL
WIN32_REDIST_DIR=~/mozilla/redist/x86
export WIN32_REDIST_DIR
#WIN32_CRT_SRC_DIR=~/mozilla/crtsrc
mk_add_options BUILD_OFFICIAL=1
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/fx-obj
mk_add_options RUN_AUTOCONF_LOCALLY=1
mk_add_options AUTOCONF=/usr/local/bin/autoconf-2.13
ac_add_options --enable-optimize=-O2
ac_add_options --disable-debug
ac_add_options --enable-tests
ac_add_options --disable-crashreporter
ac_add_options --disable-metro
ac_add_options --enable-jemalloc
ac_add_options --without-x
mk_add_options MOZ_MAKE_FLAGS="-j4"

Comment 11

[Mike Hommey [:glandium]]

(In reply to Bill Gianopoulos [:WG9s] from comment #10)
> My original build that failed was from a full source clobber built on the
> same changeset as today's nightly.  Backing out the fix for bug 1033958
> fixed the issue for me.
> 
> my .mozconfig looks as follows:
> 
> BUILD_OFFICIAL=1
> export BUILD_OFFICIAL
> WIN32_REDIST_DIR=~/mozilla/redist/x86
> export WIN32_REDIST_DIR
> #WIN32_CRT_SRC_DIR=~/mozilla/crtsrc
> mk_add_options BUILD_OFFICIAL=1
> mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/fx-obj
> mk_add_options RUN_AUTOCONF_LOCALLY=1
> mk_add_options AUTOCONF=/usr/local/bin/autoconf-2.13
> ac_add_options --enable-optimize=-O2
> ac_add_options --disable-debug
> ac_add_options --enable-tests
> ac_add_options --disable-crashreporter
> ac_add_options --disable-metro
> ac_add_options --enable-jemalloc
> ac_add_options --without-x
> mk_add_options MOZ_MAKE_FLAGS="-j4"

Building 81691a55e60f with that mozconfig (except for the WIN32_REDIST_DIR lines) works for me.

Comment 12

[Masatoshi Kimura [:emk]]

My local build from the clobber also fails with bug 1033958 and succeeds with the backout.
.mozconfig:
ac_add_options --enable-warnings-as-errors
ac_add_options --disable-optimize
ac_add_options --enable-metro
mk_add_options AUTOCLOBBER=1
WIN32_REDIST_DIR="$VCINSTALLDIR\redist\x86\Microsoft.VC120.CRT"

Comment 13

[Mike Hommey [:glandium]]

To anyone who can reproduce this issue, I'd need the *complete* log for the following commands, starting from a clobber:
- mozmake -f client.mk configure
- mozmake -C whatever-your-objdir-is/intl/icu buildicu
as well as the contents of config.status and .mozconfig.json from the objdir.

Comment 14

[:Gijs (he/him)]

(In reply to Mike Hommey [:glandium] from comment #8)
> And of course, I can't reproduce on 74d9bc471cc5, which is the changeset
> before the backout. The only reason the backout worked for you is that it
> triggered a reconfigure, which fixed up whatever broken state you were in.

No, because I clobbered every time I built trying to figure out where the culprit for the problems I was seeing was located. In other words, every build had a new configure, and every build with bug 1033958 broke, and every build without it succeeded.

> I
> still don't see how that could ever happen, considering the build rules. But
> since you're not giving any more detail about your build environment...
> (mozconfig, paths, etc)

You weren't around to ask for them, and so many people's builds broke that folks in #developers thought backing out was a better option than waiting (especially as Greg is now in the middle of a long july 4th weekend, and your weekend already started as well).

As noted already, several people had this issue, all with varying mozconfigs... my own mozconfig is almost empty, save for defining an objdir. I'll add it in a bit when I've booted up my Windows machine - but it seems unlikely it's related to mozconfig; more like it's related to mozmake/mozillabuild/something else.

(In reply to Mike Hommey [:glandium] from comment #6)
> O_O you'll have to tell me where this / comes from because here's what the
> source tree says:
> http://mxr.mozilla.org/mozilla-central/search?string=usePoolBundle
> The ones in makedata.mak have "-k" with nothing.
> The ones in Makefile.in don't have a visiblt "-k", but they have
> $(GENRBOPTS) after --usePoolBundle, and that comes from there:
> http://mxr.mozilla.org/mozilla-central/source/intl/icu/Makefile.in#57
> where, as you can see, GENRBOPTS is set to "-k -R"

If I'd known where it'd come from, I would have tried fixing it. As it was, I had enough evidence that it was caused by this patch, and that it affected a number of people, all with the same symptoms, and several people in #developers who suggested we should back the problematic change out, so that was what I did.

Comment 15

[:Gijs (he/him)]

MOZCONFIG:
Gijs@HERMES /e/dev/fx-team
$ cat $MOZCONFIG
export MOZ_PSEUDO_DERECURSE=1
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../builds/opt-@CONFIG_GUESS@

ac_add_options --enable-profiling

Comment 16

[:Gijs (he/him)]

Attachment: configure-output.txt

Comment 17

[:Gijs (he/him)]

Attachment: buildicu-output.txt

Comment 18

[Mike Hommey [:glandium]]

(In reply to :Gijs Kruitbosch from comment #17)
> Created attachment 8451228 [details]
> buildicu-output.txt

This isn't failing... what the hell?

Can you attach a complete log of a clobber make -f client.mk (if it fails)?

Comment 19

[:Gijs (he/him)]

Attachment: ./mach clobber && ./mach build

Comment 20

[:Gijs (he/him)]

Maybe it has to do with  0:20.27 checking for E:/mozilla-build/msys/bin/mozmake.EXE... e:/dev/fx-team/configure: line 5420: test: /e/tortoisehg//E: binary operator expected ? iow, if it's using something other than mozmake for icu...

Comment 21

[Mike Hommey [:glandium]]

Also, what is the output of mozmake --version?

Comment 22

[Mike Hommey [:glandium]]

(In reply to :Gijs Kruitbosch from comment #19)
> Created attachment 8451230 [details]
> ./mach clobber && ./mach build

The output of mach build is useless. Try again with -v, but try make -f client.mk first.

Comment 23

[:Gijs (he/him)]

Gijs@HERMES /e/dev/fx-team
$ make -f client.mk build &> plain-make-clientmk-build.txt

Gijs@HERMES /e/dev/fx-team
$ cat plain-make-clientmk-build.txt
make -f /e/dev/fx-team/client.mk realbuild CREATE_MOZCONFIG_JSON=
Adding client.mk options from e:/dev/configs/opt.mozconfig:
    MOZ_OBJDIR=e:/dev/builds/opt-i686-pc-mingw32
    OBJDIR=e:/dev/builds/opt-i686-pc-mingw32
    FOUND_MOZCONFIG=e:/dev/configs/opt.mozconfig
make[1]: Entering directory `/e/dev/fx-team'
make -j8 -C e:/dev/builds/opt-i686-pc-mingw32
make[2]: Entering directory `/e/dev/builds/opt-i686-pc-mingw32'
e:/dev/fx-team/config/baseconfig.mk:35: *** MSYS make is not supported.  Stop.
make[2]: Leaving directory `/e/dev/builds/opt-i686-pc-mingw32'
make[1]: *** [realbuild] Error 2
make[1]: Leaving directory `/e/dev/fx-team'
make: *** [build] Error 2

(NB: this was without an extra clobber after the previous (clobber) ./mach build... would that make much of a difference?)

Gijs@HERMES /e/dev/fx-team
$ mozmake --version
GNU Make 4.0-23-g9d58570
Built for Windows32
Copyright (C) 1988-2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Gijs@HERMES /e/dev/fx-team
$ make --version
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i686-pc-msys

Comment 24

[Mike Hommey [:glandium]]

(In reply to :Gijs Kruitbosch from comment #20)
> Maybe it has to do with  0:20.27 checking for
> E:/mozilla-build/msys/bin/mozmake.EXE... e:/dev/fx-team/configure: line
> 5420: test: /e/tortoisehg//E: binary operator expected ? iow, if it's using
> something other than mozmake for icu...

mozmake.EXE[7]: *** [out/build/icudt52l/root.res] Error 1

That says it *is* using mozmake.

Comment 25

[Mike Hommey [:glandium]]

(In reply to :Gijs Kruitbosch from comment #23)
> Gijs@HERMES /e/dev/fx-team
> $ make -f client.mk build &> plain-make-clientmk-build.txt
> 
> Gijs@HERMES /e/dev/fx-team
> $ cat plain-make-clientmk-build.txt
> make -f /e/dev/fx-team/client.mk realbuild CREATE_MOZCONFIG_JSON=
> Adding client.mk options from e:/dev/configs/opt.mozconfig:
>     MOZ_OBJDIR=e:/dev/builds/opt-i686-pc-mingw32
>     OBJDIR=e:/dev/builds/opt-i686-pc-mingw32
>     FOUND_MOZCONFIG=e:/dev/configs/opt.mozconfig
> make[1]: Entering directory `/e/dev/fx-team'
> make -j8 -C e:/dev/builds/opt-i686-pc-mingw32
> make[2]: Entering directory `/e/dev/builds/opt-i686-pc-mingw32'
> e:/dev/fx-team/config/baseconfig.mk:35: *** MSYS make is not supported. 
> Stop.
> make[2]: Leaving directory `/e/dev/builds/opt-i686-pc-mingw32'
> make[1]: *** [realbuild] Error 2
> make[1]: Leaving directory `/e/dev/fx-team'
> make: *** [build] Error 2

Err sorry, I meant mozmake -f client.mk.

Comment 26

[Mike Hommey [:glandium]]

> Gijs@HERMES /e/dev/fx-team
> $ mozmake --version
> GNU Make 4.0-23-g9d58570
> Built for Windows32
> Copyright (C) 1988-2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.

Try upgrading MozillaBuild. You should have version 4.0-43-g8411528

Comment 27

[:Gijs (he/him)]

(I'm running a clobber && mozmake -f client.mk build right now, but I need to run out. Will post the results of that and a mozillabuild update later today)

Comment 28

[Mike Hommey [:glandium]]

Note that maybe you have MozillaBuild 1.9, but your path picks an old mozmake you installed yourself:
E:/mozilla-build/msys/bin/mozmake.EXE

The one from MozillaBuild is in mozilla-build/mozmake.

Comment 29

[Mike Hommey [:glandium]]

Note on my end I'm trying a build with mozmake 4.0-23-g9d58570.

Comment 30

[Mike Hommey [:glandium]]

Ok, I could reproduce the failure with 4.0-23-g9d58570.

Comment 31

[:Gijs (he/him)]

Attachment: clobber-plain-mk.txt

Comment 32

[:Gijs (he/him)]

(In reply to Mike Hommey [:glandium] from comment #28)
> Note that maybe you have MozillaBuild 1.9, but your path picks an old
> mozmake you installed yourself:
> E:/mozilla-build/msys/bin/mozmake.EXE
> 
> The one from MozillaBuild is in mozilla-build/mozmake.

This is the case for me. I'll try a clobber and rebuild with the other mozmake.

Comment 33

[Mike Hommey [:glandium]]

I'm now able to reproduce, even with mozmake . I don't know what changed, but I'm using a different AWS instance now than i was using earlier, so maybe this is related. Paths changed, too. Anyways, this looks like a bad bug in (moz)make on Windows.

Comment 34

[Mike Hommey [:glandium]]

So, the problem lies, in fact, in msys, and it doesn't happen consistently (if it did, we'd have had busted windows builds on tbpl, and my attempts at reproducing would all have worked).

The minimal testcase is this:

$ cat > Makefile <<EOF
foo:
        $(MAKE) bar A='a a'

bar:
        (cd . && $(MAKE) baz)

baz:
        echo $(A)
EOF
$ mozmake B=

This displays "a/" instead of "a a" you'd get with "mozmake" (without B=)

The problem is make sets MAKEFLAGS to contain "A=a\ a" when executing the rule for bar, and msys transforms that to "A=a/ a"

Luckily, genrb allows flags to be stuck together, so we can change GENRBOPTS to not have spaces, working around the issue.

Comment 35

[Mike Hommey [:glandium]]

Attachment: Work around msys silliness breaking the build for some people after bug 1033958

Comment 36

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #34)
> The problem is make sets MAKEFLAGS to contain "A=a\ a" when executing the
> rule for bar, and msys transforms that to "A=a/ a"

Note that fiddling with the name of variables and their values makes things change.

Comment 37

[Bill Gianopoulos [:WG9s]]

(In reply to Mike Hommey [:glandium] from comment #34)
> So, the problem lies, in fact, in msys, and it doesn't happen consistently
> (if it did, we'd have had busted windows builds on tbpl, and my attempts at
> reproducing would all have worked).
> 
> The minimal testcase is this:
> 
> $ cat > Makefile <<EOF
> foo:
>         $(MAKE) bar A='a a'
> 
> bar:
>         (cd . && $(MAKE) baz)
> 
> baz:
>         echo $(A)
> EOF
> $ mozmake B=
> 
> This displays "a/" instead of "a a" you'd get with "mozmake" (without B=)
> 
> The problem is make sets MAKEFLAGS to contain "A=a\ a" when executing the
> rule for bar, and msys transforms that to "A=a/ a"
> 
> Luckily, genrb allows flags to be stuck together, so we can change GENRBOPTS
> to not have spaces, working around the issue.

So it would seem it thinks this is a file path, so is replacing windows '\' path separators with '/'.

Comment 38

[:Gijs (he/him)]

(In reply to Mike Hommey [:glandium] from comment #33)
> I'm now able to reproduce, even with mozmake . I don't know what changed,
> but I'm using a different AWS instance now than i was using earlier, so
> maybe this is related. Paths changed, too. Anyways, this looks like a bad
> bug in (moz)make on Windows.

I'm assuming you meant a newer mozmake? I'm indeed still seeing the same issue with the newer version of mozmake...

Comment 39

[Bill Gianopoulos [:WG9s]]

(In reply to Mike Hommey [:glandium] from comment #35)
> Created attachment 8451247 [details] [diff] [review]
> Work around msys silliness breaking the build for some people after bug
> 1033958

My build worked fine today with this patch applied.

Comment 40

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Comment on attachment 8451247 [details] [diff] [review]
Work around msys silliness breaking the build for some people after bug 1033958

Review of attachment 8451247 [details] [diff] [review]:
-----------------------------------------------------------------

This fixed my build too.

Comment 41

[Gregory Szorc [:gps]]

Comment on attachment 8451247 [details] [diff] [review]
Work around msys silliness breaking the build for some people after bug 1033958

Review of attachment 8451247 [details] [diff] [review]:
-----------------------------------------------------------------

Another reminder that the BS abbreviation for Build System is appropriate. Hacks all the way down.

Comment 43

[Mike Hommey [:glandium]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/93e07b92f879

Comment 44

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/93e07b92f879