Closed Bug 1034594 Opened 8 years ago Closed 8 years ago

Trunk Build fails with Makefile:723: recipe for target 'out/build/icudt52l/root.res' failed (genrb.exe)

Categories

(Firefox Build System :: General, defect)

x86
Windows 7
defect
Not set
major

Tracking

(Not tracked)

RESOLVED FIXED
mozilla33

People

(Reporter: cbook, Assigned: glandium)

References

Details

Attachments

(5 files, 1 obsolete file)

Attached file log output
Tried to build a Windows 7 Trunk Debug Build today and it failed with

15:18.42 c:\Users\mozilla\debug-builds\mozilla-central\firefox-debug\intl\icu\target\bin\genrb.exe: error in command line argument "-kC:/mozilla-build/msys"

15:18.45 Makefile:723: recipe for target 'out/build/icudt52l/root.res' failed

not sure what changed, was fine yesterday ?
My opt builds are also dead.
Summary: Debug Trunk Build fails with Makefile:723: recipe for target 'out/build/icudt52l/root.res' failed → Trunk Build fails with Makefile:723: recipe for target 'out/build/icudt52l/root.res' failed
Building with --verbose shows:

 0:20.16 PATH=../lib:../stubdata:../tools/ctestfw:$PATH  ../bin/genrb --usePoolBundle -k/ -i ./out/build/icudt52l -s e:/dev/fx-team/intl/icu/source/data/locales -d ./out/build/icudt52l root.txt


The forward slash after the -k gets resolved to the msys root, which is what ends up being shown in the error, it looks like...
Confirmed by local backout that this is due to bug 1033958.
Blocks: 1033958
Mike / Greg, any idea what's up here?
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(gps)
Fixed by backing out bug 1033958:

remote:   https://hg.mozilla.org/mozilla-central/rev/e268bb51e084
Status: NEW → RESOLVED
Closed: 8 years ago
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(gps)
Resolution: --- → FIXED
O_O you'll have to tell me where this / comes from because here's what the source tree says:
http://mxr.mozilla.org/mozilla-central/search?string=usePoolBundle
The ones in makedata.mak have "-k" with nothing.
The ones in Makefile.in don't have a visiblt "-k", but they have $(GENRBOPTS) after --usePoolBundle, and that comes from there:
http://mxr.mozilla.org/mozilla-central/source/intl/icu/Makefile.in#57
where, as you can see, GENRBOPTS is set to "-k -R"
I'll add this: bug 1033958 changed nothing about how icu is built.
And of course, I can't reproduce on 74d9bc471cc5, which is the changeset before the backout. The only reason the backout worked for you is that it triggered a reconfigure, which fixed up whatever broken state you were in. I still don't see how that could ever happen, considering the build rules. But since you're not giving any more detail about your build environment... (mozconfig, paths, etc)
I seriously doubt whatever problem was there was actually fixed by the backout. I backed out the backout anyways.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
My original build that failed was from a full source clobber built on the same changeset as today's nightly.  Backing out the fix for bug 1033958 fixed the issue for me.

my .mozconfig looks as follows:

BUILD_OFFICIAL=1
export BUILD_OFFICIAL
WIN32_REDIST_DIR=~/mozilla/redist/x86
export WIN32_REDIST_DIR
#WIN32_CRT_SRC_DIR=~/mozilla/crtsrc
mk_add_options BUILD_OFFICIAL=1
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/fx-obj
mk_add_options RUN_AUTOCONF_LOCALLY=1
mk_add_options AUTOCONF=/usr/local/bin/autoconf-2.13
ac_add_options --enable-optimize=-O2
ac_add_options --disable-debug
ac_add_options --enable-tests
ac_add_options --disable-crashreporter
ac_add_options --disable-metro
ac_add_options --enable-jemalloc
ac_add_options --without-x
mk_add_options MOZ_MAKE_FLAGS="-j4"
(In reply to Bill Gianopoulos [:WG9s] from comment #10)
> My original build that failed was from a full source clobber built on the
> same changeset as today's nightly.  Backing out the fix for bug 1033958
> fixed the issue for me.
> 
> my .mozconfig looks as follows:
> 
> BUILD_OFFICIAL=1
> export BUILD_OFFICIAL
> WIN32_REDIST_DIR=~/mozilla/redist/x86
> export WIN32_REDIST_DIR
> #WIN32_CRT_SRC_DIR=~/mozilla/crtsrc
> mk_add_options BUILD_OFFICIAL=1
> mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/fx-obj
> mk_add_options RUN_AUTOCONF_LOCALLY=1
> mk_add_options AUTOCONF=/usr/local/bin/autoconf-2.13
> ac_add_options --enable-optimize=-O2
> ac_add_options --disable-debug
> ac_add_options --enable-tests
> ac_add_options --disable-crashreporter
> ac_add_options --disable-metro
> ac_add_options --enable-jemalloc
> ac_add_options --without-x
> mk_add_options MOZ_MAKE_FLAGS="-j4"

Building 81691a55e60f with that mozconfig (except for the WIN32_REDIST_DIR lines) works for me.
My local build from the clobber also fails with bug 1033958 and succeeds with the backout.
.mozconfig:
ac_add_options --enable-warnings-as-errors
ac_add_options --disable-optimize
ac_add_options --enable-metro
mk_add_options AUTOCLOBBER=1
WIN32_REDIST_DIR="$VCINSTALLDIR\redist\x86\Microsoft.VC120.CRT"
To anyone who can reproduce this issue, I'd need the *complete* log for the following commands, starting from a clobber:
- mozmake -f client.mk configure
- mozmake -C whatever-your-objdir-is/intl/icu buildicu
as well as the contents of config.status and .mozconfig.json from the objdir.
(In reply to Mike Hommey [:glandium] from comment #8)
> And of course, I can't reproduce on 74d9bc471cc5, which is the changeset
> before the backout. The only reason the backout worked for you is that it
> triggered a reconfigure, which fixed up whatever broken state you were in.

No, because I clobbered every time I built trying to figure out where the culprit for the problems I was seeing was located. In other words, every build had a new configure, and every build with bug 1033958 broke, and every build without it succeeded.

> I
> still don't see how that could ever happen, considering the build rules. But
> since you're not giving any more detail about your build environment...
> (mozconfig, paths, etc)

You weren't around to ask for them, and so many people's builds broke that folks in #developers thought backing out was a better option than waiting (especially as Greg is now in the middle of a long july 4th weekend, and your weekend already started as well).

As noted already, several people had this issue, all with varying mozconfigs... my own mozconfig is almost empty, save for defining an objdir. I'll add it in a bit when I've booted up my Windows machine - but it seems unlikely it's related to mozconfig; more like it's related to mozmake/mozillabuild/something else.

(In reply to Mike Hommey [:glandium] from comment #6)
> O_O you'll have to tell me where this / comes from because here's what the
> source tree says:
> http://mxr.mozilla.org/mozilla-central/search?string=usePoolBundle
> The ones in makedata.mak have "-k" with nothing.
> The ones in Makefile.in don't have a visiblt "-k", but they have
> $(GENRBOPTS) after --usePoolBundle, and that comes from there:
> http://mxr.mozilla.org/mozilla-central/source/intl/icu/Makefile.in#57
> where, as you can see, GENRBOPTS is set to "-k -R"

If I'd known where it'd come from, I would have tried fixing it. As it was, I had enough evidence that it was caused by this patch, and that it affected a number of people, all with the same symptoms, and several people in #developers who suggested we should back the problematic change out, so that was what I did.
MOZCONFIG:
Gijs@HERMES /e/dev/fx-team
$ cat $MOZCONFIG
export MOZ_PSEUDO_DERECURSE=1
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../builds/opt-@CONFIG_GUESS@

ac_add_options --enable-profiling
Attached file configure-output.txt
Attached file buildicu-output.txt
(In reply to :Gijs Kruitbosch from comment #17)
> Created attachment 8451228 [details]
> buildicu-output.txt

This isn't failing... what the hell?

Can you attach a complete log of a clobber make -f client.mk (if it fails)?
Attached file ./mach clobber && ./mach build (obsolete) —
Maybe it has to do with  0:20.27 checking for E:/mozilla-build/msys/bin/mozmake.EXE... e:/dev/fx-team/configure: line 5420: test: /e/tortoisehg//E: binary operator expected ? iow, if it's using something other than mozmake for icu...
Also, what is the output of mozmake --version?
(In reply to :Gijs Kruitbosch from comment #19)
> Created attachment 8451230 [details]
> ./mach clobber && ./mach build

The output of mach build is useless. Try again with -v, but try make -f client.mk first.
Gijs@HERMES /e/dev/fx-team
$ make -f client.mk build &> plain-make-clientmk-build.txt

Gijs@HERMES /e/dev/fx-team
$ cat plain-make-clientmk-build.txt
make -f /e/dev/fx-team/client.mk realbuild CREATE_MOZCONFIG_JSON=
Adding client.mk options from e:/dev/configs/opt.mozconfig:
    MOZ_OBJDIR=e:/dev/builds/opt-i686-pc-mingw32
    OBJDIR=e:/dev/builds/opt-i686-pc-mingw32
    FOUND_MOZCONFIG=e:/dev/configs/opt.mozconfig
make[1]: Entering directory `/e/dev/fx-team'
make -j8 -C e:/dev/builds/opt-i686-pc-mingw32
make[2]: Entering directory `/e/dev/builds/opt-i686-pc-mingw32'
e:/dev/fx-team/config/baseconfig.mk:35: *** MSYS make is not supported.  Stop.
make[2]: Leaving directory `/e/dev/builds/opt-i686-pc-mingw32'
make[1]: *** [realbuild] Error 2
make[1]: Leaving directory `/e/dev/fx-team'
make: *** [build] Error 2

(NB: this was without an extra clobber after the previous (clobber) ./mach build... would that make much of a difference?)

Gijs@HERMES /e/dev/fx-team
$ mozmake --version
GNU Make 4.0-23-g9d58570
Built for Windows32
Copyright (C) 1988-2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Gijs@HERMES /e/dev/fx-team
$ make --version
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i686-pc-msys
(In reply to :Gijs Kruitbosch from comment #20)
> Maybe it has to do with  0:20.27 checking for
> E:/mozilla-build/msys/bin/mozmake.EXE... e:/dev/fx-team/configure: line
> 5420: test: /e/tortoisehg//E: binary operator expected ? iow, if it's using
> something other than mozmake for icu...

mozmake.EXE[7]: *** [out/build/icudt52l/root.res] Error 1

That says it *is* using mozmake.
(In reply to :Gijs Kruitbosch from comment #23)
> Gijs@HERMES /e/dev/fx-team
> $ make -f client.mk build &> plain-make-clientmk-build.txt
> 
> Gijs@HERMES /e/dev/fx-team
> $ cat plain-make-clientmk-build.txt
> make -f /e/dev/fx-team/client.mk realbuild CREATE_MOZCONFIG_JSON=
> Adding client.mk options from e:/dev/configs/opt.mozconfig:
>     MOZ_OBJDIR=e:/dev/builds/opt-i686-pc-mingw32
>     OBJDIR=e:/dev/builds/opt-i686-pc-mingw32
>     FOUND_MOZCONFIG=e:/dev/configs/opt.mozconfig
> make[1]: Entering directory `/e/dev/fx-team'
> make -j8 -C e:/dev/builds/opt-i686-pc-mingw32
> make[2]: Entering directory `/e/dev/builds/opt-i686-pc-mingw32'
> e:/dev/fx-team/config/baseconfig.mk:35: *** MSYS make is not supported. 
> Stop.
> make[2]: Leaving directory `/e/dev/builds/opt-i686-pc-mingw32'
> make[1]: *** [realbuild] Error 2
> make[1]: Leaving directory `/e/dev/fx-team'
> make: *** [build] Error 2

Err sorry, I meant mozmake -f client.mk.
> Gijs@HERMES /e/dev/fx-team
> $ mozmake --version
> GNU Make 4.0-23-g9d58570
> Built for Windows32
> Copyright (C) 1988-2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.

Try upgrading MozillaBuild. You should have version 4.0-43-g8411528
(I'm running a clobber && mozmake -f client.mk build right now, but I need to run out. Will post the results of that and a mozillabuild update later today)
Note that maybe you have MozillaBuild 1.9, but your path picks an old mozmake you installed yourself:
E:/mozilla-build/msys/bin/mozmake.EXE

The one from MozillaBuild is in mozilla-build/mozmake.
Note on my end I'm trying a build with mozmake 4.0-23-g9d58570.
Ok, I could reproduce the failure with 4.0-23-g9d58570.
Attached file clobber-plain-mk.txt
Attachment #8451230 - Attachment is obsolete: true
(In reply to Mike Hommey [:glandium] from comment #28)
> Note that maybe you have MozillaBuild 1.9, but your path picks an old
> mozmake you installed yourself:
> E:/mozilla-build/msys/bin/mozmake.EXE
> 
> The one from MozillaBuild is in mozilla-build/mozmake.

This is the case for me. I'll try a clobber and rebuild with the other mozmake.
I'm now able to reproduce, even with mozmake . I don't know what changed, but I'm using a different AWS instance now than i was using earlier, so maybe this is related. Paths changed, too. Anyways, this looks like a bad bug in (moz)make on Windows.
So, the problem lies, in fact, in msys, and it doesn't happen consistently (if it did, we'd have had busted windows builds on tbpl, and my attempts at reproducing would all have worked).

The minimal testcase is this:

$ cat > Makefile <<EOF
foo:
        $(MAKE) bar A='a a'

bar:
        (cd . && $(MAKE) baz)

baz:
        echo $(A)
EOF
$ mozmake B=

This displays "a/" instead of "a a" you'd get with "mozmake" (without B=)

The problem is make sets MAKEFLAGS to contain "A=a\ a" when executing the rule for bar, and msys transforms that to "A=a/ a"

Luckily, genrb allows flags to be stuck together, so we can change GENRBOPTS to not have spaces, working around the issue.
Assignee: nobody → mh+mozilla
Status: REOPENED → ASSIGNED
(In reply to Mike Hommey [:glandium] from comment #34)
> The problem is make sets MAKEFLAGS to contain "A=a\ a" when executing the
> rule for bar, and msys transforms that to "A=a/ a"

Note that fiddling with the name of variables and their values makes things change.
(In reply to Mike Hommey [:glandium] from comment #34)
> So, the problem lies, in fact, in msys, and it doesn't happen consistently
> (if it did, we'd have had busted windows builds on tbpl, and my attempts at
> reproducing would all have worked).
> 
> The minimal testcase is this:
> 
> $ cat > Makefile <<EOF
> foo:
>         $(MAKE) bar A='a a'
> 
> bar:
>         (cd . && $(MAKE) baz)
> 
> baz:
>         echo $(A)
> EOF
> $ mozmake B=
> 
> This displays "a/" instead of "a a" you'd get with "mozmake" (without B=)
> 
> The problem is make sets MAKEFLAGS to contain "A=a\ a" when executing the
> rule for bar, and msys transforms that to "A=a/ a"
> 
> Luckily, genrb allows flags to be stuck together, so we can change GENRBOPTS
> to not have spaces, working around the issue.

So it would seem it thinks this is a file path, so is replacing windows '\' path separators with '/'.
(In reply to Mike Hommey [:glandium] from comment #33)
> I'm now able to reproduce, even with mozmake . I don't know what changed,
> but I'm using a different AWS instance now than i was using earlier, so
> maybe this is related. Paths changed, too. Anyways, this looks like a bad
> bug in (moz)make on Windows.

I'm assuming you meant a newer mozmake? I'm indeed still seeing the same issue with the newer version of mozmake...
(In reply to Mike Hommey [:glandium] from comment #35)
> Created attachment 8451247 [details] [diff] [review]
> Work around msys silliness breaking the build for some people after bug
> 1033958

My build worked fine today with this patch applied.
Comment on attachment 8451247 [details] [diff] [review]
Work around msys silliness breaking the build for some people after bug 1033958

Review of attachment 8451247 [details] [diff] [review]:
-----------------------------------------------------------------

This fixed my build too.
Attachment #8451247 - Flags: feedback+
Summary: Trunk Build fails with Makefile:723: recipe for target 'out/build/icudt52l/root.res' failed → Trunk Build fails with Makefile:723: recipe for target 'out/build/icudt52l/root.res' failed (genrb.exe)
Comment on attachment 8451247 [details] [diff] [review]
Work around msys silliness breaking the build for some people after bug 1033958

Review of attachment 8451247 [details] [diff] [review]:
-----------------------------------------------------------------

Another reminder that the BS abbreviation for Build System is appropriate. Hacks all the way down.
Attachment #8451247 - Flags: review?(gps) → review+
Duplicate of this bug: 1035340
https://hg.mozilla.org/mozilla-central/rev/93e07b92f879
Status: ASSIGNED → RESOLVED
Closed: 8 years ago8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla33
Duplicate of this bug: 1062201
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.