462573 - Try server Windows build machine can't build

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[(no longer active)]

Here's the email I sent to [e-mail] about the problem I've observed:

I've tried pushing some changesets to the try server for the Private
Browsing work (bug 248970) and they're failing all the time on the
Windows build machine.  I don't have a clue why but my patch doesn't
tocuh anything near the failure location.  I also noted other people's
builds were busted with the same error so I thought this might be a
problem with the build machine.

Here are some logs for further investigation:

<http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1225409278.1225411625.23497.gz>
<http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1225440358.1225441993.32021.gz>
<http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1225414198.1225416016.2626.gz>
<http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1225444378.1225446098.9671.gz>

Looking at the tinderbox
<http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTry&maxdate=1225416478&hours=24&legend=0&norules=1>,
the first time I see this happening is with this log
<http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1225392618.1225394611.10927.gz>
on 2008/10/30 12:17.  After that a single build has been successful
<http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1225398598.1225404083.4169.gz>
but all other builds on this machine have failed with the same error.


Benjamin's reply:

The actual error message is:

/bin/perl
/e/builds/sendchange-slave/sendchange-win32-hg/mozilla/build/autoconf/make-makefile
- -t /e/builds/sendchange-slave/sendchange-win32-hg/mozilla -d ..
mathml/Makefile
creating content/mathml/Makefile
make[4]: Leaving directory
`/e/builds/sendchange-slave/sendchange-win32-hg/mozilla/objdir/content'
make[3]: Leaving directory
`/e/builds/sendchange-slave/sendchange-win32-hg/mozilla/objdir'
make[2]: Leaving directory
`/e/builds/sendchange-slave/sendchange-win32-hg/mozilla/objdir'
make[4]: *** No rule to make target `|', needed by `export'.  Stop.

This is caused by the PARALLEL_DIRS change... this uses order-only
prerequisites, which was a feature added in GNU make 3.80.

This indicates to me that the tryserver slave is either using a very old
version of MozillaBuild or for some reason has the wrong copy of make on the
path. And reading the build log, this does indeed seem to be the case:

PATH=d:\sdks\v6.0\bin;D:\msvs8\Common7\IDE;D:\msvs8\VC\bin;C:\msvs8\SDK\bin;D:\msvs8\VC;D:\msvs8\Common7\Tools;D:\msvs8\Common7\Tools\bin;d:\mercurial;d:\mozilla-build\moztools\bin;d:\mozilla-build\msys\bin;d:\mozilla-build\msys\local\bin;d:\mozilla-build\7zip;d:\mozilla-build\upx203w;d:\mozilla-build\python25;d:\mozilla-build\blat261\full;d:\mozilla-build\info-zip;d:\mozilla-build\wget;d:\mozilla-build\nsis-2.22;
PATHEXT

d:\mozilla-build\msys\bin;d:\mozilla-build\msys\local\bin is the wrong
order: local\bin should be first. The current mozillabuild start scripts use
the correct order, which is why I think we must be using an old
mozillabuild, or launching mozillabuild in an unusual manner for that slave:

http://mxr.mozilla.org/mozilla/source/tools/build-environment/win32/profile-extrapaths.sh#5

Comment 1

[(not currently active) Ted Mielczarek]

Moving this over to RelEng, as it really deals more with the build configuration than the Try Server webapp.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Quick fix

This is the quick fix following Benjamin's comments, which I've checked in and deployed. According to 
  http://mxr.mozilla.org/mozilla/search?string=win32-ref-platform
and greping over build-configs in hg, the tryserver is the only consumer of this set of environment variables.

The codepath is 
  http://mxr.mozilla.org/seamonkey/source/tools/buildbot-configs/tryserver/master.cfg#681
  http://mxr.mozilla.org/seamonkey/source/tools/buildbotcustom/tryserver/env.py#5
  http://mxr.mozilla.org/seamonkey/source/tools/buildbotcustom/env.py

I've also confirmed that /d/mozilla-build/msys/local/bin/make is v3.81.90

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Build got past content/ OK.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

... and completed OK, so we're back in action. Leaving open to confirm review, and maybe discuss how to handle this type of issue in future.