Closed Bug 820769 Opened 13 years ago Closed 13 years ago

Intermittent disconnect during Windows builds in WebRTC "Updating projects from gyp files..."

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Platform:
x86
Windows 7
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
mozilla20

People

(Reporter: emorley, Assigned: ted)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

fix mozmake.py to generate correct common.mk include
7.95 KB, patch
jesup
: review+
Details | Diff | Splinter Review
WINNT 5.2 mozilla-inbound leak test build on 2012-12-11 05:11:07 PST for push 88fd42043056 slave: w64-ix-slave94 https://tbpl.mozilla.org/php/getParsedLog.php?id=17834416&tree=Mozilla-Inbound { Updating projects from gyp files... e:\builds\moz2_slave\m-in-w32-dbg\build\obj-firefox\media\webrtc\signaling\common.mk:59:0$ e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/_virtualenv/Scripts/python.exe ../../../../media/webrtc/trunk/build/gyp_chromium --format=mozmake --include=e:/builds/moz2_slave/m-in-w32-dbg/build/media/webrtc/webrtc_config.gypi --depth=../../../../media/webrtc/trunk --generator-output=../../../media/webrtc/signaling --toplevel-dir=../../../.. -G OBJDIR=../../.. -Dbuild_with_mozilla=1 -DFORCED_INCLUDE_FILE=e:/builds/moz2_slave/m-in-w32-dbg/build/media/webrtc/webrtc_config.gypi -Dtarget_arch=ia32 -DMSVS_VERSION=2010 -DMSVS_OS_BITS=32 -Dbuild_for_test=0 ../../../../media/webrtc/signaling/signaling.gyp Updating projects from gyp files... e:\builds\moz2_slave\m-in-w32-dbg\build\obj-firefox\media\webrtc\signaling\common.mk:59:0$ e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/_virtualenv/Scripts/python.exe ../../../../media/webrtc/trunk/build/gyp_chromium --format=mozmake --include=e:/builds/moz2_slave/m-in-w32-dbg/build/media/webrtc/webrtc_config.gypi --depth=../../../../media/webrtc/trunk --generator-output=../../../media/webrtc/signaling --toplevel-dir=../../../.. -G OBJDIR=../../.. -Dbuild_with_mozilla=1 -DFORCED_INCLUDE_FILE=e:/builds/moz2_slave/m-in-w32-dbg/build/media/webrtc/webrtc_config.gypi -Dtarget_arch=ia32 -DMSVS_VERSION=2010 -DMSVS_OS_BITS=32 -Dbuild_for_test=0 ../../../../media/webrtc/signaling/signaling.gyp Updating projects from gyp files... e:\builds\moz2_slave\m-in-w32-dbg\build\obj-firefox\media\webrtc\signaling\common.mk:59:0$ e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/_virtualenv/Scripts/python.exe ../../../../media/webrtc/trunk/build/gyp_chromium --format=mozmake --include=e:/builds/moz2_slave/m-in-w32-dbg/build/media/webrtc/webrtc_config.gypi --depth=../../../../media/webrtc/trunk --generator-output=../../../media/webrtc/signaling --toplevel-dir=../../../.. -G OBJDIR=../../.. -Dbuild_with_mozilla=1 -DFORCED_INCLUDE_FILE=e:/builds/moz2_slave/m-in-w32-dbg/build/media/webrtc/webrtc_config.gypi -Dtarget_arch=ia32 -DMSVS_VERSION=2010 -DMSVS_OS_BITS=32 -Dbuild_for_test=0 ../../../../media/webrtc/signaling/signaling.gyp Updating projects from gyp files... e:\builds\moz2_slave\m-in-w32-dbg\build\obj-firefox\media\webrtc\signaling\common.mk:59:0$ e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/_virtualenv/Scripts/python.exe ../../../../media/webrtc/trunk/build/gyp_chromium --format=mozmake --include=e:/builds/moz2_slave/m-in-w32-dbg/build/media/webrtc/webrtc_config.gypi --depth=../../../../media/webrtc/trunk --generator-output=../../../media/webrtc/signaling --toplevel-dir=../../../.. -G OBJDIR=../../.. -Dbuild_with_mozilla=1 -DFORCED_INCLUDE_FILE=e:/builds/moz2_slave/m-in-w32-dbg/build/media/webrtc/webrtc_config.gypi -Dtarget_arch=ia32 -DMSVS_VERSION=2010 -DMSVS_OS_BITS=32 -Dbuild_for_test=0 ../../../../media/webrtc/signaling/signaling.gyp Updating projects from gyp files... e:\builds\moz2_slave\m-in-w32-dbg\build\obj-firefox\media\webrtc\signaling\common.mk:59:0$ e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/_virtualenv/Scripts/python.exe ../../../../media/webrtc/trunk/build/gyp_chromium --format=mozmake --include=e:/builds/moz2_slave/m-in-w32-dbg/build/media/webrtc/webrtc_config.gypi --depth=../../../../media/webrtc/trunk --generator-output=../../../media/webrtc/signaling --toplevel-dir=../../../.. -G OBJDIR=../../.. -Dbuild_with_mozilla=1 -DFORCED_INCLUDE_FILE=e:/builds/moz2_slave/m-in-w32-dbg/build/media/webrtc/webrtc_config.gypi -Dtarget_arch=ia32 -DMSVS_VERSION=2010 -DMSVS_OS_BITS=32 -Dbuild_for_test=0 ../../../../media/webrtc/signaling/signaling.gyp Updating projects from gyp files... e:\builds\moz2_slave\m-in-w32-dbg\build\obj-firefox\media\webrtc\signaling\common.mk:59:0$ e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/_virtualenv/Scripts/python.exe ../../../../media/webrtc/trunk/build/gyp_chromium --format=mozmake --include=e:/builds/moz2_slave/m-in-w32-dbg/build/media/webrtc/webrtc_config.gypi --depth=../../../../media/webrtc/trunk --generator-output=../../../media/webrtc/signaling --toplevel-dir=../../../.. -G OBJDIR=../../.. -Dbuild_with_mozilla=1 -DFORCED_INCLUDE_FILE=e:/builds/moz2_slave/m-in-w32-dbg/build/media/webrtc/webrtc_config.gypi -Dtarget_arch=ia32 -DMSVS_VERSION=2010 -DMSVS_OS_BITS=32 -Dbuild_for_test=0 ../../../../media/webrtc/signaling/signaling.gyp Updating projects from gyp files... remoteFailed: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. }
Note total runtime is crazy: ========= Finished compile interrupted (results: 4, elapsed: 7 hrs, 27 mins, 42 secs) (at 2012-12-11 12:52:22.150724) =========
catlee, should we be aborting the build sooner than 7+ hrs in? (comment 1)
WINNT 5.2 mozilla-inbound leak test build on 2012-12-11 05:23:00 PST for push e594a31444ee slave: w64-ix-slave72 https://tbpl.mozilla.org/php/getParsedLog.php?id=17834294&tree=Mozilla-Inbound WINNT 5.2 mozilla-inbound build on 2012-12-11 05:23:00 PST for push e594a31444ee slave: w64-ix-slave81 https://tbpl.mozilla.org/php/getParsedLog.php?id=17834355&tree=Mozilla-Inbound WINNT 5.2 mozilla-inbound leak test build on 2012-12-11 06:42:00 PST for push 6fe9b3023000 slave: w64-ix-slave88 https://tbpl.mozilla.org/php/getParsedLog.php?id=17834397&tree=Mozilla-Inbound WINNT 5.2 mozilla-inbound pgo-build on 2012-12-11 07:44:24 PST for push 1114382b1f56 slave: w64-ix-slave84 https://tbpl.mozilla.org/php/getParsedLog.php?id=17847091&tree=Mozilla-Inbound WINNT 5.2 mozilla-inbound build on 2012-12-11 07:55:20 PST for push e84e0de4949d slave: w64-ix-slave08 https://tbpl.mozilla.org/php/getParsedLog.php?id=17846504&tree=Mozilla-Inbound WINNT 5.2 mozilla-inbound leak test build on 2012-12-11 08:26:00 PST for push db376fcebe62 slave: w64-ix-slave71 https://tbpl.mozilla.org/php/getParsedLog.php?id=17846966&tree=Mozilla-Inbound WINNT 5.2 mozilla-inbound build on 2012-12-11 09:56:07 PST for push 3bb5a5a91af6 slave: w64-ix-slave91 https://tbpl.mozilla.org/php/getParsedLog.php?id=17847151&tree=Mozilla-Inbound WINNT 5.2 mozilla-inbound build on 2012-12-11 10:11:13 PST for push 2fb469965c13 slave: w64-ix-slave87 https://tbpl.mozilla.org/php/getParsedLog.php?id=17847122&tree=Mozilla-Inbound WINNT 5.2 mozilla-inbound leak test build on 2012-12-11 10:13:13 PST for push bf46c86deb67 slave: w64-ix-slave76 https://tbpl.mozilla.org/php/getParsedLog.php?id=17847028&tree=Mozilla-Inbound
This is hitting some sort of infinite loop in Makefile regeneration from gyp files. This is unfortunate.
This is a bug in mozmake.py. We're building two sets of Makefiles from one gyp file (signaling.gyp), but it writes out an include to the same common.mk from both, so building in signalingtest rebuilds the Makefiles in signaling instead.
Assignee: nobody → ted
Ugly, but no uglier than the rest of mozmake.py. This fixes things locally. Before if I would touch $srcdir/media/webrtc/signaling/signaling.gyp then make in $objdir/media/webrtc/signalingtest, it would run GYP but rebuild the Makefiles in signaling, not signalingtest. Now it rebuilds the correct Makefiles.
Attachment #691367 - Flags: review?(rjesup)
(In reply to Ed Morley [UTC+0; email:edmorley@moco] from comment #2) > catlee, should we be aborting the build sooner than 7+ hrs in? (comment 1) what's a reasonable upper limit for builds these days? 7 hours probably comes from the era of PGO-builds-on-slow-VMs.
Attachment #691367 - Flags: review?(rjesup) → review+
(In reply to Chris AtLee [:catlee] from comment #11) > (In reply to Ed Morley [UTC+0; email:edmorley@moco] from comment #2) > > catlee, should we be aborting the build sooner than 7+ hrs in? (comment 1) > > what's a reasonable upper limit for builds these days? 7 hours probably > comes from the era of PGO-builds-on-slow-VMs. Win PGO is the long pole - looking at the last half dozen jobs on m-c (mins): 224, 231, 217, 225, 222, 217 Happy for me to file a bug to lower max runtime to say 270 mins/4.5 hrs?
(In reply to Ed Morley [UTC+0; email:edmorley@moco] from comment #12) > Win PGO is the long pole - looking at the last half dozen jobs on m-c (mins): > 224, 231, 217, 225, 222, 217 > > Happy for me to file a bug to lower max runtime to say 270 mins/4.5 hrs? Can't that be even lower? (assuming the timeout applies to lack of output)
(In reply to Mike Hommey [:glandium] from comment #14) > Can't that be even lower? (assuming the timeout applies to lack of output) There are both 'max runtime' & also 'no output' timeouts. This bug still caused output to be generated, but still should have hit the max runtime.
(In reply to Ed Morley [UTC+0; email:edmorley@moco] from comment #12) > (In reply to Chris AtLee [:catlee] from comment #11) > > (In reply to Ed Morley [UTC+0; email:edmorley@moco] from comment #2) > > > catlee, should we be aborting the build sooner than 7+ hrs in? (comment 1) > > > > what's a reasonable upper limit for builds these days? 7 hours probably > > comes from the era of PGO-builds-on-slow-VMs. > > Win PGO is the long pole - looking at the last half dozen jobs on m-c (mins): > 224, 231, 217, 225, 222, 217 > > Happy for me to file a bug to lower max runtime to say 270 mins/4.5 hrs? Sounds good, thanks! This was set to 3 hours in http://hg.mozilla.org/build/buildbotcustom/rev/65e500d98367. Can we set a maxTime of 4.5 hours, and lower the timeout back down to 2 hours?
https://hg.mozilla.org/mozilla-central/rev/0cf8797b8fc7
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla20
(In reply to Chris AtLee [:catlee] from comment #16) > Can we set a maxTime of 4.5 hours, and lower the timeout back down to 2 > hours? Sure; filed bug 822821.
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: