801489 - Various Windows builds stuck in infinite loops due to bug in makefiles

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Hal Wine [:hwine] use NI!]

The defect underlying bug 800847 was present in production long enough to get at a large number of windows builders hung in an infinite loop. That effectively removed those machines from the pool, and led to noticeable wait times even on the weekend.

Comment 1

[Hal Wine [:hwine] use NI!]

Symptom was the following in build log files:
    Updating projects from gyp files...
    e:\builds\moz2_slave\m-in-w32-dbg\build\obj-firefox\media\webrtc\signaling\common.mk:59:0$ e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/_virtualenv/Scripts/python.exe ../../../../media/webrtc/trunk/build/gyp_chromium --format=mozmake --depth=../../../../media/webrtc/trunk --generator-output=../../../media/webrtc/signaling --toplevel-dir=../../../.. -G OBJDIR=../../..  -Dbuild_with_mozilla=1 -DMSVS_VERSION=2010 -DMSVS_OS_BITS=32 -Dbuild_for_test=0 ../../../../media/webrtc/signaling/signaling.gyp

repeating forEVAH

Impacted the following 43 windows build hosts:

    w64-ix-slave06
    w64-ix-slave07
    w64-ix-slave08
    w64-ix-slave09
    w64-ix-slave12
    w64-ix-slave13
    w64-ix-slave14
    w64-ix-slave15
    w64-ix-slave16
    w64-ix-slave17
    w64-ix-slave18
    w64-ix-slave19
    w64-ix-slave24
    w64-ix-slave64
    w64-ix-slave65
    w64-ix-slave66
    w64-ix-slave67
    w64-ix-slave68
    w64-ix-slave69
    w64-ix-slave70
    w64-ix-slave72
    w64-ix-slave73
    w64-ix-slave75
    w64-ix-slave77
    w64-ix-slave78
    w64-ix-slave81
    w64-ix-slave82
    w64-ix-slave84
    w64-ix-slave88
    w64-ix-slave89
    w64-ix-slave91
    w64-ix-slave92
    w64-ix-slave94 
    w64-ix-slave95
    w64-ix-slave97
    w64-ix-slave101
    w64-ix-slave102
    w64-ix-slave104
    w64-ix-slave105
    w64-ix-slave106
    w64-ix-slave107
    w64-ix-slave108
    w64-ix-slave109

Comment 2

[Hal Wine [:hwine] use NI!]

All above machines had the job cancelled via buildbot, then rebooted the hosts.

Running jobs reports no more hanging w64 builder jobs