Various Windows builds stuck in infinite loops due to bug in makefiles

RESOLVED FIXED

Status

Product:
Release Engineering
Component:
General
Status:
RESOLVED FIXED
Reported:
6 years ago
Modified:
5 years ago

People

(Reporter: hwine, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details
(Reporter)

Description

•
6 years ago 
The defect underlying bug 800847 was present in production long enough to get at a large number of windows builders hung in an infinite loop. That effectively removed those machines from the pool, and led to noticeable wait times even on the weekend.
(Reporter)

Comment 1

•
6 years ago 
Symptom was the following in build log files:
    Updating projects from gyp files...
    e:\builds\moz2_slave\m-in-w32-dbg\build\obj-firefox\media\webrtc\signaling\common.mk:59:0$ e:/builds/moz2_slave/m-in-w32-dbg/build/obj-firefox/_virtualenv/Scripts/python.exe ../../../../media/webrtc/trunk/build/gyp_chromium --format=mozmake --depth=../../../../media/webrtc/trunk --generator-output=../../../media/webrtc/signaling --toplevel-dir=../../../.. -G OBJDIR=../../..  -Dbuild_with_mozilla=1 -DMSVS_VERSION=2010 -DMSVS_OS_BITS=32 -Dbuild_for_test=0 ../../../../media/webrtc/signaling/signaling.gyp

repeating forEVAH

Impacted the following 43 windows build hosts:

    w64-ix-slave06
    w64-ix-slave07
    w64-ix-slave08
    w64-ix-slave09
    w64-ix-slave12
    w64-ix-slave13
    w64-ix-slave14
    w64-ix-slave15
    w64-ix-slave16
    w64-ix-slave17
    w64-ix-slave18
    w64-ix-slave19
    w64-ix-slave24
    w64-ix-slave64
    w64-ix-slave65
    w64-ix-slave66
    w64-ix-slave67
    w64-ix-slave68
    w64-ix-slave69
    w64-ix-slave70
    w64-ix-slave72
    w64-ix-slave73
    w64-ix-slave75
    w64-ix-slave77
    w64-ix-slave78
    w64-ix-slave81
    w64-ix-slave82
    w64-ix-slave84
    w64-ix-slave88
    w64-ix-slave89
    w64-ix-slave91
    w64-ix-slave92
    w64-ix-slave94 
    w64-ix-slave95
    w64-ix-slave97
    w64-ix-slave101
    w64-ix-slave102
    w64-ix-slave104
    w64-ix-slave105
    w64-ix-slave106
    w64-ix-slave107
    w64-ix-slave108
    w64-ix-slave109
(Reporter)

Comment 2

•
6 years ago 
All above machines had the job cancelled via buildbot, then rebooted the hosts.

Running jobs reports no more hanging w64 builder jobs
 (Reporter)

Updated

•
6 years ago
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
 (Assignee)

Updated

•
5 years ago
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.