466328 - Investigate intermittent talos failures mozilla-central

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Particularly on Mac, but also on windows, there have been alot of Ts and Tgfx failures on Talos as the watchdog thread patch gets landed and backed out (bug 453157). We should figure out what's going on.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Mac Talos data:

build start - slave - clobber/dep - revision (watchdog in/out) - talos results
22/11 11:27 - moz2-darwin9-slave02 - dep - flaf606... (out) - 5x talos Ts fail
22/11 09:27 - bm-xserve18 - dep - 5d3767... (out) - 5x talos Ts fail
22/11 07:36 - bm-xserve17 - clobber - 5d3767.. (out) - 5x talos green
22/11 05:27 - bm-xserve18 - dep - 04cecb... (in) - 2 green, 2 Ts fail, 1 Tp crash
22/11 03:27 - moz2-darwin9-slave02 - dep - 04cecb... (in) - 2 green, 1 Tgfx fail
22/11 01:27 - bm-xserve18 - dep - 04cecb... (in) - 3 green, 2 Tgfx fail

Working hypothesis - from the 07:36 build, a clobber is required to make the patch apply/unapply properly (something about the patch doesn't get rebuilt properly). So I've clobbered the the builds on bm-xserve16,17,18,19,moz2-darwin9-slave02 (both optimised and leak test).

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Also clobbered the build (but not leak) on moz2-win32-slave1,02,17,18, since those have all produced builds that Talos barfed on in the last few hours.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Ok, all is green now (except for random unit test failures).

I'm not sure what is causing the Mac builds to be broken when bug 453157 lands and is backed out, except for maybe the start of build clean up not working as well there. The build system does 
 make  -C obj-firefox/ppc
 rm -f -rf ./dist/sdk
 rm -f -rf ./dist/include
 rm -f -rf ./dist/private
 rm -f -rf ./dist/public
 rm -f -rf ./dist/bin/components
 rm -f -rf _tests
and the same for i386 in it's turn. The universal build is created elsewhere though. bug 455940 is related.

If you take a universal mac build without the patch from bug 453157, are any steps necessary for the patch to build cleanly ? Bear in mind that "make package" in <objdir>/ppc/ will give you a dmg that may differ slightly from what is in dist/bin.