514615 - MozillaTry tree is broken

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

It's blank, or has some builders, but not the full set. Tinderbox seems to be very confused about what is "Current" (see the admin page).

Timeline:
* Talos boxes starting hanging around 1430 PDT Wednesday, and were fixedup between 2030 and 2100 PDT Thursday
* Immediately prior to the fix up all the builds marked (*) were present on the waterfall, the rest of the talos boxes had dropped of the waterfall as normal (more than 12 hours)
* Shortly after 2115 there was no build data shown on the waterfall, then some things like "Linug try hg build" show up, then disapper again

The complete set of active machines should be
Linux try hg build (*)
Linux try hg unit test (*)
Linux try talos
Linux try talos dirty
Linux try talos jss
Linux try talos tp4
MacOSX Darwin 8.8.1 try talos
MacOSX Darwin 8.8.1 try talos dirty
MacOSX Darwin 8.8.1 try talos jss
MacOSX Darwin 8.8.1 try talos tp4
MacOSX Darwin 9.0.0 try talos
MacOSX Darwin 9.0.0 try talos dirty
MacOSX Darwin 9.0.0 try talos jss
MacOSX Darwin 9.0.0 try talos tp4
Maemo try hg build (*)
OS X 10.5.2 try hg build (*)
OS X 10.5.2 try hg unit test (*)
WINNT 5.1 try talos
WINNT 5.1 try talos dirty
WINNT 5.1 try talos jss
WINNT 5.1 try talos tp4
WINNT 5.2 try hg build (*)
WINNT 5.2 try hg unit test (*)
WINNT 6.0 try talos  (*)
WINNT 6.0 try talos dirty (*)
WINNT 6.0 try talos jss (*)
WINNT 6.0 try talos tp4 (*)
WinMo try hg build (*)

I suspect the problem is that we're still lying about the start time of talos runs, and it's known that tinderbox doesn't like that. So then it's catch 22
a) restart the talos master and not get 50 odd builds tested
b) wait for the master to catch up, still not get the historical data displayed, and not show new builds in the mean time
c) stop lying about the start time and hope that fixes it

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Attempted fix

http://hg.mozilla.org/build/buildbot-configs/rev/76a80c926b45

Stop lying, at least until we catch up all the talos backlog.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

That appears to be working better already.

Comment 3

[Aki Sasaki (not active)]

Looks like this was caused by a major (12+hr) try backlog + talos lying, both of which are fixed.
MozillaTry looks good right now; resolving fixed.