MozillaTry tinderbox not displaying any columns or build data at all

RESOLVED FIXED

Status

task
P1
critical
RESOLVED FIXED
10 years ago
4 years ago

People

(Reporter: anodelman, Assigned: reed)

Tracking

Details

MozillaTry is currently not displaying available info on several Talos boxes.  The error does not appear to be on the side of the builders as other Talos machines with the same configuration are successfully adding their information.

Missing columns:

qm-pubuntu-try01
qm-pubunut-try02
qm-pxp-try01

Several other columns were missing but reed resurrected them through running a cleaning script.
cls, I'll get you another tarball.
Assignee: build → reed
Component: Tinderbox Configuration → Server Operations
QA Contact: ccooper → mrz
Reporter

Comment 2

10 years ago
More missing columns:

qm-pxp-try02
qm-pvista-try01
qm-pvista-try02
Assignee: reed → build
Component: Server Operations → Tinderbox Configuration
QA Contact: mrz → ccooper

Comment 3

10 years ago
I'm not sure why I'd need another tarball.  The next time the problem occurs, do:
   tail -200 build.dat | perl -e 'while(<STDIN>) { chomp; ($e,$b,$n) = split(/\|/,$_); $t=localtime($b); print "$t -- $n\n";}'

If you see a bunch of dates older than 14 hrs or so, then you're hitting bug 225735 .
Reporter

Comment 4

10 years ago
Well, it is still happening right now - as those 6 columns are not being displayed.  Is there anything that can be done to get them back on board?
[root@dm-webtools02 MozillaTry]# tail -200 build.dat | perl -e 'while(<STDIN>) { chomp; ($e,$b,$n) =
> split(/\|/,$_); $t=localtime($b); print "$t -- $n\n";}'
Thu Mar 26 04:04:06 2009 -- Try server linux hg builder
Thu Mar 26 01:52:39 2009 -- Try linux hg unittest builder
Thu Mar 26 04:05:09 2009 -- Try server linux hg builder
Thu Mar 26 02:04:19 2009 -- Try linux hg unittest builder
Thu Mar 26 01:53:39 2009 -- Try win32 hg unittest builder
Thu Mar 26 04:17:41 2009 -- Try server win32 hg builder
Thu Mar 26 04:19:53 2009 -- Try server linux hg builder
Thu Mar 26 02:16:39 2009 -- Try linux hg unittest builder
Thu Mar 26 04:23:55 2009 -- Try server win32 hg builder
Thu Mar 26 01:53:12 2009 -- Try win32 hg unittest builder
Wed Mar 25 23:07:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01
Thu Mar 26 04:36:50 2009 -- Try server win32 hg builder
Thu Mar 26 01:54:36 2009 -- Try win32 hg unittest builder
Wed Mar 25 15:59:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01
Wed Mar 25 16:26:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01
Wed Mar 25 15:59:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02
Wed Mar 25 16:26:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02
Thu Mar 26 03:18:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02
Thu Mar 26 03:18:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01
Thu Mar 26 04:52:09 2009 -- Try server mac hg builder
Thu Mar 26 04:52:10 2009 -- Try macosx hg unittest builder
Thu Mar 26 03:19:00 2009 -- Try server mac hg builder
Thu Mar 26 04:54:03 2009 -- Try server win32 hg builder
Thu Mar 26 04:00:00 2009 -- Try server win32 hg builder
Wed Mar 25 13:55:00 2009 -- WINNT 5.1 talos try qm-pxp-try02
Wed Mar 25 13:55:00 2009 -- WINNT 6.0 talos try qm-pvista-try01
Wed Mar 25 13:55:00 2009 -- WINNT 5.1 talos try qm-pxp-try01
Wed Mar 25 13:55:00 2009 -- WINNT 6.0 talos try qm-pvista-try02
Wed Mar 25 14:52:00 2009 -- Linux talos try qm-pubuntu-try01
Wed Mar 25 14:52:00 2009 -- Linux talos try qm-pubuntu-try02
Thu Mar 26 04:19:53 2009 -- Try server linux hg builder
Thu Mar 26 05:09:06 2009 -- Try server linux hg builder
Thu Mar 26 05:20:07 2009 -- Try linux hg unittest builder
Thu Mar 26 03:56:49 2009 -- Try server linux hg builder
Thu Mar 26 05:22:50 2009 -- Try server linux hg builder
Thu Mar 26 04:04:06 2009 -- Try server linux hg builder
Thu Mar 26 05:26:09 2009 -- Try linux hg unittest builder
Thu Mar 26 04:05:09 2009 -- Try server linux hg builder
Thu Mar 26 03:19:00 2009 -- Try macosx hg unittest builder
Wed Mar 25 16:26:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01
Wed Mar 25 19:25:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01
Wed Mar 25 16:26:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02
Wed Mar 25 19:25:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02
Thu Mar 26 03:18:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02
Thu Mar 26 03:18:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01
Wed Mar 25 14:52:00 2009 -- Linux talos try qm-pubuntu-try01
Wed Mar 25 14:52:00 2009 -- Linux talos try qm-pubuntu-try02
Wed Mar 25 13:14:00 2009 -- Linux talos try qm-pubuntu-try02
Wed Mar 25 13:14:00 2009 -- Linux talos try qm-pubuntu-try01
Thu Mar 26 05:59:27 2009 -- Try server linux hg builder
Thu Mar 26 05:09:06 2009 -- Try server linux hg builder
Wed Mar 25 15:59:00 2009 -- WINNT 5.1 talos try qm-pxp-try01
Wed Mar 25 15:59:00 2009 -- WINNT 5.1 talos try qm-pxp-try02
Wed Mar 25 15:59:00 2009 -- WINNT 6.0 talos try qm-pvista-try02
Wed Mar 25 15:59:00 2009 -- WINNT 6.0 talos try qm-pvista-try01
Wed Mar 25 15:59:00 2009 -- WINNT 5.1 talos try qm-pxp-try02
Wed Mar 25 15:59:00 2009 -- WINNT 5.1 talos try qm-pxp-try01
Wed Mar 25 15:59:00 2009 -- WINNT 6.0 talos try qm-pvista-try01
Wed Mar 25 15:59:00 2009 -- WINNT 6.0 talos try qm-pvista-try02
Thu Mar 26 04:52:10 2009 -- Try macosx hg unittest builder
Thu Mar 26 06:17:37 2009 -- Try win32 hg unittest builder
Thu Mar 26 04:23:55 2009 -- Try server win32 hg builder
Wed Mar 25 19:25:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01
Wed Mar 25 23:07:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01
Wed Mar 25 19:25:00 2009 -- WINNT 5.1 talos try qm-pxp-try02
Wed Mar 25 19:25:00 2009 -- WINNT 6.0 talos try qm-pvista-try01
Wed Mar 25 19:25:00 2009 -- WINNT 6.0 talos try qm-pvista-try02
Wed Mar 25 19:25:00 2009 -- WINNT 5.1 talos try qm-pxp-try01
Thu Mar 26 04:51:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01
Thu Mar 26 04:51:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02
Thu Mar 26 04:52:09 2009 -- Try server mac hg builder
Thu Mar 26 06:32:00 2009 -- Try server mac hg builder
Thu Mar 26 06:32:00 2009 -- Try macosx hg unittest builder
Wed Mar 25 19:25:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02
Wed Mar 25 23:07:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02
Thu Mar 26 04:51:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01
Thu Mar 26 04:51:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02
Thu Mar 26 06:37:00 2009 -- Try macosx hg unittest builder
Thu Mar 26 06:37:01 2009 -- Try server mac hg builder
Wed Mar 25 13:14:00 2009 -- Linux talos try qm-pubuntu-try02
Wed Mar 25 13:14:00 2009 -- Linux talos try qm-pubuntu-try01
Thu Mar 26 06:40:45 2009 -- Try server win32 hg builder
Thu Mar 26 04:54:03 2009 -- Try server win32 hg builder
Wed Mar 25 13:55:00 2009 -- Linux talos try qm-pubuntu-try02
Wed Mar 25 13:55:00 2009 -- Linux talos try qm-pubuntu-try01
Thu Mar 26 06:42:30 2009 -- Try linux hg unittest builder
Thu Mar 26 05:22:50 2009 -- Try server linux hg builder
Thu Mar 26 06:43:32 2009 -- Try win32 hg unittest builder
Thu Mar 26 04:17:41 2009 -- Try server win32 hg builder
Thu Mar 26 04:36:50 2009 -- Try server win32 hg builder
Thu Mar 26 06:44:33 2009 -- Try server win32 hg builder
Thu Mar 26 06:57:29 2009 -- Try server linux hg builder
Thu Mar 26 05:59:27 2009 -- Try server linux hg builder
Wed Mar 25 23:07:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01
Thu Mar 26 03:18:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01
Thu Mar 26 05:26:09 2009 -- Try linux hg unittest builder
Thu Mar 26 07:22:19 2009 -- Try linux hg unittest builder
Wed Mar 25 23:07:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02
Wed Mar 25 13:55:00 2009 -- Linux talos try qm-pubuntu-try02
Wed Mar 25 13:55:00 2009 -- Linux talos try qm-pubuntu-try01
Thu Mar 26 03:18:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02
Wed Mar 25 14:20:00 2009 -- Linux talos try qm-pubuntu-try02
Wed Mar 25 14:20:00 2009 -- Linux talos try qm-pubuntu-try01
Wed Mar 25 19:25:00 2009 -- WINNT 5.1 talos try qm-pxp-try02
Wed Mar 25 19:25:00 2009 -- WINNT 6.0 talos try qm-pvista-try01
Wed Mar 25 19:25:00 2009 -- WINNT 6.0 talos try qm-pvista-try02
Wed Mar 25 19:25:00 2009 -- WINNT 5.1 talos try qm-pxp-try01
Wed Mar 25 14:52:00 2009 -- WINNT 6.0 talos try qm-pvista-try01
Wed Mar 25 14:52:00 2009 -- WINNT 5.1 talos try qm-pxp-try02
Wed Mar 25 14:52:00 2009 -- WINNT 6.0 talos try qm-pvista-try02
Wed Mar 25 14:52:00 2009 -- WINNT 5.1 talos try qm-pxp-try01
Thu Mar 26 07:50:29 2009 -- Try server linux hg builder
Thu Mar 26 06:57:29 2009 -- Try server linux hg builder
Thu Mar 26 07:57:44 2009 -- Try server mac hg builder
Thu Mar 26 06:32:00 2009 -- Try macosx hg unittest builder
Thu Mar 26 08:03:47 2009 -- Try macosx hg unittest builder
Thu Mar 26 06:37:00 2009 -- Try macosx hg unittest builder
Thu Mar 26 03:18:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01
Thu Mar 26 04:51:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01
Thu Mar 26 06:35:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01
Thu Mar 26 06:35:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02
Thu Mar 26 06:37:01 2009 -- Try server mac hg builder
Wed Mar 25 14:20:00 2009 -- Linux talos try qm-pubuntu-try02
Wed Mar 25 14:20:00 2009 -- Linux talos try qm-pubuntu-try01
Wed Mar 25 15:59:00 2009 -- Linux talos try qm-pubuntu-try02
Wed Mar 25 15:59:00 2009 -- Linux talos try qm-pubuntu-try01
Thu Mar 26 06:32:00 2009 -- Try server mac hg builder
Thu Mar 26 03:18:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02
Thu Mar 26 04:51:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02
Thu Mar 26 08:18:03 2009 -- Try linux hg unittest builder
Thu Mar 26 05:20:07 2009 -- Try linux hg unittest builder
Thu Mar 26 04:51:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01
Thu Mar 26 06:35:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02
Thu Mar 26 06:35:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01
Thu Mar 26 06:35:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01
Thu Mar 26 06:30:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02
Thu Mar 26 06:30:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01
Thu Mar 26 08:26:42 2009 -- Try macosx hg unittest builder
Thu Mar 26 04:51:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02
Thu Mar 26 08:26:41 2009 -- Try server mac hg builder
Thu Mar 26 06:35:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02
Thu Mar 26 06:35:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01
Thu Mar 26 06:30:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01
Thu Mar 26 06:35:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02
Thu Mar 26 06:30:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02
Thu Mar 26 08:45:36 2009 -- Try server linux hg builder
Thu Mar 26 07:50:29 2009 -- Try server linux hg builder
Thu Mar 26 08:56:27 2009 -- Try win32 hg unittest builder
Thu Mar 26 06:17:37 2009 -- Try win32 hg unittest builder
Wed Mar 25 14:52:00 2009 -- WINNT 5.1 talos try qm-pxp-try02
Wed Mar 25 15:59:00 2009 -- Linux talos try qm-pubuntu-try01
Wed Mar 25 15:59:00 2009 -- Linux talos try qm-pubuntu-try02
Wed Mar 25 14:52:00 2009 -- WINNT 6.0 talos try qm-pvista-try01
Wed Mar 25 14:52:00 2009 -- WINNT 6.0 talos try qm-pvista-try02
Wed Mar 25 16:26:00 2009 -- Linux talos try qm-pubuntu-try01
Wed Mar 25 16:26:00 2009 -- WINNT 6.0 talos try qm-pvista-try01
Wed Mar 25 16:26:00 2009 -- WINNT 5.1 talos try qm-pxp-try02
Wed Mar 25 14:52:00 2009 -- WINNT 5.1 talos try qm-pxp-try01
Wed Mar 25 16:26:00 2009 -- Linux talos try qm-pubuntu-try02
Thu Mar 26 09:01:41 2009 -- Try server win32 hg builder
Wed Mar 25 16:26:00 2009 -- WINNT 6.0 talos try qm-pvista-try02
Thu Mar 26 06:40:45 2009 -- Try server win32 hg builder
Wed Mar 25 16:26:00 2009 -- WINNT 5.1 talos try qm-pxp-try01
Thu Mar 26 06:30:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01
Thu Mar 26 07:22:19 2009 -- Try linux hg unittest builder
Thu Mar 26 06:30:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02
Thu Mar 26 09:11:42 2009 -- Try linux hg unittest builder
Thu Mar 26 09:13:33 2009 -- Try win32 hg unittest builder
Thu Mar 26 06:44:33 2009 -- Try server win32 hg builder
Thu Mar 26 06:30:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01
Thu Mar 26 09:25:45 2009 -- Try server linux hg builder
Thu Mar 26 06:42:30 2009 -- Try linux hg unittest builder
Thu Mar 26 06:30:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02
Thu Mar 26 09:34:34 2009 -- Try server mac hg builder
Thu Mar 26 07:57:44 2009 -- Try server mac hg builder
Thu Mar 26 08:45:36 2009 -- Try server linux hg builder
Thu Mar 26 09:38:36 2009 -- Try linux hg unittest builder
Wed Mar 25 16:26:00 2009 -- Linux talos try qm-pubuntu-try01
Wed Mar 25 16:26:00 2009 -- Linux talos try qm-pubuntu-try02
Wed Mar 25 19:25:00 2009 -- Linux talos try qm-pubuntu-try01
Wed Mar 25 19:25:00 2009 -- Linux talos try qm-pubuntu-try02
Thu Mar 26 08:26:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01
Thu Mar 26 08:26:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01
Thu Mar 26 08:26:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02
Thu Mar 26 08:26:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02
Thu Mar 26 10:08:34 2009 -- Try macosx hg unittest builder
Thu Mar 26 08:26:41 2009 -- Try server mac hg builder
Thu Mar 26 10:11:01 2009 -- Try server mac hg builder
Thu Mar 26 08:03:47 2009 -- Try macosx hg unittest builder
Thu Mar 26 10:13:29 2009 -- Try server win32 hg builder
Thu Mar 26 06:43:32 2009 -- Try win32 hg unittest builder
Wed Mar 25 16:26:00 2009 -- WINNT 5.1 talos try qm-pxp-try02
Wed Mar 25 16:26:00 2009 -- WINNT 6.0 talos try qm-pvista-try01
Thu Mar 26 10:11:01 2009 -- Try server mac hg builder
Thu Mar 26 10:16:50 2009 -- Try macosx hg unittest builder
Wed Mar 25 23:07:00 2009 -- WINNT 5.1 talos try qm-pxp-try02
Wed Mar 25 16:26:00 2009 -- WINNT 6.0 talos try qm-pvista-try02
Wed Mar 25 23:07:00 2009 -- WINNT 6.0 talos try qm-pvista-try01
Wed Mar 25 16:26:00 2009 -- WINNT 5.1 talos try qm-pxp-try01
Wed Mar 25 23:07:00 2009 -- WINNT 6.0 talos try qm-pvista-try02

Comment 6

10 years ago
It looks like the problem doesn't occur in the last 200 build mails.  20 consecutive 'greater than 14 hr difference' builds is what causes tbox to stop reading build.dat.

However, you clearly have a problem with the clocks on your servers.  They shouldn't be jumping around like that.  I'd expect them to consistently increase. (That's what they do on my local trees anyway.)
Assignee: build → server-ops
Severity: normal → major
Component: Tinderbox Configuration → Server Operations
QA Contact: ccooper → mrz
Is the time shown the start time of the build ?
Assignee: server-ops → reed
Developers rely on the tryserver now. It's instrumental in keeping the tree healthy, so requesting that this be a P1 blocker.
Severity: major → blocker
Priority: -- → P1
updating summary to reflect the current state of the waterfall page.
Summary: MozillaTry tinderbox not displaying all columns. → MozillaTry tinderbox not displaying any columns or build data at all
(In reply to comment #8)
> Developers rely on the tryserver now. It's instrumental in keeping the tree
> healthy, so requesting that this be a P1 blocker.

The Firefox 3.0.8 release is taking priority over this, so not sure when I'll be able to get to it. I'll get it done sometime later today, though (best I can do is remake the tree).
I ran the clean script again with only 3 days of logs kept, but there's obviously issues with the build machines, so passing this to RE to fix the time on all the machines.
Assignee: reed → nobody
Component: Server Operations → Release Engineering
QA Contact: mrz → release
No longer blocks: 483407
Assignee

Updated

10 years ago
Duplicate of this bug: 485869
guh, I ran the clean script last week as root rather than tinderbox, which caused tinderbox to not be able to update build.dat due to permission issues (plus, there was a stale semaphore lock file). While I don't know why this problem started, I at least know why it hasn't resolved itself yet, and I've fixed that.
Assignee: nobody → reed
Status: NEW → RESOLVED
Closed: 10 years ago
Component: Release Engineering → Server Operations
QA Contact: release → mrz
Resolution: --- → FIXED
Looks like the win32 talos machines have disappeared again...
Severity: blocker → critical
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
I just fixed bug 485015. Are you still seeing this?
Assignee

Updated

10 years ago
Status: REOPENED → RESOLVED
Closed: 10 years ago10 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.