795624 - builds-4hr.js.gz not updating

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

On recent pushes, TBPL is not showing results of jobs when they finish. The results are visible in self-serve.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

FYI, all trees are currently closed due to this.

Comment 2

[Phil Ringnalda (:philor)]

http://builddata.pub.build.mozilla.org/buildjson/builds-4hr.js.gz says it was last modified Sat Sep 29 09:34:08 2012.

Comment 3

[Mounir Lamouri (:mounir)]

For information, with basecamp feature freeze being on Sunday midnight, this bug is very critical.

Comment 4

[Chris AtLee [:catlee]]

So the export script is choking on some bad properties data in the DB. This data came from this job:
http://buildbot-master13.build.scl1.mozilla.com:8001/builders/TB%20WINNT%205.2%20comm-central%20build/builds/210

and has properties like,
basedir 	2086735202 [sig] sh 1660 sig_dispatch_pending: Win32 error 298 releasing sigcatch_nosync(0xF8) 2086737081 [sig] sh 1660 sig_dispatch_pending: Win32 error 298 releasing sigcatch_nosync(0xF8) 2086737169 [sig] sh 1660 sig_dispatch_pending: Win32 error 298 releasing sigcatch_nosync(0xF8) 2086737259 [sig] sh 1660 sig_dispatch_pending: Win32 error 298 releasing sigcatch_nosync(0xF8) 2086737290 [sig] sh 1660 sig_dispatch_pending: Win32 error 298 releasing sigcatch_nosync(0xF8) 2086737378 [sig] sh 1660 .. [property value too long]

The simple bash calls to set the basedir properties have hundreds of lines of those error messages in them.

The previous job on this slave was manually cancelled via self-serve. Subsequent builds have been ok.

I've removed the bad data from the DB, and now the export script is working.

Comment 5

[Chris AtLee [:catlee]]

Follow up items:
- figure out where nagios checks for this file went
- add check for invalid data in db
- figure out if we can have builds cancelled via self-serve still reboot

Comment 6

[Chris Cooper [:coop] (he/him)]

(In reply to Chris AtLee [:catlee] from comment #5)
> Follow up items:
> - figure out where nagios checks for this file went
> - add check for invalid data in db
> - figure out if we can have builds cancelled via self-serve still reboot

catlee: are there separate bugs on file for those issues, or are we tracking them here?

Comment 7

[Chris AtLee [:catlee]]

(In reply to Chris Cooper [:coop] from comment #6)
> (In reply to Chris AtLee [:catlee] from comment #5)
> > Follow up items:
> > - figure out where nagios checks for this file went
> > - add check for invalid data in db
> > - figure out if we can have builds cancelled via self-serve still reboot
> 
> catlee: are there separate bugs on file for those issues, or are we tracking
> them here?

there are now!