536762 - Try buildbot masters are down

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

In what appears to be a coincidence, both the buildbot masters for Try builds and talos are down.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

The try talos master was oom-killed by the kernel at Dec 25 08:05:53. The most recent job to finish (7 minutes earlier) was a "MacOSX Darwin 9.0.0 try talos svg" on http://build.mozilla.org/tryserver-builds/[e-mail]-fin_opt/fin_opt-macosx.dmg (MacOSX Darwin 9.0.0 try talos svg/builds/765). That created an unusally large log file - 483M for 765-log-Run_performance_tests-stdio. I'd guess that trying to compress that log caused us to run out of memory.

This master is back up now.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

The try build master died at 2009-12-25 14:55 when the /builds partition was completely full. The problem was a mac unit test build that grew it's 'make check' log to 18G (sendchange-macosx-unittest/4576-log-check-stdio). The master won't tell me directly what was built for that but, reading between the lines it was: 
 Changed by: [e-mail]
 Changed at: Fri 25 Dec 2009 04:17:17
 Branch: HG_TRY

 Changed files:
    * identifier: fin_opt
    * mozconfig: 1261743382-411-
    * mozillaRepoPath: http://hg.mozilla.org/tracemonkey
    * patchFile: 1261743382-411-opt_obj_final.patch
    * patchLevel: 1

I've truncated the log file to 1M to resolve the issue, and restarted the master.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

The talos log it begins
 Running test tsvg: 
                Started Fri, 25 Dec 2009 06:46:45
then many many repeats of the following:
  firefox-bin(198,0xb00a1000) malloc: *** error for object 0x1bfef008: Non-aligned pointer being freed
  *** set a breakpoint in malloc_error_break to debug

In the make check log it's repeated
 xpcshell(93958,0xb0081000) malloc: *** error for object 0x9b4b778: Non-aligned  pointer being freed

---

Bug 536764 filed to prevent this from happning in the future.