380594 - Auto-reboot red Tinderbox on certain error?

Status: RESOLVED WONTFIX

(Release Engineering :: General, enhancement)

Description

[Mats Palmgren (inactive)]

"MacOSX Darwin 7.9.0 pawn Dep" is red
http://tinderbox.mozilla.org/Camino/

The reason is, as countless times before, the following error:

make[6]: ../../../config/config.mk:802: pipe: Too many open files

Maybe it's worth spending a couple of minutes whipping up a cron script
that does something like this:

IF tinderbox-status == red AND grep tinderbox-log "Too many open files" THEN
  reboot

Comment 1

[Samuel Sidler (old account; do not CC)]

I'm not sure this will matter much in the near future given that we're (Camino) planning on switching to 10.4-only building. It'd definitely help right now, but mento has some other things that can handle this, iirc.

Comment 2

[Mark Mentovai]

A reboot won't help, it needs to be clobbered in the directory with the dependency file that's causing make to open too many files.  It's a bug in the version of make that Xcode 1.5 uses.  There's a bug open about this.  Xcode 1.5 is the only supported toolchain on 10.3.9, and these Camino tinderboxes are the last 10.3.9 boxes around.  We had a workaround, based on the fact that once make hit this, some variables wouldn't be expanded properly and it would try to execute a command "m", so I put a script at /usr/bin/m to do the clobber.  It worked until a few weeks ago, when something in a core makefile changed enough that make no longer attempts to run "m".  I don't think it's worth worrying about too much, since we're going to be getting rid of the 10.3.9 tinderboxes in short order.

Comment 3

[Mark Mentovai]

The affected boxes are almost dead anyway.  WONTFIX.