122504 - tinderbox.mozilla.org not receiving/processing mail from build machines properly

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Darin Fisher]

tinderbox.mozilla.org not receiving/processing mail from build machines properly.

we're getting a lot of yellow boxes on tinderbox, which jj believes is a mail
problem.  that is, the tinderbox build machines are successfully completing
builds but the tinderbox pages are not reflecting the results... they are just
showing yellow boxes :-(

Comment 1

[Risto Kotalampi]

Looks like lounge is in heavy load and just queueing emails. Currently there's 
825 emails in queue - most going to tinderbox-daemon. Reason for loads: bunch 
of showbuild.pl processes.

Comment 2

[Daniel (Leaf) Nunes]

maybe there was a change to the sidebar code that's causing people to constantly
hit the tinderbox page; is there a way to see what arguments are being passed to
showbuilds?

Comment 3

[Risto Kotalampi]

Now when the mail queue is seattled down load is gone... and I compared 
accesses to showbuilds.cgi and they looked normal (nothing alarming compared to 
yesterday).

Something in those 800+ (or whatever it was at peak) must have loaded the 
system badly. Unfortunately we don't have mail logs available to look further 
into this.

Comment 4

[Risto Kotalampi]

Marking resolved.

Comment 5

[Risto Kotalampi]

It seems like we are still getting these bursts of tinderbox emails from 
iPlanet/Sun from svbld to NSS tree.

Comment 6

[Dawn Endico]

bursts? how often? i thought we were only getting one build every
40 minutes or so.

Comment 7

[Risto Kotalampi]

I don't know how often but after this afternoon I've seen one more burst from 
iPlanet systems.

Comment 8

[Risto Kotalampi]

There was an issue with lounge's syslog (name resolution never hit "files") that
prevented mail logs to be saved. I have fixed it so next time we should be able
to make better analyzis of what happened and track the spammer down.