465129 - NSS tinderbox pages frozen since yesterday

Status: RESOLVED INVALID

(mozilla.org Graveyard :: Server Operations, task)

Description

[Nelson Bolyard (seldom reads bugmail)]

The Tinderbox pages 
http://tinderbox.mozilla.org/showbuilds.cgi?tree=NSS  and
http://tinderbox.mozilla.org/showbuilds.cgi?tree=NSS-Stable-Branch
appear to be frozen in time as of about 7 AM today. 
The top row on each page shows the information from just minutes before 7AM.
None of the activity of any of the build systems since that time is shown.

I'd like someone to make them come alive and begin to show the updates that
have taken place since 7 AM.

Comment 1

[Reed Loden [:reed]]

"Tinderbox Configuration" isn't what you think it is... it's a deprecated component for configuring mozilla.org tinderbox client machines...

You really want Server Operations.

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

OK Reed, thanks.

I wonder whether the problem is:
a) that the tinderbox software has stopped updating the info for the NSS 
page, or 
b) the tinderbox server has received no emails from any of the build+test 
systems since 7 AM.

Any way to know?

Comment 3

[Nelson Bolyard (seldom reads bugmail)]

NSS development is blocked by this.  
We're holding the NSS tree closed for this.  
(I know, bonsai doesn't show that, but it's true nonetheless.)

Comment 4

[Dave Miller [:justdave]]

where are these tinderboxes, and what email address do they use to send the mail FROM?

Comment 5

[Nelson Bolyard (seldom reads bugmail)]

I'm not sure of the exact email address, but I suspect it is 
[e-mail]

I'm sure it is from an email address in one of these domains (or subdomains
thereof):
  sun.com
  iplanet.com

Comment 6

[Nelson Bolyard (seldom reads bugmail)]

Slavo, can you answer the question in comment 4?

Comment 7

[Nelson Bolyard (seldom reads bugmail)]

This URL shows the final state of the NSS trunk inderbox page, the state 
it was in when I reported this bug.

http://tinderbox.mozilla.org/showbuilds.cgi?tree=NSS&maxdate=1226772722&legend=0&norules=1

One thing that I find odd about it is that at the time that the updates appear
to have stopped, there were builds in progress.  I would have expected that 
tinderbox would have continued to show tall columns of yellow for a long time 
after that.  Instead, it stopped showing any updates as of ~7 AM.  

Slavo, Are these outgoing tinderbox emails sent from the individual Tinderbox
systems? or are they all sent from one central system?  
Is it possible to check the system(s) from which the mails are sent to see if
the outgoing emails are queued up there?

Comment 8

[Nelson Bolyard (seldom reads bugmail)]

I have learned that a power outage took out the entire room where all the
NSS Tinderbox build systems are located.  They are coming back up now.

Comment 9

[Reed Loden [:reed]]

(In reply to comment #8)
> I have learned that a power outage took out the entire room where all the
> NSS Tinderbox build systems are located.  They are coming back up now.

So, not a tinderbox or mail problem. --> RESOLVED INVALID

Comment 10

[Alexei Volkov]

NSS tinderbox is frozen again

Comment 11

[Dave Miller [:justdave]]

Has anyone checked the machines in question?  Last time this happened they were actually down.

Comment 12

[Alexei Volkov]

Sorry. This time the problem is actually on our side.