448019 - stage-old/surf is down, closing Firefox/Firefox3.0/Mozilla1.8 trees

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

justdave told me it would be down for a few minutes to have its RAID battery replaced (IIRC). That was hours ago. Can we get an update and ETA here?

Comment 1

[Mark Smith [:xb95]]

Why is this dependent on stage-old?

Seems like a hardware issue with surf.  We are on it but no hard ETA at this moment.

Comment 2

[Brian Crowder]

Even a soft ETA would be useful, if you care to hazard a guess.

Comment 3

[[e-mail] (:bhearsum)]

Because stage-old is generally stage.mozilla.org and this is where we upload all of our builds to. We also depend on shell access to it for certain things involved with releases.

Comment 4

[matthew zeier [:mrz]]

No idea on an ETA, box appears to have come up and then stopped although the console is totally unresponsive.  

Are you unable to upload builds to stage?  

Comment 5

[[e-mail] (:bhearsum)]

We can upload builds OK but we cannot do some of the final things we need to do before releasing.

Comment 6

[matthew zeier [:mrz]]

If/when we finally switch over to dm-stage02, how is this supposed to work?  Are there steps we can run now to unblock this while we troubleshoot?

Comment 7

[[e-mail] (:bhearsum)]

That's a bridge we're planning to cross when we make the switchover.

If there's any way we can get a proper shell for cltbld that would be a tremendous help.

Comment 8

[Mark Smith [:xb95]]

I'm on my way to the data center to try to get surf going again, as it doesn't seem to want to respond to our remote hands.

Still no ETA, but I'll know more when I get there and can see what's going on using the good old fashioned "eyeball method".

Comment 9

[Mark Smith [:xb95]]

surf is not responding to anything I've tried.  Nothing is visible on the VGA output at any point.

We are arranging for an HP tech to come out tonight, and will keep working on this.

Lowering the priority as it seems dm-stage02 is standing in for surf for now.

Comment 11

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Tweaking summary, as we've managed to work around this for shipping FF3.1a1.

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Talos is still failing, the latest error I see is:
...
U pageloader/report.js
U pageloader/tp-cmdline.js
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx/1217042898/:
20:40:00 ERROR 404: Not Found.
...

This is interesting because when I manually browse to that URL from my laptop, I see the files just fine. 

Comment 13

[Reed Loden [:reed]]

Motherboard died. Replaced by HP. surf's UP, dude.

Comment 14

[Dave Miller [:justdave]]

Does that machine have surf in its hosts file for ftp?  ftp.mozilla.org is NOT surf, and surf being down should have had no affect on that URL.

Comment 15

[[e-mail] (:bhearsum)]

(In reply to comment #14)
> Does that machine have surf in its hosts file for ftp?  ftp.mozilla.org is NOT
> surf, and surf being down should have had no affect on that URL.
> 

Talos was failing because our builders could not upload to Stage (because they hard-code stage-old), AFAIK.