442887 - qm-xserve01 needs repair

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Please fix qm-xserve01 (dep unit test) on the Firefox tree. I broke it last night debugging some Firefox 3.0.1 nissues, and I couldn't get it working again.

Sorry for the inconvenience.

Thanks,
~reed

Comment 1

[Mike Beltzner [:beltzner, not reading bugmail]]

Until this is resolved it's blocking a checkin required for the 1.9.0.1 builds.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

reed: what experiments were you doing on this live production machine? What did you change? 

(Any info will help figure out what needs repairs)

Comment 3

[Reed Loden [:reed]]

Uh, the machine was broken, so I attempted to fix it, but I couldn't, so that's why I mailed RE for somebody to help. There was no "experimentation." I was helping beltzner get stuff landed for Firefox 3.0.1.

beltzner: This wasn't keeping the tree closed last night, so why should it keep it closed today?

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

As qm-xserve06 is green, I feel this problem with qm-xserve01 does *not* need to close the tree, or block checkins. 

If thats ok, then I'll downgrade to p2.

Comment 5

[Samuel Sidler (old account; do not CC)]

I concur and I'm opening the tree. I just had crowder checkin while it was closed.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #3)
> Uh, the machine was broken, so I attempted to fix it
What exactly was broken?
What exactly did you do while attempting to fix it?

(The only information we have so far is your email, pasted in entirety in
comment#0...and an orange machine on tinderbox waterfall.)

Comment 7

[Mike Beltzner [:beltzner, not reading bugmail]]

It was going sporatically orange for no good reason, and I was looking for someone to see if a reboot would help. Reed volunteered, and we were glad for his help at the time. It's certainly no worse now than it was before.

As Sam said, we've decided to ignore the box, so this is no longer a blocker for us. Reducing priority, clearing flag.

Comment 8

[Reed Loden [:reed]]

(In reply to comment #6)
> (In reply to comment #3)
> > Uh, the machine was broken, so I attempted to fix it
> What exactly was broken?

It was randomly failing mochitest.
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1214884260.1214886903.5808.gz&fulltext=1

> What exactly did you do while attempting to fix it?

I stopped buildbot and started buildbot. That's it. It came back with even more
fail, and I didn't know what to do because my xvncviewer on Linux was having
problems connecting to the VNC, so I e-mailed RE for help, since I knew you all
would know how to fix it, and I didn't see anything in the docs about that
particular problem. It was already broken before I looked at it. I just tried
to kick it, but that didn't help, and it's very possible it made it worse
(which is why I said "I broke it").

> (The only information we have so far is your email, pasted in entirety in
> comment#0...and an orange machine on tinderbox waterfall.)

I'm sorry if my e-mail was contrite, but it was 4am my time when I sent it, I
had been up for 22 hours, I wanted to go to bed, and it wasn't keeping the tree
closed, as we (people handling check-ins last night) decided to ignore it. I
didn't know how to fix it, as I couldn't get VNC working properly for me, so I
e-mailed the people who know how to fix this stuff (you -- RE) for assistance.

Comment 9

[Lukas Blakk [:lsblakk] use ?needinfo]

Did a clobber build - and closed iCal (which was open on the desktop when I
went in to do a clobber) and it seems to be building fine right now.  One green
run so far.

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Excellent, thanks Lukas!!

Comment 11

[Rob Campbell [:rc] (:robcee)]

(In reply to comment #8)
> > What exactly did you do while attempting to fix it?
> 
> I stopped buildbot and started buildbot. That's it. It came back with even more
> fail, and I didn't know what to do because my xvncviewer on Linux was having
> problems connecting to the VNC, so I e-mailed RE for help, since I knew you all
> would know how to fix it, and I didn't see anything in the docs about that
> particular problem. It was already broken before I looked at it. I just tried
> to kick it, but that didn't help, and it's very possible it made it worse
> (which is why I said "I broke it").
> 
> > (The only information we have so far is your email, pasted in entirety in
> > comment#0...and an orange machine on tinderbox waterfall.)
> 
> I'm sorry if my e-mail was contrite, but it was 4am my time when I sent it, I
> had been up for 22 hours, I wanted to go to bed, and it wasn't keeping the tree
> closed, as we (people handling check-ins last night) decided to ignore it. I
> didn't know how to fix it, as I couldn't get VNC working properly for me, so I
> e-mailed the people who know how to fix this stuff (you -- RE) for assistance.

Not to be a nag or anything, but if you've been awake for 22 hours, you probably shouldn't be logging into these machines trying to "fix" anything. You're likely too tired to make clear-headed judgements. Same goes for check-ins, really.

Thanks for fixing, lukas!