1450855 - sea-mini-osx64-3 is back to being awol

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Edmund Wong (:ewong)]

I'm beginning to think we're starting to lose -3. ;/ 

Anyway, can someone please give sea-mini-osx64-3 a swift kick?  It's not
accepting ssh connections or pings.

Comment 1

[Van Le [:van]]

rebooted host.

[[e-mail]3 ~]$ fping !$
fping sea-mini-osx64-3.community.scl3.mozilla.com
sea-mini-osx64-3.community.scl3.mozilla.com is alive

Comment 2

[Edmund Wong (:ewong)]

it's back down again.

Can someone give it another boot?

Comment 3

[Van Le [:van]]

kicked again.

Comment 4

[Edmund Wong (:ewong)]

it's back down again after doing 2 jobs. :/

I'm somewhat concerned that this system's hdd is starting to fail.  If you have a chance,
can you do a hdd scan?

Comment 5

[Van Le [:van]]

>I'm somewhat concerned that this system's hdd is starting to fail.  If you have a chance,
can you do a hdd scan?


i kicked the host but i don't know the credentials. can you send me a gpg encrypted email with the root/administrative login and password?

Comment 6

[Edmund Wong (:ewong)]

(In reply to Van Le [:van] from comment #5)
> >I'm somewhat concerned that this system's hdd is starting to fail.  If you have a chance,
> can you do a hdd scan?
> 
> 
> i kicked the host but i don't know the credentials. can you send me a gpg
> encrypted email with the root/administrative login and password?

Hi van, thanks for giving it a kick again.  

It looks like it's back up and running the repack jobs.   So perhaps I'll wait for the next downtime
to send you the login password.   

Thanks!

Comment 7

[Edmund Wong (:ewong)]

oh sigh..  it's back down again. :( 

As I'm doing a release..  I'm not sure which option is better. 
   1) Let it be kicked and have it run that final repack job
or 2) have you log on and checkdisk it?

Comment 8

[Van Le [:van]]

#2, the host reports 5 major and a few minor problems on the disk during boot up.

Comment 9

[Edmund Wong (:ewong)]

(In reply to Van Le [:van] from comment #8)
> #2, the host reports 5 major and a few minor problems on the disk during
> boot up.

ouch.  not good.  In your experience, how long will it take to fix those problems?

Comment 10

[Van Le [:van]]

it takes a few hours but there's no guarantee that it fixes anything.

Comment 11

[Edmund Wong (:ewong)]

(In reply to Van Le [:van] from comment #10)
> it takes a few hours but there's no guarantee that it fixes anything.

:van, can you just give it a kick while you're there?

Thanks

Comment 12

[Van Le [:van]]

>:van, can you just give it a kick while you're there?

i'll be in the dc tomorrow, working in MTV2 today. did you send me the password to perform the disk check, if so, i haven't received anything.

Comment 13

[Van Le [:van]]

rebooted, didn't receive an email from you last night.

Comment 14

[Van Le [:van]]

i ran fix disk permissions and it fixed a bunch of stuff. verify disk didn't see any issues but then again, it doesn't do a deep surface scan.