Closed Bug 1450855 Opened 2 years ago Closed 2 years ago

sea-mini-osx64-3 is back to being awol

Categories

(Infrastructure & Operations :: DCOps, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ewong, Assigned: van)

Details

I'm beginning to think we're starting to lose -3. ;/ 

Anyway, can someone please give sea-mini-osx64-3 a swift kick?  It's not
accepting ssh connections or pings.
rebooted host.

[vle@admin1a.private.scl3 ~]$ fping !$
fping sea-mini-osx64-3.community.scl3.mozilla.com
sea-mini-osx64-3.community.scl3.mozilla.com is alive
Assignee: server-ops-dcops → vle
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
it's back down again.

Can someone give it another boot?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
kicked again.
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → FIXED
it's back down again after doing 2 jobs. :/

I'm somewhat concerned that this system's hdd is starting to fail.  If you have a chance,
can you do a hdd scan?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
>I'm somewhat concerned that this system's hdd is starting to fail.  If you have a chance,
can you do a hdd scan?


i kicked the host but i don't know the credentials. can you send me a gpg encrypted email with the root/administrative login and password?
Flags: needinfo?(ewong)
(In reply to Van Le [:van] from comment #5)
> >I'm somewhat concerned that this system's hdd is starting to fail.  If you have a chance,
> can you do a hdd scan?
> 
> 
> i kicked the host but i don't know the credentials. can you send me a gpg
> encrypted email with the root/administrative login and password?

Hi van, thanks for giving it a kick again.  

It looks like it's back up and running the repack jobs.   So perhaps I'll wait for the next downtime
to send you the login password.   

Thanks!
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Flags: needinfo?(ewong)
Resolution: --- → FIXED
oh sigh..  it's back down again. :( 

As I'm doing a release..  I'm not sure which option is better. 
   1) Let it be kicked and have it run that final repack job
or 2) have you log on and checkdisk it?
Status: RESOLVED → REOPENED
Flags: needinfo?(vle)
Resolution: FIXED → ---
#2, the host reports 5 major and a few minor problems on the disk during boot up.
Flags: needinfo?(vle)
(In reply to Van Le [:van] from comment #8)
> #2, the host reports 5 major and a few minor problems on the disk during
> boot up.

ouch.  not good.  In your experience, how long will it take to fix those problems?
Flags: needinfo?(vle)
it takes a few hours but there's no guarantee that it fixes anything.
Flags: needinfo?(vle)
(In reply to Van Le [:van] from comment #10)
> it takes a few hours but there's no guarantee that it fixes anything.

:van, can you just give it a kick while you're there?

Thanks
>:van, can you just give it a kick while you're there?

i'll be in the dc tomorrow, working in MTV2 today. did you send me the password to perform the disk check, if so, i haven't received anything.
rebooted, didn't receive an email from you last night.
i ran fix disk permissions and it fixed a bunch of stuff. verify disk didn't see any issues but then again, it doesn't do a deep surface scan.
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.