1126162 - sea-mini-osx64-{1,2} are AWOL

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Edmund Wong (:ewong)]

Now both sea-mini-osx64-1 and -2 are exhibiting signs of AWOL'ness.
(re: bug 1120783) They respond to pings, but not to ssh or vnc connections.

From the aforementioned bug, nothing seems to be physically wrong with
them from the diagnostics so is it something in memory that's making
them hang like this?

Comment 1

[Vinh Hua [:vinh]]

sea-mini-osx64-1.community.scl3.mozilla.com boots up fine.  Screen sharing and Remote Login are enabled under "Sharing", however still unable to SSH and VNC. Was there any changes that were made prior?

sea-mini-osx64-2.community.scl3.mozilla.com "beach balled" at login.  I suspect a hard drive issue.  In case we need to replace, any special requirements on the OS and settings?  I don't think we have an image for these community mac minis.

Comment 3

[Van Le [:van]]

per  C#10 in bug 1120783.

it's back at the beach. ;/

Callek, what should we do about this?  Diagnostics have shown nothing.

re-image?

Comment 4

[Justin Wood (:Callek)]

(In reply to Van Le [:van] from comment #3)
> per  C#10 in bug 1120783.
> 
> it's back at the beach. ;/
> 
> Callek, what should we do about this?  Diagnostics have shown nothing.
> 
> re-image?

are we able to reimage based on another existing host, we don't have a stored image, or a deploystudio server able to host these, etc.

Beyond that, I may start to be forced to literally beg moco (probably dougt directly) to get a few of their OSX builders for Seamonkey... which I suspect is not going to go over well....

Comment 5

[Van Le [:van]]

we would be reimaging off a CD. in the past, if we don't format the drive and just reimage, OSX has been pretty good at keeping the configs, although i'm not sure if it'll write over the files/configs you guys need as i'm not entirely sure what it replaces. it might also not fix whatever is causing your system to hang since it's not a clean reinstall.

Comment 6

[Justin Wood (:Callek)]

yea that won't work, these are a VERY specific set of OSX versions + updates. and have many system config changes, as per an old version of mozilla's OSX builders.

We've made some updates along the way to accomodate necessary changes compared to the newer OSX systems mozilla releng uses, but have found variances in hardware and updates/configs to cause MAJOR pain in the past.

Comment 7

[Van Le [:van]]

so i guess we'll deal with the reboots until there's a better solution. in the meantime, both hosts are up and sshable.

[[e-mail]3 ~]$ ssh !$
ssh sea-mini-osx64-1.community.scl3.mozilla.com
The authenticity of host 'sea-mini-osx64-1.community.scl3.mozilla.com (63.245.223.80)' can't be established.
RSA key fingerprint is e7:19:ba:03:b7:5b:02:8a:7a:0d:e5:7d:a6:8b:2a:35.
Are you sure you want to continue connecting (yes/no)? ^C
[[e-mail]3 ~]$ ssh sea-mini-osx64-2.community.scl3.mozilla.com
The authenticity of host 'sea-mini-osx64-2.community.scl3.mozilla.com (63.245.223.81)' can't be established.
RSA key fingerprint is e7:19:ba:03:b7:5b:02:8a:7a:0d:e5:7d:a6:8b:2a:35.
Are you sure you want to continue connecting (yes/no)?