795017 - nss-vm-darwin9-1.community.scl3.mozilla.com and nss-vm-darwin10-1 are offline

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Kai Engert (:KaiE:)]

Both nss-vm-darwin9-1.community.scl3.mozilla.com and nss-vm-darwin10-1.community.scl3.mozilla.com are offline.

When I try to connect to them from jump1.community.scl3.mozilla.com
I get "no route to host".

Could you please have a look? Routing issue or VMs down? Thank you.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Looks like parallels2 *and* parallels3 are down, but not parallels1, suggesting that the sonnet shelf got unplugged or the like..


dmitchell@jump1 ~ $ ping parallels3
PING parallels3.community.scl3.mozilla.com (63.245.223.18) 56(84) bytes of data.
From jump1.community.scl3.mozilla.com (63.245.223.8) icmp_seq=1 Destination Host Unreachable
From jump1.community.scl3.mozilla.com (63.245.223.8) icmp_seq=2 Destination Host Unreachable
From jump1.community.scl3.mozilla.com (63.245.223.8) icmp_seq=3 Destination Host Unreachable
^C
--- parallels3.community.scl3.mozilla.com ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4091ms
pipe 3
dmitchell@jump1 ~ $ ping parallels2
PING parallels2.community.scl3.mozilla.com (63.245.223.17) 56(84) bytes of data.
From jump1.community.scl3.mozilla.com (63.245.223.8) icmp_seq=1 Destination Host Unreachable
From jump1.community.scl3.mozilla.com (63.245.223.8) icmp_seq=2 Destination Host Unreachable
From jump1.community.scl3.mozilla.com (63.245.223.8) icmp_seq=3 Destination Host Unreachable
^C
--- parallels2.community.scl3.mozilla.com ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3516ms
pipe 3
dmitchell@jump1 ~ $ ping parallels1
PING parallels1.community.scl3.mozilla.com (63.245.223.22) 56(84) bytes of data.
64 bytes from parallels1.community.scl3.mozilla.com (63.245.223.22): icmp_seq=1 ttl=64 time=0.282 ms
64 bytes from parallels1.community.scl3.mozilla.com (63.245.223.22): icmp_seq=2 ttl=64 time=0.312 ms
^C
--- parallels1.community.scl3.mozilla.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1566ms
rtt min/avg/max/mdev = 0.282/0.297/0.312/0.015 ms

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Both parallels2 and parallels3 had frozen, having run out of swap.  Googling around for the error message didn't really find any good explanations.

We restarted both, and upgraded them to 10.7.5.  Let's see if that's more stable.

Comment 3

[Kai Engert (:KaiE:)]

Thank you, but darwin10 is still offline (no route to host)

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Hm, it had been stuck in the "Starting" state.  I rebooted parallels3 again, and now the VM is up.

Comment 5

[Kai Engert (:KaiE:)]

Thank you.

It's really surprising how unreliable these VMs are.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

That's a kind word for it!

Comment 7

[Kai Engert (:KaiE:)]

again both are offline.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

dcops/oncall, maybe it's time for a support call to parallels?

Or, just abandon parallels altogether?  Although it's not clear how we'd find hardware for a snow-leopard system.  Maybe we should just stop supporting OS X, since Apple seems determined to make it impossible for us to ship software for it :(

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Oncall, we're kinda out of ideas here.  What do you think the best fix is?

I *think* this hardware can run Darwin10 (Lion).  I think it can't run Snow Leopard, but I'm not certain.  I suspect Darwin10 would run better outside of parallels.

Comment 10

[Kai Engert (:KaiE:)]

The problem is that we are about to release a new major release of NSS 3.14 by Thursday, we received reports about a potential Mac OSX specific bug (specific either to 64 bit or specific to 10.7), and we currently have zero testing coverage for those environments.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

DCOps, can you reboot these?

Comment 12

[Van Le [:van]]

Hosts have been rebooted and are pingable.

[[e-mail]3 ~]$ fping parallels3.community.scl3.mozilla.com
parallels3.community.scl3.mozilla.com is alive
[[e-mail]3 ~]$ fping parallels2.community.scl3.mozilla.com
parallels2.community.scl3.mozilla.com is alive

Comment 13

[Kai Engert (:KaiE:)]

[[e-mail]3 ~]$ ssh [e-mail]
ssh: connect to host nss-vm-darwin10-1.community.scl3.mozilla.com port 22: No route to host

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

I'll get that.  Thanks, Van!

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

They're both up, and I logged in on the console on both.  Sorry we got into long-term solutions and forgot the short-term solution.

Back to server-ops to figure out that long-term solution, or at least the next step.

Comment 16

[Kai Engert (:KaiE:)]

Whatever you did, thanks a lot for keeping both machines online and stable since the last comment (except darwin9 being shut off once).

Closing this bug for now.