493321 - Parallels Servers VMs networking dropping.

Status: RESOLVED WONTFIX

(mozilla.org Graveyard :: Server Operations, task)

Description

[Phong Tran [:phong]]

Parallels dropping the network off guest VMs totally more than 8.  Rebooting recovers the VMs, but other VMs will dropping off in their place.

Comment 1

[Phong Tran [:phong]]

I can't seem to figure out which VM is offline right now.  They want me to send the logs when this happen and they can investigate.

Comment 2

[Robert Kaiser]

cb-seamonkey-osx-03 and cb-seamonkey-osx-04 are offline right now and have been since the last attempt to bring everything online (which are the new ones created for bug 492224). AFAIK, every time we brought those online, cb-seamonkey-linux-02 and cb-seamonkey-win32-02 went offline and vice versa.
Bug 493450 could be related but may just be yet another Parallels issue, that one is Mac VM instability.

Comment 3

[Phong Tran [:phong]]

I'll add this the parallels ticket.

Comment 4

[Robert Kaiser]

I see that you rebooted -osx-03 and -osx-04 yesterday and they came online at came 18:44 and 18:52, respectively. As I had predicted, we lost -linux-02 at 19:07 and -win32-02 at 19:16 after that.
-osx-04 also disconnected again a few hours later at 00:00. All those three VMs can't even be pinged (which usually works when they're online), so it's the network problem for all of them, probably.

Comment 5

[Phong Tran [:phong]]

According to Parallels support, the server can only support 7 VMs for an 8 core server.  You can have more than 7 VMs, but you can only have 7 running at a time.  They want me to reproduce the errors.  Can I schedule a time with you to do this so I don't disrupt your builds and have you verify that the VM is dropping off.

Comment 6

[Robert Kaiser]

Sure, we can do that any time you like if you tell me early enough before it.
What's interesting in that light is that we can have 8 VMs running all the time...

Comment 7

[Phong Tran [:phong]]

I've dropped the OSX VMs down to just 1 CPU.  We'll see if this helps any, but let me know if this affects the performance of the VMs.

Comment 8

[matthew zeier [:mrz]]

What's the outcome here?  Did dropping the OSX VMs to 1 vCPU help any?  Are we just limited in the number of active VMs?

Comment 9

[Robert Kaiser]

Dropping to 1 CPU didn't help at all with the network problem, it did help to vastly reduce the instabilities of the VMs, but the network problem seems just to depend on the number of active VMs - we cannot run more then 8 VMs on the host, it seems.

Comment 10

[matthew zeier [:mrz]]

Phong, anything else to do?  Limitation in the software?

Comment 11

[matthew zeier [:mrz]]

I'd like to close this case.  

Appears this is just a software limitation that we can't work around at this point.

Kairo, are you okay with the VMs that are running?  Re-open if there's something here to investigate more.

Comment 12

[Robert Kaiser]

If we can't get the two VMs up that we are missing because of this, we need some other solution to get them, or the SeaMonkey build environment in unmanageable in the long run (we haven't turned on any unit tests for trunk and we already get builds queued up regularly, resulting in overly long wait times to see if a checkin caused or cleared problems).

If anything, you're not telling that this issue if FIXED but rather that you see it as WONTFIX, in which case I'd very much like to see an alternative of how we can get at least those machines that we were told we'd get.

As bad as it sounds, it looks very much like the Parallels experiment didn't really go well, as the report here and bug 494671 that cripples the existing OSX VMs to a barely working state show.

This is not any of our faults though, it simply seems that Parallels doesn't work as expected, unfortunately.

Comment 13

[matthew zeier [:mrz]]

> If anything, you're not telling that this issue if FIXED but rather that you
> see it as WONTFIX, in which case I'd very much like to see an alternative of
> how we can get at least those machines that we were told we'd get.

If there were a CANTFIX, I'd use that.

I suspect we'll have to go back to sethb on the additional machines - the only approval I had from Community Giving was for the XServe and the software.