Closed Bug 519298 Opened 15 years ago Closed 15 years ago

migration of production-master appears to have caused slaves to drop

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
x86
All
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: phong)

Details

Around 12:15 production-master was migrated. Very shortly after that (12:37-ish), we had 12 or so win32 slaves disconnect. I've looked into other possible causes:
* Load was not elevated compared to the same time on other days
* Nothing extra was running on the machine
* This did not happen on try, production-master02, or any other place that I've seen.

It's strange that only Windows slaves disconnected. Perhaps something happened with the production-master network connection that only affects the Win32 network stack?
Assignee: server-ops → phong
Production-master VM has over 3GB of RAM allocated.  Whenever RAM gets really high, it takes a longer time for VMs to migrate.
Migration shouldn't interfere with active TCP sessions.  If it does, we need to get a fix, or stop migrating VMs.
I also noticed that the CD-ROM is set to client device instead of "host device".  This could also cause it to take longer to migrate.  I've edited the settings to make the change.  I'm sending the logs over to vmware to investigate.
Phong showed me load charts from VI:

12:11: production-master VM migrated, and also showed a massive drop in connections and load at that time for the VM. 
12:25 (approx): production-master VM migrated again, this time maintaining connections & load.

The rest of the VMs, and the ESX host, looked healthy.
OS: Mac OS X → All
already done
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
(In reply to comment #3)
> I also noticed that the CD-ROM is set to client device instead of "host
> device".  This could also cause it to take longer to migrate.  I've edited the
> settings to make the change.  I'm sending the logs over to vmware to
> investigate.

Did we hear back from VMware?
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.