964730 - timestudy2.vm.labs.scl3.mozilla.com is very slow

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Labs, task)

Description

[Olivier Yiptong [:oyiptong]]

The VM found at timestudy2.vm.labs.scl3.mozilla.com is extremely slow.

Running apt-get makes the load average shoot up to 4-5 and even sometimes 12.
gozer took a quick look and thought it could be IO related.

Is the machine the VM running on perhaps having some issues?

We use the machine as a staging environment, thankfully. The production VM at timestudy2.vm.labs.scl3.mozilla.com isn't affected.

Comment 1

[Philippe M. Chiasson (:gozer)]

(In reply to Olivier Yiptong [:oyiptong] from comment #0)
> [...]
> 
> We use the machine as a staging environment, thankfully. The production VM
> at timestudy2.vm.labs.scl3.mozilla.com isn't affected.

I believe here you meant that timestudy*1* isn't affected, correct ?

Comment 2

[Olivier Yiptong [:oyiptong]]

Whoops. Yes, timestudy1.* is unaffected.

Comment 3

[C. Liang [:cyliang]]

In Labs, VMs are housed on two servers in the larger virtualization cluster: one node acts as the primary host and the secondary host is kept in synch in case the first node goes down.  The server hosting the primary node shows moderately higher IOWait times, so I attempted to live-migrate timestudy2 (i.e. swap the primary and secondary nodes).  This failed -- perhaps due to IOWait issues.

We'll be bringing in some additional hardware which should add capacity and we can try another live-migration.  I can also try a more forceful move before then, but this will require briefly shutting down the VM.  Looking at the services that are running (nginx, redis, etc.) -- do you have these service set up to start on reboot of the server or is there a script / set of scripts that can be run after a reboot?

Comment 4

[Olivier Yiptong [:oyiptong]]

Hey cyliang. The servers should restart without any issues as the init scripts are setup