662209 - Restart developer-stage9.mozilla.org

Status: RESOLVED DUPLICATE of bug 661370

(mozilla.org Graveyard :: Server Operations, task)

Description

[Eric Shepherd [:sheppy]]

We need the wiki restarted on the developer-stage9 server; it's not responding at the moment.

Comment 1

[Jay Patel [:jay]]

stage9 has been very unstable and sluggish for a couple of days now.  it would be great if IT can investigate and see if we can figure out what happened.

let's hope a restart helps... but if it doesn't, there might be other stuff going on after the mindtouch v10 upgrade. :-(

Comment 2

[Eric Shepherd [:sheppy]]

It's hypothetically possible this is related to the tests I'm doing of the wiki importer tool, but realistically, it's only affecting one file or article at a time (albeit a lot of them one after another), so that seems fairly unlikely. There shouldn't be that much going on at any given moment.

Comment 3

[Dumitru Gherman [:dumitru]]

Looks like both servers that serve this staging site are in a high load because of this process:

 7365 dekiwiki  18   0 1439m 711m 1676 S 198.1 70.4  30:30.63 mono.bin

I will kick now dekiwiki, but indeed, this has been going on the whole weekend.

Comment 4

[Eric Shepherd [:sheppy]]

That's so weird. This should not be loading things that much.

Comment 5

[Ashish Vijayaram [:ashish]]

restarted services on sm-devmostage{01,02}:

[root@sm-devmostage01 ~]# service dekiwiki status
dekiwiki is running: 31584
[root@sm-devmostage01 ~]# service dekiwiki9 status
dekiwiki9                        RUNNING    pid 31644, uptime 0:34:54

[root@sm-devmostage02 ~]# service dekiwiki status
dekiwiki is running: 11519
[root@sm-devmostage02 ~]# service dekiwiki9 status
dekiwiki9                        RUNNING    pid 9071, uptime 1:36:25

Comment 6

[Ashish Vijayaram [:ashish]]

As Dumitru pointed, it does look like dekiwiki is taking up all the memory on the host causing it to swap and jack up load on them, resulting in slow loads/unresponsiveness for brief periods of time.

Comment 7

[Corey Shields [:cshields]]

reassigning to Jake, will ask him to investigate when he gets in.

Comment 8

[Jake Maul [:jakem]]

Now that the restart part of the bug is complete (by the way, I just restarted it again), I'm going to mark this as a duplicate of bug 661370, where we're investigating the load issue itself.