512002 - sm-cesium01.mozilla.org very slow

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Ryan Doherty (:rdoherty)]

sm-cesium01.mozilla.org usually takes 5-20 seconds to finish requests. Our local test servers run fine and the db is virtually empty, so we're not sure why it is so slow.

Comment 1

[matthew zeier [:mrz]]

We're going to plan on upgrading the ESX host but that's about a week or so off right now.

Comment 2

[Ryan Doherty (:rdoherty)]

Talked with Matthew, getting this over to another machine sooner is important as we're showing this to people and need a qa server. Would like this on another machine early next week.

Comment 3

[Shyam Mani [:fox2mike]]

I'll probably move this along with im-webdev01.

Comment 4

[matthew zeier [:mrz]]

Wondering why we don't just put this on mrapp-stage - sounds like an almost better place since it'll eventually roll into prod?

Comment 5

[Michael Morgan [:morgamic]]

We'd have to fix whatever Python hosting issues are still out there with RHEL, no?  What's on sm-cesium01?

Comment 6

[matthew zeier [:mrz]]

Nothing at the moment.  I told Ryan that if a week or so is too long we'll move this VM to a physical server (one of the older boxes) early next week.  Didn't know it had python dependencies.

Comment 7

[Shyam Mani [:fox2mike]]

And it needs X + VNC server etc. I'd rather not introduce all that into mrapp-stage02, it's better off in a VM?

Comment 8

[Michael Morgan [:morgamic]]

Yeah, the mythical "fast VM" is needed here.  :)

Comment 9

[Shyam Mani [:fox2mike]]

mrz,

Good to move this to cm-cesium01? (like the webdev VM)? 

Ryan,

Can this move be done at anytime? or would you guys like some notice?

Comment 10

[matthew zeier [:mrz]]

Yes.

Comment 11

[Ryan Doherty (:rdoherty)]

(In reply to comment #9)
> Can this move be done at anytime? or would you guys like some notice?

Anytime is fine :) Thanks!

Comment 12

[Shyam Mani [:fox2mike]]

Migration complete, the box is now cm-cesium01. Over to mrz to update the devlb settings and I'll modify the external DNS once that's done. 

Ryan,

Your keys are also on the box. Login as root to cm-cesium01.mozilla.org and su- cesium to perform actions as the cesium user. The install is at /opt/cesium and the virtual_env is there too.

Comment 13

[Ryan Doherty (:rdoherty)]

Awesome, thanks!

Comment 14

[Ryan Doherty (:rdoherty)]

I'm able to login fine, but http://cm-cesium01.mozilla.org/ times out when I try to load any page.

Comment 15

[Shyam Mani [:fox2mike]]

Should work fine from within the VPN. Outside of that, it's not been setup yet.

mrz, comment #12 :D How do I mess with the devlb settings?

Comment 16

[matthew zeier [:mrz]]

ssh nsroot@devlb01 .  Need to add the new host into the service.  I'll walk you through.

Comment 17

[Shyam Mani [:fox2mike]]

mrz, I thought I'd done this, but it doesn't seem to work :(

devlb01> show service svc-cm-cesium01
        svc-cm-cesium01 (10.2.72.52:80) - HTTP
        State: DOWN
        Last state change was at Tue Sep  1 17:12:04 2009 (+997 ms)
        Time since last state change: 5 days, 19:59:43.910
        Server Name: cm-cesium01 
        Server ID : 0   Monitor Threshold : 0
        Max Conn: 0     Max Req: 0      Max Bandwidth: 0 kbits
        Use Source IP: NO
        Client Keepalive(CKA): NO
        Access Down Service: NO
        TCP Buffering(TCPB): NO
        HTTP Compression(CMP): YES
        Idle timeout: Client: 180 sec   Server: 360 sec
        Client IP: ENABLED X-Forwarded-For
        Cacheable: NO
        SC: OFF
        SP: OFF
        Down state flush: ENABLED

1)      Monitor Name: tcp-default
                State: UNKNOWN  Weight: 1
                Probes: 0       Failed [Total: 0 Current: 0]
                Last response: No free ports available or mapped IP / subnet IP is not configured
                Response Time: N/A

Any ideas?

Comment 18

[matthew zeier [:mrz]]

That netscaler needs some network config - I'll hit this later today.

Comment 19

[matthew zeier [:mrz]]

Okay, up.  Sorry for the delay.