489762 - Stage server appears to be down

Status: VERIFIED WORKSFORME

(mozilla.org Graveyard :: Server Operations, task, P1)

Description

[Paul Booker]

Attachment: Browser message

Stage server appears to be down

Comment 2

[matthew zeier [:mrz]]

out or memory all over the console - first line was:

httpd invoked oom-killer

power cycling, box is coming back online

Comment 3

[matthew zeier [:mrz]]

something tells me this should be monitored too.

Comment 4

[Michael Morgan [:morgamic]]

That'd be awesome.

Comment 5

[Michael Morgan [:morgamic]]

We need this back online ASAP for spreadfirefox testing.  Is there an ETA?

Comment 6

[Michael Morgan [:morgamic]]

BTW - i get pinwheeled and things time out, so I don't think this is fixed yet.  Could someone peek?

Comment 7

[Michael Morgan [:morgamic]]

This blocks SFX and SUMO, so -> blocker.

Comment 8

[Jeremy Orem [:oremj]]

msnbot was crawling spreadfirefox.authstage.mozilla.com, which was killing the box. I put up http auth to prevent it from happening again.

Comment 9

[Alex Buchanan [:abuchanan]]

woot, thanks.

what can we do to prevent this production?  or will the production servers just handle this without problems?

Comment 10

[Alex Buchanan [:abuchanan]]

i still get connection interrupted.  maybe we didn't find the root cause?

Comment 11

[Laura Thomson :laura]

Still down.  reassigning to server-ops so somebody gets paged.

Comment 12

[Jeremy Orem [:oremj]]

Works for me (https://support-stage.mozilla.org/en-US/kb/).  Code issue?

Comment 13

[Michael Morgan [:morgamic]]

OK - SUMO works for stage, closing.  SFX team needs to provide instructions on how to fix SFX stage.  Will get more info from them -- until then, downgrading severity but leaving this open.

Comment 14

[Laura Thomson :laura]

SUMO is now serving up blank pages.  I get a blank homepage right now, sometimes it's the login or forum pages.  Something is still broken.

Comment 15

[Alex Buchanan [:abuchanan]]

OK, sounds like the problem is stemming from a missing table on SFx.

oremj, could you run this query.  it will disable the offending module for now.

update system set status = 0 where name = 'sfx_stats';

Thanks

Comment 16

[Jeremy Orem [:oremj]]

mysql> update system set status = 0 where name = 'sfx_stats';
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Comment 17

[Jeremy Orem [:oremj]]

(In reply to comment #14)
> SUMO is now serving up blank pages.  I get a blank homepage right now,
> sometimes it's the login or forum pages.  Something is still broken.

Restarting memcache fixed it.

Comment 18

[Stephen Donner [:stephend] Not actively reading bugmail]

https://spreadfirefox.authstage.mozilla.com/ isn't really "down", now, but it's wicked slow -- reopening.

Comment 19

[Jeremy Orem [:oremj]]

Other sites on the staging server are pretty quick, for example https://support-stage.mozilla.org/en-US/kb/.  The cachegrind for loading spreadfirefox.authstage says there was 31588 invocations of mysql_query, which explains the slowness.

Comment 20

[Jeremy Orem [:oremj]]

Looks like it is all coming from:
Called From	Count	Total Call Cost	
  sfx_stats_count_new_users @ 136 	14360 	13473 	O
  sfx_stats_count_user_logins @ 160 	14360 	12126 	O
  sfx_stats_count_new_nodes @ 90 	1861 	1760 	O
  sfx_stats_count_new_comments @ 113 	575 	571 	O

Comment 21

[Alex Buchanan [:abuchanan]]

i've turned off the sfx_stats module for now.  I think the way I used Drupal's
hook_menu() (which gets called a ton) in that module was causing the problems.


Fixing now.

Comment 22

[Stephen Donner [:stephend] Not actively reading bugmail]

Thanks, this is much zippier now -- will file a new bug if we see it slow down again.