693078 - Army of Awesome website returning an error

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Verdi [:verdi] Best to slack me]

The site was reported down about 40 minutes ago. I don't know how long it was down before that.

Comment 1

[James Socol [:jsocol, :james]]

The fastest way to bring the site back up is for someone in IT to run the following:

$ python26 manage.py shell
>>> from django.core.cache import cache
>>> from django.conf import settings
>>> cache.delete_many([settings.CC_TOP_CONTRIB_CACHE_KEY, settings.CC_TWEET_ACTIVITY_CACHE_KEY])

This is a result of bad data coming from Metrics, so additionally please stop (for now) the "get_customercare_stats" cron (it may be running on both the admin box and sumocelery1).

I will file a follow up to handle bad data better, as well as a metrics bug to understand why the data is coming in wrong, but both of those will take time. This will get AoA back up in the meantime.

Comment 2

[James Socol [:jsocol, :james]]

Please text or call me on my cell if the above doesn't work.

Comment 3

[Swarnava Sengupta (:Swarnava)]

when it will be fix? :|

Comment 4

[Michael Burns [:mburns]]

James, thanks for that insight. The site is back up and running now.

Comment 5

[Swarnava Sengupta (:Swarnava)]

this bug again

https://support.mozilla.com/en-US/forums/contributors/707685

Comment 6

[James Socol [:jsocol, :james]]

Thanks Michael!

Swarnava, like I said in comment 1, and in that thread, this can happen from time to time when we get bad data from metrics. The statistics will come back when we get that sorted out.

Comment 7

[Swarnava Sengupta (:Swarnava)]

problem is back again :|

Comment 8

[James Socol [:jsocol, :james]]

Apparently the cron was in comment 1 was not disabled and it horked itself again.

Please re-run the steps in comment 1 and disable the "get_customercare_stats" cron.

Comment 9

[Ashish Vijayaram [:ashish]]

Site back up now. Commented the get_customercare_stats cron job on sumocelery1.

Comment 10

[James Socol [:jsocol, :james]]

Thanks Ashish!

Before we close this, we should have someone familiar with SUMO's deploy process make sure this cron isn't going to get turned on again before we push a fix in code. Unless something has changed, I think that's still possible.

Comment 11

[Ashish Vijayaram [:ashish]]

:jsocol - Agreed. I've updated the next SUMO push Bug 691793 so that this doesn't get lost.

Comment 12

[James Socol [:jsocol, :james]]

It's down again, please rerun the commands in comment 1, and we really, really need to turn the cron off, for real.

Comment 13

[Michael Burns [:mburns]]

re-ran the quickfix commands.

Comment 14

[James Socol [:jsocol, :james]]

It's going to keep breaking if we can't fix the cron. It's clearly either running in two places (sumocelery and the admin box) or it's getting restored, or both. Jake Maul, Shyam, Jeremy, or Jason should be familiar enough with SUMO deploys to help.

Comment 15

[Michael Burns [:mburns]]

Agreed. I tracked the cron job down (set to get run every 6 hours) and commented it out. We can coordinate with the SUMO folks to turn it back on

root@sumocelery1:/data/www/support.mozilla.com/kitsune/scripts/crontab/crontab.tpl

Comment 16

[Shyam Mani [:fox2mike]]

Looks like this is bust again.

Also looks like crons are generated from code, so can we please get this removed from the script for now? Everything is from git so removing that will certainly help solving this faster.

Comment 17

[James Socol [:jsocol, :james]]

(In reply to Shyam Mani [:fox2mike] from comment #16)
> Looks like this is bust again.

And again, or still.

> Also looks like crons are generated from code, so can we please get this
> removed from the script for now? Everything is from git so removing that
> will certainly help solving this faster.

By the time we have our next push, we'll have worked around the problem. There can't *not* be a way to turn this off, temporarily.

The uptime over the past few days has been ridiculous, because we can't figure out how to make a cron not automatically restore itself? What is wrong with our deploy infrastructure that this is impossible. The only thing we need to do to keep this site up right now is disable this cron and keep it disabled, and re-run the steps in comment 1.

Comment 18

[Swarnava Sengupta (:Swarnava)]

when its back,statistic still not comes up :|

Comment 19

[Ashish Vijayaram [:ashish]]

oremj - assigning this bug to you, please take a look at this while pushing out Bug 691793. Thanks!

Comment 20

[Jeremy Orem [:oremj]]

Looks like someone fixed this already. We can enable it after the push again, if it is ready.

Comment 21

[James Socol [:jsocol, :james]]

(In reply to Jeremy Orem [[e-mail]] from comment #20)
> Looks like someone fixed this already. We can enable it after the push
> again, if it is ready.

Hopefully it won't re-enable itself again. We have a fix ready to go out this afternoon so we can restart the cron after our push.

Comment 22

[Jeremy Orem [:oremj]]

It looks like someone finally fixed the re-enable this morning.

Comment 23

[Swarnava Sengupta (:Swarnava)]

in right tab of statastic yesterday waa missing along with twitter contributor!

Comment 24

[William Reynolds [:williamr]]

(In reply to Swarnava Sengupta (:Swarnava) from comment #23)
> in right tab of statastic yesterday waa missing along with twitter
> contributor!

Hi Swarnava, the list of top contributors in the sidebar should be appearing again shortly. More background information can be found in James' comment on the SUMO community discussion forum.

https://support.mozilla.com/en-US/forums/contributors/707707?last=43005

Apologies for the inconveniences the past few days.

Comment 25

[Swarnava Sengupta (:Swarnava)]

Thanks for the link,

Comment 26

[Jake Maul [:jakem]]

I fixed this yesterday, but forgot to update this bug.

The cron needed to be commented out on the copy on mradm02, then committed to the internal git repo there. From there, it will propagate out to ip-admin02 (phx admin node), then to sumocelery1, where it will be made "live" (in this case, live meaning commented out).

See Bug 669388#c4 for an explanation of how this all works.


I modified the 'www' copy on mradm02. This will be overwritten on the next push, which will automatically result in the job being re-enabled.

Comment 27

[James Socol [:jsocol, :james]]

FTR, we pushed a change that fixes this on our end, so the cron can continue running as normal.

But it's good that we have a strategy for turning them off--and keeping them that way--in the future!