392358 - Need to diagnose what's wrong with Socorro production (crash-stats.m.c)

Status: RESOLVED INVALID

(mozilla.org Graveyard :: Server Operations, task)

Description

[Adam Guthrie]

15:17 < aravind> wow.. clicking on the links in the top crasher page on the
                 reporter makes the paster process consume about 370 MB
                 resident memory and takes forever to return.
15:17 < aravind> is this a known issue?
15:18 < ispiked> no.
15:18 < ispiked> so it appears to be leaking?
15:19 < aravind> either that or its stuck in some kind of loop
15:20 < aravind> and even after that page is done loading the paster process
                 seems to be consuming more and more memory.
15:21 < aravind> its now upto 460MB resident..
15:25 < aravind> yeah, I am killing it, I think its just going to consume
                 more and more memory..

Benjamin said that he and Aravind discussed this some yesterday, but it's still unclear what caused the problem.

Comment 1

[Aravind Gottipati [:aravind]]

Let me know what you guys need to troubleshoot this.  Can you replicate this on your dev environments, etc?  I can provide production db dumps if it helps.

Comment 2

[Benjamin Smedberg]

Aravind, can we try removing the contents of the cache directory?

Comment 3

[Aravind Gottipati [:aravind]]

In the production.ini file
cache_dir = /tmp/reporter_data

And that directory doesn't even exist on the server.

Comment 4

[Benjamin Smedberg]

Hrm... do you know if that always has been the case? I think we need to have a cache directory.

Comment 5

[Aravind Gottipati [:aravind]]

I think that even though this stuff has been configured, its never been used.  I seem to remember something vague about not finding this stuff when I was trying to troubleshoot stuff with ispiked a while back either.

Comment 6

[Justin Fitzhugh]

Morgamic is handling the memory bugs - nothing for serverops to do.