1331056 - memory of treeherder tabs increases faster

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The memory allocation for treeherder tabs increased earlier this week (KWierso also observed this behavior).

While in the past a treeherder tab might have allocated up to 1 GB if there was much push activity, I reloaded the treeherder tabs for all 7 branches earlier today and they have already increased in total to 5.8 GB (1.6 GB each for autoland and inbound).

jgraham's observation using the memory snapshot tools: Chrome had gone from 55Mb to 69Mb in maybe 10-15 minutes, Firefox from 34-51Mb in a little longer.

Using Firefox's about:memory, the explicit allocation for mozilla-inbound doubles (to 100 MB) in less than 15 minutes.

Comment 1

[William Lachance (:wlach)]

This sounds pretty serious, I suspect we should probably get someone to look into it sooner than later. :needinfo'ing jgriffin

Comment 2

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Most of the memory usage in my case is showing up as "orphan-nodes" ("those that are only reachable from JavaScript objects"). Which seems pretty leaky.

Comment 3

[William Lachance (:wlach)]

Semi-related, I've felt that the Treeherder front-end is rather inefficient at using memory for quite a while, finally got around to filing bug 1331399 about it. I don't necessarily think what I describe in that bug is the problem here (in fact I suspect it isn't), but it's something to look into if we're concerned about this issue in general.

Comment 4

[Jonathan Griffin (:jgriffin)]

Cam, any idea what we might be leaking here?

Comment 5

[Cameron Dawson [:camd]]

Hey Jonathan--  I'll take a look.

Comment 6

[Phil Ringnalda (:philor)]

Please ram the fix into production as soon as humanly possible, and comment in the bug about when it hits production, not just about when it hits staging, because I've given up. I managed to keep all seven treeherder tabs that sheriffing requires open over the weekend, by using every trick I know and some I created (and mostly by having very very few pushes running over a holiday weekend), but even a weekday evening much less a weekday day, it's just intolerable, so I've closed all my treeherder tabs, and we just aren't going to have the trees starred for a while.

Comment 7

[William Lachance (:wlach)]

(In reply to Phil Ringnalda (:philor) from comment #6)
> Please ram the fix into production as soon as humanly possible, and comment
> in the bug about when it hits production, not just about when it hits
> staging, because I've given up. I managed to keep all seven treeherder tabs
> that sheriffing requires open over the weekend, by using every trick I know
> and some I created (and mostly by having very very few pushes running over a
> holiday weekend), but even a weekday evening much less a weekday day, it's
> just intolerable, so I've closed all my treeherder tabs, and we just aren't
> going to have the trees starred for a while.

It would have been good to know that this was in the "blocker" rather than the "merely annoying" category earlier, but I'll take a look now (since Cam isn't in yet) now that we know that it is.

Comment 8

[William Lachance (:wlach)]

Ok, after some digging I think it's bug 1242905 that's the culprit, specifically commit 9d54ed77c7d8eaeb27ee3bb03175be223b28c129.

Reverting that, on Chrome I get the following numbers doing a heap profile and loading the same set of pushes on mozilla-inbound (as of about 8:15 EST):

* With revert: 62.2M
* Without revert: 67.9M

But re-running the same profile on the same window after ~20 mins (and a few more pushes have loaded):

* With revert: 67.8M
* Without revert: 81.5M

In parallel, I ran the test on a single push: even there the memory increased much faster (17.6->23.1M vs. 14.3->17.4M)

I'm going to revert that particular commit, hopefully Casey can come up with a fix and we can reland.

Comment 9

[William Lachance (:wlach)]

Revert deployed to production, hopefully this is fixed now.