Closed Bug 681201 Opened 13 years ago Closed 11 years ago

[meta] Not-yet-restored empty tabs take more memory than they should

Categories

(Core :: General, defect)

defect
Not set
normal

Tracking

()

RESOLVED FIXED

People

(Reporter: glandium, Unassigned)

References

(Blocks 2 open bugs)

Details

(Keywords: meta, Whiteboard: [MemShrink:P3][Snappy:P2])

Attachments

(3 files)

This is something I noticed on my 450+ tabs session at startup with "Don't load tabs until selected" preference checked: It takes a lot more memory than starting Firefox without any tabs loaded.
This is what about:memory?verbose displays after I create ~200 new tabs (ctrl+t)
For reference, this is the output of about:memory?verbose when running "firefox about:memory?verbose" from the command line, with no session-restore. That is, about:memory is the only tab loaded.
The increases are mostly due to:

- js/system-compartment: 8.6MB -> 43.5MB
- js/about:blank:          0   -> 15.2MB
- layout:                785KB -> 27.5MB
- heap-unclassified:    12.9MB -> 45.3MB

The system-compartment one is the most surprising/worry to me.  Layout's also odd, that's a lot of memory to display blank pages.

Mike, what add-ons do you have enabled?  If you have some enabled, does disabling them change the result?
> Mike, what add-ons do you have enabled?  If you have some enabled, does
> disabling them change the result?

I checked the test profile I was using for the tests that got me these values, and the only add-on installed is DOM inspector, and it's disabled.
Whiteboard: [MemShrink] → [MemShrink:P1]
We made this a P1 because lazy tab loading is getting increasing emphasis now that there's UI to turn it on, and there's a rumour it might be turned on by default in the future?
From chatting with jst, the immediate thing the front-end may be able to do here is to see how we can avoid creating all these about:blanks, by having Session Restore change how we create deferred-loading tabs. Sounds like will require lazily creating the <browser> in each -- that could create some interesting fun for other code/addons that might be expecting that to always exist (though they may be partially-broken anyway in the deferred restore world...).

zpao, can you take a look at this?

njn: was this bug for more than just the above front-end / session-restore work? IE, shall we do that in a new, dependent bug? jst mentioned that there's other work going on to optimize the memory usage of about:blank et all as well, wasn't sure how broad the scope of this bug is.
(In reply to Justin Dolske [:Dolske] from comment #6)
> 
> njn: was this bug for more than just the above front-end / session-restore
> work? IE, shall we do that in a new, dependent bug? jst mentioned that
> there's other work going on to optimize the memory usage of about:blank et
> all as well, wasn't sure how broad the scope of this bug is.

Lazy session-restore seems like the only interesting/realistic case where you can end up with dozens or hundreds of blank tabs.  Hitting Ctrl-T 100 times in a row doesn't seem interesting/realistic :)

I'm not sure what that other work is that jst mentioned;  the only thing I can think of is bug 673208 but that's basically the same issue so I'll make it a dup of this bug.

I've retitled this bug accordingly.
Summary: Empty tabs take a lot of memory → Not-yet-restored empty tabs take a lot of memory
The other work I was alluding to is the collective work to minimize the amount of overhead per compartment, all the clown shoes work, and all the other MemShrink work that's going on, nothing specific that we'll need this bug for. Giving this to zpao, since it sounded like he'd be most likely to be working on this.
Assignee: nobody → paul
So a quick test with session restore off (syntax error in there so the component isn't even loaded) shows for 100 tabs the about:blank compartment using 7.06 MB and sys princ at 25MB (layout ~13mb)

I opened the 100 tabs with this run from the error console
> for (var i = 0; i<100; i++) { window.top.opener.gBrowser.addTab("about:blank") }

I then took a session with 100+ tabs (this was a browsing session I've been building over a couple weeks, so not overly simple). I did 2 things: cut out all of the work that session restore does to see how that would affect things. I didn't set up session history and I stopped caching data on the browser.

I was at 6.84 MB and sys princ was at 30MB

Then under a normal restore with restore on demand enabled, 6.8MB and sys princ at 30.6MB

Anyway, that all just says that simply having the tabs created is pushing memory usage up. It's not directly related to session restore.

I can look into doing some magic to make tab creation somewhat lazy, but that sounds like dangerous magic.
It has been reported that BarTab (in 3.6, maybe a beta in 4.x) used less memory per unloaded tab than avoiding the load with max_concurrent_tabs=0.  As to why, I'm not sure, but the BarTab code is available.  Note: I didn't measure it myself, and this was for an older version.  The code causing that might have changed.
(In reply to Paul O'Shannessy [:zpao] from comment #10)
> Anyway, that all just says that simply having the tabs created is pushing
> memory usage up. It's not directly related to session restore.

Yes, I think you're right, the creation of the tabs alone, and the DOM windows inside them (i.e. the <browser> element) in particular is what's costing here. I don't think session restore data is what's using a lot of memory here.

> I can look into doing some magic to make tab creation somewhat lazy, but
> that sounds like dangerous magic.

That would be awesome, at least if we could come up with something that's stable enough to experiment with we could get numbers and determine whether it's worth doing this (which I think it is).
Improving session restore for this bug may (or may not) improve another thing I noticed with my 450+ tabs session: It takes 7 seconds to reach the sessionRestored state in a warm startup.
(In reply to Randell Jesup [:jesup] from comment #11)
> It has been reported that BarTab (in 3.6, maybe a beta in 4.x) used less
> memory per unloaded tab than avoiding the load with max_concurrent_tabs=0. 

True, that was bug 632012.
(In reply to Johnny Stenback (:jst, jst@mozilla.com) from comment #12)
> (In reply to Paul O'Shannessy [:zpao] from comment #10)
> > Anyway, that all just says that simply having the tabs created is pushing
> > memory usage up. It's not directly related to session restore.
> 
> Yes, I think you're right, the creation of the tabs alone, and the DOM
> windows inside them (i.e. the <browser> element) in particular is what's
> costing here. I don't think session restore data is what's using a lot of
> memory here.

Couldn't we create all that stuff lazily, only when we need them?
The number of tabs and the number of browsers (gBrowser.tabs and gBrowser.browsers) being out of sync would certainly cause problems.
(In reply to Dão Gottwald [:dao] from comment #17)
> The number of tabs and the number of browsers (gBrowser.tabs and
> gBrowser.browsers) being out of sync would certainly cause problems.

Is it possible for us to stop depending on that invariant?
Sure, somebody could audit our code. I do not think it's worth it, and we can't do it for add-ons.
So the conclusion here is that this is too hard to be worth it? Is that for the current default behavior, or does that hold true in a future version where we don't load stuff in background tabs by default on session restore as well?

I'm still of the opinion that this will be a significant memory gain, and probably a significant session restore performance gain when lazily loading background tabs too. But feel free to tell me I'm wrong.
(In reply to Johnny Stenback (:jst, jst@mozilla.com) from comment #20)
> So the conclusion here is that this is too hard to be worth it? Is that for
> the current default behavior, or does that hold true in a future version
> where we don't load stuff in background tabs by default on session restore
> as well?

I think the point we're at is that we need to figure out if half loading a tab is even a realistic possibility. So much of our code & addons expect gBrowser.tabs & gBrowser.browsers to be populated with tabs & browsers. And they expect to be able to poke & prod these things. We could fix our code to be better to avoid fully loading a tab, but we can't do that for addons.

> I'm still of the opinion that this will be a significant memory gain, and
> probably a significant session restore performance gain when lazily loading
> background tabs too. But feel free to tell me I'm wrong.

All I know is my gut says, "maybe". The memory/performance gains may come at more cost than they're worth. Creating a tab takes some memory. Storing the data for that not-yet-restored tab takes some memory. But (I think) almost any web page is going to use more memory than what we just did.
FWIW, it not only sucks (a little) memory, it sucks seconds of startup time too.
Does anyone have an actual plan for moving this bug forward?  A big chunk of the memory looks to be layout, specifically style data.  What's the style data for a blank or unloaded page look like?  Could it be simplified?  Why do we have any style data at all in that case?

For example, here's the style data measurements (which are underestimates! See bug 671299) for a session just started with 149 unloaded tabs:

├───24,539,432 B (18.22%) -- layout
│   ├──22,755,048 B (16.90%) -- shell(about:blank)
│   │  ├──20,548,040 B (15.26%) -- styledata [155]
│   │  └───2,207,008 B (01.64%) -- arenas [155]

That's more than double the JS memory used for those pages (most of which is due to internal fragmentation):

│   ├──11,104,000 B (08.25%) -- compartment(about:blank)
│   │  ├──10,268,672 B (07.63%) -- gc-heap
│   │  │  ├───5,592,472 B (04.15%) -- arena
│   │  │  │   ├──5,379,320 B (03.99%) -- unused [155]
│   │  │  │   ├────132,928 B (00.10%) -- padding [155]
│   │  │  │   └─────80,224 B (00.06%) -- headers [155]
│   │  │  ├───2,418,024 B (01.80%) -- objects
│   │  │  │   ├──2,082,688 B (01.55%) -- function [155]
│   │  │  │   └────335,336 B (00.25%) -- non-function [155]
│   │  │  ├───2,115,072 B (01.57%) -- shapes
│   │  │  │   ├──2,099,136 B (01.56%) -- tree [155]
│   │  │  │   └─────15,936 B (00.01%) -- dict
│   │  │  ├─────112,768 B (00.08%) -- type-objects [155]
│   │  │  └──────30,336 B (00.02%) -- scripts [155]
│   │  ├─────565,600 B (00.42%) -- object-slots [155]
│   │  └─────269,728 B (00.20%) -- shapes-extra
│   │        ├──129,824 B (00.10%) -- tree-tables [155]
│   │        ├───90,720 B (00.07%) -- empty-shape-arrays [155]
│   │        ├───45,056 B (00.03%) -- tree-shape-kids [155]
│   │        └────4,128 B (00.00%) -- dict-tables
DMD told me that RuleHash was taking up a fair amount of space, esp. mTagTable.  Turns out it's initialized to a capacity of 64, when the other tables in RuleHash are initialized to 16.  I loaded Gmail and TechCrunch and got this distribution of sizes when RuleHashes died:

1458 lines:
( 1)    417 (28.6%, 28.6%): die: 0
( 2)    416 (28.5%, 57.1%): die: 1
( 3)    177 (12.1%, 69.3%): die: 2
( 4)    153 (10.5%, 79.8%): die: 233
( 5)     96 ( 6.6%, 86.4%): die: 5
( 6)     80 ( 5.5%, 91.8%): die: 3
( 7)     63 ( 4.3%, 96.2%): die: 6
( 8)     15 ( 1.0%, 97.2%): die: 53
( 9)     13 ( 0.9%, 98.1%): die: 25
(10)     13 ( 0.9%, 99.0%): die: 20
(11)      4 ( 0.3%, 99.2%): die: 23
(12)      4 ( 0.3%, 99.5%): die: 52
(13)      2 ( 0.1%, 99.7%): die: 7
(14)      2 ( 0.1%, 99.8%): die: 30
(15)      1 ( 0.1%, 99.9%): die: 33
(16)      1 ( 0.1%, 99.9%): die: 14
(17)      1 ( 0.1%,100.0%): die: 240

85% of the tables would easily fit in a capacity of 16, so it looks like that table should be made smaller to begin with.  Given that each entry is 56 bytes (on 64-bit platforms) that could easily save a few MB.
(In reply to Nicholas Nethercote [:njn] from comment #24)
> 85% of the tables would easily fit in a capacity of 16, so it looks like
> that table should be made smaller to begin with.  Given that each entry is
> 56 bytes (on 64-bit platforms) that could easily save a few MB.

Most would even fit in a capacity of 8.
> Most would even fit in a capacity of 8.

The current minimum is 16, thought it wouldn't be hard to change.  In bug 697933 I changed js::HashTable to allow it to be as small as 4, which was a win in certain cases.
BTW, I saw that something like 10 or 12 RuleHash objects are created each time you open a new, blank tab.  I'd love to know why they even need to be created.
(In reply to Nicholas Nethercote [:njn] from comment #27)
> BTW, I saw that something like 10 or 12 RuleHash objects are created each
> time you open a new, blank tab.  I'd love to know why they even need to be
> created.

Basically, because (as of somewhat recently), we have not only a separate RuleHash for each level of the cascade (as we always have), but also a separate RuleHash for each pseudo-element (see nsCSSPseudoElementList.h) that occurs in the style sheets at that level.  Given that, it probably does make sense to make them smaller by default -- perhaps by not initializing each of the tables until it's needed.
Also, we should share author and user level rule processors across documents (bug 77999).
> Basically, because (as of somewhat recently), we have not only a separate
> RuleHash for each level of the cascade (as we always have), but also a
> separate RuleHash for each pseudo-element (see nsCSSPseudoElementList.h)
> that occurs in the style sheets at that level.

Sorry for the noob questions, but why do we have multiple levels of cascade and pseudo-elements for a blank tab?
Blank isn't *really* blank :-)

Could bug 77999 lead to a lot of extra mem use when you have hundreds of unloaded tabs?  Some obviously, but how much?
Levels of the cascade are described in http://www.w3.org/TR/CSS21/cascade.html#cascade and perhaps more precisely in our model in http://hg.mozilla.org/mozilla-central/file/53f4c8abf558/layout/style/nsStyleSet.h#l244 http://hg.mozilla.org/mozilla-central/file/53f4c8abf558/layout/style/nsStyleSet.cpp#l253

A blank tab is displaying about:blank, which is a simple (but nonempty) HTML document, which we use CSS style sheets to display.  Thus user and UA style sheets still apply.  (If somebody has body { background: red } in a user style sheet, or unusual color preferences in the preference style sheet, it should apply to about:blank.)

For many users, the user style sheet level is empty.  And I believe there are no pseudo elements at all in our UA sheets.
(In reply to Randell Jesup [:jesup] from comment #31)
> Could bug 77999 lead to a lot of extra mem use when you have hundreds of
> unloaded tabs?  Some obviously, but how much?

Do you mean could fixing it lead to that, or could the lack of fixing it lead to that?

I'd think we should need only one rule processor for user sheets and one for UA sheets, and then one for each XUL document (which we already cache in the XUL prototype cache).
lack of fixing it means we're using extra memory, I assume - the question is how much.
I agree that we should consider lazy init for the rulehash hashtables, and furthermore that we should reevaluate their default sizes.  Filed bug 700914.

The reason there are a bunch of rulehashes created for each about:blank is in fact that our UA sheets have pseudo-element styles in them.  In particular, ones for ::-moz-list-bullet, ::-moz-list-number, ::-moz-progress-bar, ::before (for broken image/object styles and <q>), ::after (for <1>), ::-moz-focus-inner (we should get rid of this).  So we get those 6 rulehashes, plus the two "main" rulehashes for the UA and user levels.

Fixing bug 77999 would let us share most of this stuff across all tabs, of course.
Depends on: 700914
Depends on: 77999
Is it the exact same operation that allocates too much memory to a not yet (and potentially never in this session) loaded tab that also slows down startup to a crawl accordingly? From a UX POV, not much needs to be loaded from cache/initialised about a tab in the session that isn't the currently active one (and may not be used at all) besides obviously url, favicon & page title.
Hard to say without profiling.
See comment 13. We also appear to be doing something that I'm not sure what it is that leads to having security popups when a tab in the session that is being restored was on a security warning state (and is not the one restored at startup), which suggests more than creating the tabs and about:blank is happening.
(In reply to Mike Hommey [:glandium] from comment #38)
> See comment 13. We also appear to be doing something that I'm not sure what
> it is that leads to having security popups when a tab in the session that is
> being restored was on a security warning state (and is not the one restored
> at startup), which suggests more than creating the tabs and about:blank is
> happening.

We're restoring session history into each browser, just not telling shistory to load the entry. But afaik that shouldn't trigger any security popups. Do you have an example where this is happening?
Assignee: paul → nobody
Keywords: meta
OS: Linux → All
Hardware: x86_64 → All
Whiteboard: [MemShrink:P1] → [MemShrink:P1][Snappy:P2]
Bug 711193 is proposing to make lazy session restore the default.  If that happens, this bug becomes more important.
Depends on: 711193
Arguably, switching lazy session restore as the default won't make things worse than they are already.
I agree that lazy session restore won't make things worse, but I also agree that it will make this bug more important.

The disparity between expectation & reality equals frustration and many users will think that an unloaded tab equals zero bytes and will be frustrated when this is clearly not the case (to the point of affecting performance).
Slightly technical users will also now understand that they can use more tabs with greater freedom, potentially increasing our pool of heavy tab users.
[Add in a vision of users managing large numbers of tabs in tab groups across sessions and] I also think that this bug is going to become much more visible with bug 711193.

(I do understand we cannot make unloaded tabs zero, but there from the discussion here it sounds like there are wins to be had.)
The simplest way to minimize the memory use, imo, is to simply not create the <browser> elements until they're actually needed.  Or maybe I just say that because that's not a core change.  ;)
See also bug 715612 - related, but specifically about the start-up time effect of creating lots of un-restored tabs.
Summary: Not-yet-restored empty tabs take a lot of memory → Not-yet-restored empty tabs take more memory than they could
Whiteboard: [MemShrink:P1][Snappy:P2] → [MemShrink:P2][Snappy:P2]
Blocks: 735914
Summary: Not-yet-restored empty tabs take more memory than they could → Not-yet-restored empty tabs take more memory than they should
since someone asked of how much memory, unloaded tabs use, here is a real-world example : in ffx14b(osx) 900 unloaded tabs take ~775MB of memory (in addition to the 125MB an instance with 1 tab takes (loaded in both cases)).

that amounts to ~1.16MB per unloaded tab.

considering that from a UI perspective only the favicon, page title, url & tabgroup relation are user-visible (~5 show thumbnails in panorama) and from a programmers perspective additionally only some sort of id to load the saved tab state (including history,…) are necessary the total data needed per unloaded tab is only a few KB.

now i understand this is purely theoretical but looking at gBrowser.tabs & browser all i see about 20 objects in browser (boolean, int, enums & strings should be marginal e.g. less than a few KB) which for an unloaded tab could all be null.

so essentially i still think the possible savings are around 1MB per tab.
Also restoreable (but not user-visible until you start to load the tab) is the forward/back history.  Invisible (but needed once you touch the tab) is session storage data (and I assume other things).
a quick search on AMO for "history" suggests quite a few addons ("tab history menu", "CopyAllUrls", "duplicate tab") actually use the history on other than the current tab (& make them user visible). if i am assuming correctly others like "tab preview" access the session storage data for previews. of course these are only the most visible extensions.

Wouldn't it be possible to change the implementation on the concerning properties getters (on browser objects in gBrowser.browsers) to load & instantiate the necessary objects (into the current unloaded state) an api call to one of the properties happens? and wouldn't this remove almost all of the memory usage of a tab? 

sry if this is obviously not possible but i am unfamiliar with ffx code.
I had heard that about:memory would eventually be able to give us a better idea of per-tab memory usage. If that will be the case then I'm guessing that would really help us in tracking down what is using up unloaded tab memory...no?

Or are unloaded tabs stored in a big blob/file of some sort?
(In reply to Mark S. from comment #48)
> I had heard that about:memory would eventually be able to give us a better
> idea of per-tab memory usage. If that will be the case then I'm guessing
> that would really help us in tracking down what is using up unloaded tab
> memory...no?

We'll see when the patches in bug 687724 land, which should be later today.
(In reply to Claude Nobs from comment #45)

> in ffx14b(osx) 900 unloaded tabs take ~775MB of memory

That's frightening. And Camino seems to have the same bloat problem. The memory amount used by Firefox should not increase with the number of not-loaded tabs — or only by an epsilon.
(In reply to Randell Jesup [:jesup] from comment #46)

> Also restoreable (but not user-visible until you start to load the tab) is
> the forward/back history.  Invisible (but needed once you touch the tab) is
> session storage data (and I assume other things).

The auto-saved form data entered has to be restored too. But we don't need to load all this kind of data in memory at once. If we do that, we get a loooooong startup time and a memory bloat. It would we nicer to leave this kind of data in a file on disk (I think the file currently used is named "sessionstore"), make sure the data does not get erased, and load each piece of data lazily, only when needed.
(In reply to Randell Jesup [:jesup] from comment #46)

> Also restoreable (but not user-visible until you start to load the tab) is
> the forward/back history.  Invisible (but needed once you touch the tab) is
> session storage data (and I assume other things).

The auto-saved form data entered has to be restored too. But we don't need to load all this kind of data in memory at once. If we do that, we get a loooooong startup time and a memory bloat. It would be nicer to leave this kind of data in a file on disk (I think the file currently used is named "sessionstore"), make sure the data does not get erased, and load each piece of data lazily, only when needed.
dumping form data to a file has some security implications in the event of a crash - i would not want a credit card number or SSN to be dumped into a plaintext (or any auto-decryptable) file when a tab gets unloaded. i imagine this kind of data does not take much space anyhow - no form is so massive, it should be kept in mem IMO.
Forms can be pretty large - see pastebin.  However, by modern disk standards they aren't large, or modern VM standards, outside of a few possible edge cases.  The size *may* be slightly relevant on Mobile, however.

Form security is already an issue for restoring tab states after shutdown or crash, and this should add no *new* issues to that - just requires the same security environment and filters as are already in use.
Does anyone have measurements on the difference that bug 776928 made in practice?
I did some coarse measurements:

I compared todays nightly
http://hg.mozilla.org/mozilla-central/rev/0c4fa25f637b
to the previous one.

Both times I started with 222 tabs and about:blank as start page
and waited some minutes until memory usage stabilized.

Previous version:
Resident: 352 MB (Memchaser 0.4)
Usage per Tab:
│  │   ├───0.37 MB (00.09%) -- top(about:blank, id=232)/active/window(about:blank)
│  │   │   ├──0.20 MB (00.05%) ++ js/compartment(about:blank)
│  │   │   ├──0.16 MB (00.04%) ++ layout
│  │   │   └──0.01 MB (00.00%) ++ dom

Todays nightly:
Resident: 312 MB (Memchaser 0.4)
Usage per Tab:
│   │  ├───0.20 MB (00.07%) -- top(about:blank, id=6)/active/window(about:blank)
│   │  │   ├──0.20 MB (00.06%) ++ js/compartment(about:blank)
│   │  │   └──0.01 MB (00.00%) ++ dom


According to about:memory data above savings should be
0.17 kb * 222 tabs = 37.74 MB

That's roughly the difference measured by Memchaser.
(In reply to Stefan Fleiter (:sfleiter) from comment #56)

So, bottom line is that fixing bug 776928 saves us 0.17 kb per unrestored tab on average which is a 46% reduction.
Keeping in mind this is, of course, one sample from one platform. It still sounds pretty good.
Someone please correct me if I got any of that wrong.
Thanks, Stefan!  I'm seeing a 0.17MB improvement in my own Linux 64-bit build, so I guess you're on a 64-bit platform as well.

The not-yet-loaded tabs are remarkably consistent now.  In the session I just started I have 10 tabs with identical stats:

│   ├─────215,096 B (00.12%) -- top(about:blank, id=18)/active/window(about:blank)
│   │     ├──208,952 B (00.11%) -- js/compartment(about:blank)
│   │     │  ├──129,952 B (00.07%) ── analysis-temporary
│   │     │  ├───61,440 B (00.03%) -- gc-heap
│   │     │  │   ├──31,336 B (00.02%) ── unused-gc-things
│   │     │  │   ├──20,584 B (00.01%) ── sundries
│   │     │  │   └───9,520 B (00.01%) ── shapes/tree
│   │     │  └───17,560 B (00.01%) ── other-sundries
│   │     └────6,144 B (00.00%) -- dom
│   │          ├──5,632 B (00.00%) ── other [2]
│   │          └────512 B (00.00%) ── element-nodes

(That's a lot better than it was when this bug was opened, which is great.)

So it's now almost entirely JS.  The majority of that is the dreaded analysis-temporary.  The remainder is mostly unused space/fragmentation due to our GC heap structure, which won't change in the short term but generation GC might fix it in the long term.  (Note that this fragmentation is sort of like external fragmentation, except that the space will be used usefully once the page is loaded
and more things get allocated for it.)  Hitting "minimize memory usage" makes "other-sundries" drop by about 4KB but nothing else changes notably.


I'm also wondering why the hidden window needs memory for layout data:

│   ├─────274,616 B (00.15%) -- top(resource://gre-resources/hiddenWindow.html, id=3)/active/window(resource://gre-resources/hiddenWindow.html)
│   │     ├──211,840 B (00.11%) -- layout
│   │     │  ├──154,928 B (00.08%) ── style-sets
│   │     │  ├───41,976 B (00.02%) ── pres-shell
│   │     │  ├────6,192 B (00.00%) ── rule-nodes
│   │     │  ├────4,160 B (00.00%) ── style-contexts
│   │     │  ├────2,776 B (00.00%) ── frames/sundries
│   │     │  ├────1,744 B (00.00%) ── pres-contexts
│   │     │  └───────64 B (00.00%) ── line-boxes
│   │     ├───55,704 B (00.03%) -- js/compartment(resource://gre-resources/hiddenWindow.html)
│   │     │   ├──45,056 B (00.02%) -- gc-heap
│   │     │   │  ├──29,056 B (00.02%) ── unused-gc-things
│   │     │   │  └──16,000 B (00.01%) ── sundries
│   │     │   └──10,648 B (00.01%) ── other-sundries
│   │     └────7,072 B (00.00%) -- dom
│   │          ├──5,808 B (00.00%) ── other [2]
│   │          ├────928 B (00.00%) ── element-nodes
│   │          └────336 B (00.00%) ── comment-nodes

Though maybe a constant 200KB improvement isn't worth much effort.
(In reply to Nicholas Nethercote [:njn] from comment #58)
> Thanks, Stefan!  I'm seeing a 0.17MB improvement in my own Linux 64-bit
> build, so I guess you're on a 64-bit platform as well.

Sorry, forgot to mention that. I am on Linux 64-bit, too.

This fix is a big improvement for heavy tab users, so thanks a lot to the team for the hard work!
> Hitting "minimize memory usage" makes "other-sundries" drop by about 4KB but nothing else changes 
> notably.

I thought analysis-temporary was supposed to go away on (some) gc's?  Maybe we're not GC'ing hard enough in about:memory?
> I thought analysis-temporary was supposed to go away on (some) gc's?  

That's the theory, but I've never seen it happen quite like that in practice -- I've never seen "analysis-temporary" disappear completely.
Flags: sec-review?(dveditz)
Uh, can you provide some logic for marking this 'sec-review?'?
> │   ├─────215,096 B (00.12%) -- top(about:blank,
> id=18)/active/window(about:blank)
> │   │     ├──208,952 B (00.11%) -- js/compartment(about:blank)
> │   │     │  ├──129,952 B (00.07%) ── analysis-temporary

I used my patch from bug 789398 to get more info.  All the type inference data is under "type-inference/type-pool", which measures JSCompartment::typeLifoAlloc, and which is the biggest bucket under "type-inference/".  So nothing terribly surprising there.
Flags: sec-review?(dveditz)
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #62)
> Uh, can you provide some logic for marking this 'sec-review?'?

putting sec-review flag back
because dveditz saw sufficient possible issues to want to do a code review
Flags: sec-review?(dveditz)
Well there's no code in this bug ...
Here's a current measurement of an unloaded tab in a Linux64 debug build

│  └─────223,944 B (00.28%) -- top(about:blank, id=22)/active/window(about:blank)
│        ├──217,000 B (00.28%) -- js/compartment(about:blank)
│        │  ├──143,360 B (00.18%) -- gc-heap
│        │  │  ├───62,520 B (00.08%) ── unused-gc-things
│        │  │  ├───37,088 B (00.05%) ── objects/function
│        │  │  ├───30,952 B (00.04%) -- shapes
│        │  │  │   ├──16,912 B (00.02%) ── base
│        │  │  │   └──14,040 B (00.02%) ── dict
│        │  │  └───12,800 B (00.02%) ── sundries
│        │  ├───32,768 B (00.04%) ── type-inference/type-pool
│        │  ├───21,056 B (00.03%) ── shapes-extra/compartment-tables
│        │  └───19,816 B (00.03%) ── other-sundries
│        ├────6,640 B (00.01%) -- dom
│        │    ├──5,616 B (00.01%) ── other [2]
│        │    ├────512 B (00.00%) ── element-nodes
│        │    └────512 B (00.00%) ── event-targets [2]
│        └──────304 B (00.00%) ── style-sheets

I was going to suggest that zones (bug 759585) would help by reducing the unused-gc-things, but each tab will probably end up with its own zone, so it probably won't help.

So at this point we just need to execute less JS code.  What JS code runs for an unrestored tab?
What constitutes JS code for something like this? There's the XUL elements that construct the tab, which is generated DOM via the XBL bindings and JS in tabbrowser.xml, and the HTML document in about:blank. Not sure what else.
I'd be surprised if any code is running in the about:blank document itself, it doesn't have any actual content after all.

I'm curious if those numbers reflect actual usage, or just some kind of preallocated empty space? But does that 217,000 really even matter much? AIUI it's a shared compartment, so it's a fixed cost that wouldn't vary with number of tabs. (200K would still be a nice win, but is a drop compared to the ~45MB this bug was filed about.)

As dietrich notes, there's the holistic cost for having a tab (like the XUL for its tab in the tabstrip), which is going to be hard to account for in detail.
(In reply to Justin Dolske [:Dolske] from comment #69)
> I'd be surprised if any code is running in the about:blank document itself,
> it doesn't have any actual content after all.

The JS numbers indicate that some code is at least present.  Not sure if it's running.

You're right that it is surprising.  That's the whole point of this bug.

 
> I'm curious if those numbers reflect actual usage, or just some kind of
> preallocated empty space? But does that 217,000 really even matter much?
> AIUI it's a shared compartment, so it's a fixed cost that wouldn't vary with
> number of tabs. (200K would still be a nice win, but is a drop compared to
> the ~45MB this bug was filed about.)

It's not shared.  Every not-yet-loaded tab costs 200+ KB.


> As dietrich notes, there's the holistic cost for having a tab (like the XUL
> for its tab in the tabstrip), which is going to be hard to account for in
> detail.

JS accounts for 217,000 out of 223,944 bytes per tab.  The other stuff is insignificant.
Oh. I was assuming

  217,000 B (00.28%) -- js/compartment(about:blank)

was a single instance just as with a JSM like

  216,112 B (00.02%) -- compartment([System Principal], resource:///modules/PageThumbs.jsm)
(In reply to Nicholas Nethercote [:njn] from comment #70)
> > As dietrich notes, there's the holistic cost for having a tab (like the XUL
> > for its tab in the tabstrip), which is going to be hard to account for in
> > detail.
> 
> JS accounts for 217,000 out of 223,944 bytes per tab.  The other stuff is
> insignificant.

I don't fully understand what exactly you're measuring, so maybe I'm just misunderstanding. What I was trying to say is that the other stuff is where much of the JS is - stuffed into XBL bindings like tabbrowser.xml. Is that the JS code you're referring to?
The JS code being reported there might actually be the content scripts, of which there is one per tab regardless of what document is loaded? To test, someone could comment out the loadFrameScript call in browser.js.

http://mxr.mozilla.org/mozilla-central/source/browser/base/content/content.js

I think it would be easy enough to eliminate that entirely now that we've dropped e10s support.
Flags: sec-review?(dveditz)
Let me break this down, because it's causing confusion.


> │  └─────223,944 B (00.28%) -- top(about:blank,
> id=22)/active/window(about:blank)
> │        ├──217,000 B (00.28%) -- js/compartment(about:blank)
> │        │  ├──143,360 B (00.18%) -- gc-heap
> │        │  │  ├───62,520 B (00.08%) ── unused-gc-things
> │        │  │  ├───37,088 B (00.05%) ── objects/function
> │        │  │  ├───30,952 B (00.04%) -- shapes
> │        │  │  │   ├──16,912 B (00.02%) ── base
> │        │  │  │   └──14,040 B (00.02%) ── dict
> │        │  │  └───12,800 B (00.02%) ── sundries
> │        │  ├───32,768 B (00.04%) ── type-inference/type-pool
> │        │  ├───21,056 B (00.03%) ── shapes-extra/compartment-tables
> │        │  └───19,816 B (00.03%) ── other-sundries
> │        ├────6,640 B (00.01%) -- dom
> │        │    ├──5,616 B (00.01%) ── other [2]
> │        │    ├────512 B (00.00%) ── element-nodes
> │        │    └────512 B (00.00%) ── event-targets [2]
> │        └──────304 B (00.00%) ── style-sheets


- The "top(...)" entry is for the top window object, which corresponds to the unloaded tab.  Everything that we know belongs to that tab sums to 223,944 bytes.  The nested entries below the first line show how that 223,944 bytes are broken down.

- The top window has a JS compartment (because a window object is a JS global object, and each global object has a compartment, thanks to compartment-per-global).  It accounts for 217,000 bytes, i.e. almost all of the 223,944 bytes that we know belong to the tab.  Those 217,000 bytes can be broken down further -- function objects, shapes, wasted GC space, etc.

- The other 6,944 bytes were dom stuff (6,640 bytes) and style-sheets (304 bytes).  Insignificant compared to the JS memory.

- Every not-yet-loaded tab has an entry in about:memory that is identical to this.  I.e. we know that closing each of these tabs will free up at least 223,944 bytes.  (And possibly more, because we don't measure everything.)

So, to summarize in very simple terms:

- Every not-yet-loaded tab takes up at least 200 KB.

- Almost all of that is due to the JS engine.

So, the obvious question is:  what's the JS code that is present/run for a not-yet-loaded tab, and can we get rid of it?  Hopefully it's content.js, as suggested by Gavin in comment 73.
There are a few frame scripts that are currently loaded through the message manager: the content.js one, a form history one, and an add-ons manager one (there's also a panorama one that is only lazily loaded so I assume it's not there).

The content.js one is the smallest one but all of those together could add up. To test the theory you could add an early return here: http://mxr.mozilla.org/mozilla-central/source/content/base/src/nsFrameMessageManager.cpp#276 and see how it changes those numbers
actually you probably need to deactivate this one: http://mxr.mozilla.org/mozilla-central/source/content/base/src/nsInProcessTabChildGlobal.cpp#333
or both

I can test this tomorrow if no one gets to it first
> To test, someone could comment out the loadFrameScript call in browser.js.

I tried that and it made no difference.

Felipe, I don't understand the changes you are describing well enough to test them myself.  If you could try them, that would be great.
So I disabled all the content scripts that are currently in use but it made a very small difference, 6k per tab. (both were measured after clicking "minimize memory usage")

Before:

│  ├─────227,272 B (00.26%) -- top(about:blank, id=12)/active/window(about:blank)
│  │     ├──219,160 B (00.25%) -- js/compartment(about:blank)
│  │     │  ├──143,360 B (00.16%) -- gc-heap
│  │     │  │  ├───62,496 B (00.07%) ── unused-gc-things
│  │     │  │  ├───37,088 B (00.04%) ── objects/function
│  │     │  │  ├───30,952 B (00.03%) -- shapes
│  │     │  │  │   ├──16,912 B (00.02%) ── base
│  │     │  │  │   └──14,040 B (00.02%) ── dict
│  │     │  │  └───12,824 B (00.01%) ── sundries
│  │     │  ├───32,768 B (00.04%) ── type-inference/type-pool
│  │     │  ├───21,976 B (00.02%) ── other-sundries
│  │     │  └───21,056 B (00.02%) ── shapes-extra/compartment-tables
│  │     ├────7,792 B (00.01%) -- dom
│  │     │    ├──6,688 B (00.01%) ── other [2]
│  │     │    ├────560 B (00.00%) ── element-nodes
│  │     │    └────544 B (00.00%) ── event-targets [2]
│  │     └──────320 B (00.00%) ── style-sheets

After:

│  ├─────221,128 B (00.23%) -- top(about:blank, id=14)/active/window(about:blank)
│  │     ├──213,016 B (00.22%) -- js/compartment(about:blank)
│  │     │  ├──139,264 B (00.15%) -- gc-heap
│  │     │  │  ├───58,808 B (00.06%) ── unused-gc-things
│  │     │  │  ├───37,088 B (00.04%) ── objects/function
│  │     │  │  ├───30,840 B (00.03%) -- shapes
│  │     │  │  │   ├──16,800 B (00.02%) ── base
│  │     │  │  │   └──14,040 B (00.01%) ── dict
│  │     │  │  └───12,528 B (00.01%) ── sundries
│  │     │  ├───32,768 B (00.03%) ── type-inference/type-pool
│  │     │  ├───21,976 B (00.02%) ── other-sundries
│  │     │  └───19,008 B (00.02%) ── shapes-extra/compartment-tables
│  │     ├────7,792 B (00.01%) -- dom
│  │     │    ├──6,688 B (00.01%) ── other [2]
│  │     │    ├────560 B (00.00%) ── element-nodes
│  │     │    └────544 B (00.00%) ── event-targets [2]
│  │     └──────320 B (00.00%) ── style-sheets
I've been analyzing the function objects present in these about:blank compartments, because I suspect they're the key.  I've been printing out extra info for each function object when iterating over the heap in about:memory's code.

The first thing I discovered is that all of these function objects are representing native functions, i.e. C++ functions, not JS functions.  So it doesn't appear that we're executing JS code in these tabs.

I also tried JSObject::dump() on each function object, which prints something like this for each object:

  object 0x7fec33776040 
  class 0x7fec5520a4d8 Function
  flags:
  proto <unnamed function (:0) at 0x7fec33759040>
  parent <Window object at 0x7fec33756060>
  properties:

The "parent" field was the most interesting.  Within a single not-yet-loaded-tab about:blank compartment, I got the following frequencies:

706 counts:
( 1)    399 (56.5%, 56.5%): parent <Window object at 0x7fec33756060>
( 2)    202 (28.6%, 85.1%): parent <XPC_WN_ModsAllowed_NoCall_Proto_JSClass object at 0x7fec33757190>
( 3)     36 ( 5.1%, 90.2%): parent <DocumentPrototype object at 0x7fec3375c160>
( 4)     17 ( 2.4%, 92.6%): parent <NodePrototype object at 0x7fec3375c100>
( 5)     13 ( 1.8%, 94.5%): parent <function Object at 0x7fec33759080>
( 6)     13 ( 1.8%, 96.3%): parent <Object at 0x7fec33758020>
( 7)     11 ( 1.6%, 97.9%): parent <XPC_WN_ModsAllowed_NoCall_Proto_JSClass object at 0x7fec33757070>
( 8)      6 ( 0.8%, 98.7%): parent <unnamed function (:0) at 0x7fec33759040>
( 9)      3 ( 0.4%, 99.2%): parent <EventTargetPrototype object at 0x7fec3375c0a0>
(10)      1 ( 0.1%, 99.3%): parent <Function object at 0x7fec3375c130>
(11)      1 ( 0.1%, 99.4%): parent <Function object at 0x7fec3375c0d0>
(12)      1 ( 0.1%, 99.6%): parent <TextDecoderPrototype object at 0x7fec3375c040>
(13)      1 ( 0.1%, 99.7%): parent <TextEncoderPrototype object at 0x7fec3375c070>
(14)      1 ( 0.1%, 99.9%): parent <Function object at 0x7fec3375c190>
(15)      1 ( 0.1%,100.0%): parent <function XPCNativeWrapper at 0x7fec33759a00>

Having "Window" as the most common isn't surprising, but what is the XPC_WN_ModsAllowed_NoCall_Proto_JSClass object?
FWIW, here's the full measurements for a not-yet-restored tab with the "sundries" JS elements expanded fully.

│  └─────227,272 B (00.25%) -- top(about:blank, id=22)/active/window(about:blank)
│        ├──219,160 B (00.24%) -- js/compartment(about:blank)
│        │  ├──143,360 B (00.16%) -- gc-heap
│        │  │  ├───62,496 B (00.07%) ── unused-gc-things
│        │  │  ├───38,632 B (00.04%) -- shapes
│        │  │  │   ├──16,912 B (00.02%) ── base
│        │  │  │   ├──14,040 B (00.02%) ── dict
│        │  │  │   └───7,680 B (00.01%) -- tree
│        │  │  │       ├──6,400 B (00.01%) ── global-parented
│        │  │  │       └──1,280 B (00.00%) ── non-global-parented
│        │  │  ├───38,464 B (00.04%) -- objects
│        │  │  │   ├──37,088 B (00.04%) ── function
│        │  │  │   ├───1,280 B (00.00%) ── ordinary
│        │  │  │   ├──────96 B (00.00%) ── cross-compartment-wrapper
│        │  │  │   ├───────0 B (00.00%) ── dense-array
│        │  │  │   └───────0 B (00.00%) ── slow-array
│        │  │  ├────2,176 B (00.00%) ── arena-admin
│        │  │  ├────1,400 B (00.00%) ── type-objects
│        │  │  ├──────192 B (00.00%) ── scripts
│        │  │  ├────────0 B (00.00%) ── ion-codes
│        │  │  └────────0 B (00.00%) -- strings
│        │  │           ├──0 B (00.00%) ── normal
│        │  │           └──0 B (00.00%) ── short
│        │  ├───32,768 B (00.04%) -- type-inference
│        │  │   ├──32,768 B (00.04%) ── type-pool
│        │  │   ├───────0 B (00.00%) ── allocation-site-tables
│        │  │   ├───────0 B (00.00%) ── analysis-pool
│        │  │   ├───────0 B (00.00%) ── array-type-tables
│        │  │   ├───────0 B (00.00%) ── object-type-tables
│        │  │   ├───────0 B (00.00%) ── pending-arrays
│        │  │   ├───────0 B (00.00%) ── type-objects
│        │  │   ├───────0 B (00.00%) ── type-results
│        │  │   └───────0 B (00.00%) ── type-scripts
│        │  ├───29,440 B (00.03%) -- shapes-extra
│        │  │   ├──21,056 B (00.02%) ── compartment-tables
│        │  │   ├───6,496 B (00.01%) ── dict-tables
│        │  │   ├───1,600 B (00.00%) ── tree-tables
│        │  │   └─────288 B (00.00%) ── tree-shape-kids
│        │  ├────5,904 B (00.01%) -- objects-extra
│        │  │    ├──5,696 B (00.01%) ── slots
│        │  │    ├────208 B (00.00%) ── regexp-statics
│        │  │    ├──────0 B (00.00%) ── arguments-data
│        │  │    ├──────0 B (00.00%) ── ctypes-data
│        │  │    ├──────0 B (00.00%) ── elements
│        │  │    └──────0 B (00.00%) ── property-iterator-data
│        │  ├────4,096 B (00.00%) ── compartment-object
│        │  ├────2,048 B (00.00%) ── cross-compartment-wrapper-table
│        │  ├────1,024 B (00.00%) ── regexp-compartment
│        │  ├──────512 B (00.00%) ── debuggees-set
│        │  ├────────8 B (00.00%) ── script-data
│        │  ├────────0 B (00.00%) ── ion-data
│        │  ├────────0 B (00.00%) ── jaeger-data
│        │  └────────0 B (00.00%) ── string-chars/non-huge
│        ├────7,792 B (00.01%) -- dom
│        │    ├──6,688 B (00.01%) ── other [2]
│        │    ├────560 B (00.00%) ── element-nodes
│        │    ├────544 B (00.00%) ── event-targets [2]
│        │    └──────0 B (00.00%) ── orphan-nodes
│        └──────320 B (00.00%) ── style-sheets
For each of the native function objects, I tried printing the atom_ field, which is a "name for diagnostics and decompiling", according to the comments.  Here are the top 20 occurrences.

579 counts:
( 1)    392 (67.7%, 67.7%): (NIL)
( 2)      3 ( 0.5%, 68.2%): "addEventListener"
( 3)      3 ( 0.5%, 68.7%): "toString"
( 4)      3 ( 0.5%, 69.3%): "dispatchEvent"
( 5)      3 ( 0.5%, 69.8%): "removeEventListener"
( 6)      2 ( 0.3%, 70.1%): "hasChildNodes"
( 7)      2 ( 0.3%, 70.5%): "loadBindingDocument"
( 8)      2 ( 0.3%, 70.8%): "createTreeWalker"
( 9)      2 ( 0.3%, 71.2%): "getSelection"
(10)      2 ( 0.3%, 71.5%): "getUserData"
(11)      2 ( 0.3%, 71.8%): "toString"
(12)      2 ( 0.3%, 72.2%): "elementFromPoint"
(13)      2 ( 0.3%, 72.5%): "querySelectorAll"
(14)      2 ( 0.3%, 72.9%): "createTextNode"
(15)      2 ( 0.3%, 73.2%): "querySelector"
(16)      2 ( 0.3%, 73.6%): "insertBefore"
(17)      2 ( 0.3%, 73.9%): "createNSResolver"
(18)      2 ( 0.3%, 74.3%): "createComment"
(19)      2 ( 0.3%, 74.6%): "replaceChild"
(20)      2 ( 0.3%, 75.0%): "contains"

Two thirds lack an atom, which isn't interesting.  But among those that do, many of the atoms occur 2 or 3 times.  Some of the ones not shown on this list sound specialized, e.g. "caretPositionFromPoint", "isDefaultNamespace".  A lot of them appear to be IDL methods.  I'm not sure how to interpret this.
cc:ing peterv and bz as no doubt at least some of the functions present in about:blank are from the DOM bindings...
Depends on: 839973
Nit - when CCing people with a specific question, please indicate that question such that it appears in the same bugmail as the CC, possibly using needInfo.

(In reply to Nicholas Nethercote [:njn] from comment #79)
> Having "Window" as the most common isn't surprising, but what is the
> XPC_WN_ModsAllowed_NoCall_Proto_JSClass object?

That's a JSClass of certain XPConnect prototype objects. In this case, it's probably the prototype for |Window|, and it holds function objects for all of the XPIDL methods discoverable by Window's nsIClassInfo implementation.

We probably don't want to mess with that stuff in the XPConnect case since it's on its way out, anyway. But it's worth discussing whether we can make that lazy somehow for the about:blank case in the new bindings.
> ( 1)    392 (67.7%, 67.7%): (NIL)

Would the "atom" thing be empty for the functions generated for JSNative property getters/setters?

> But among those that do, many of the atoms occur 2 or 3 times.

An about:blank document contains the following DOM nodes: an HTMLDocument, an HTMLHtmlElement, an HTMLHeadElement, and an HTMLBodyElement.  It also has a Window.  I assume the Window and HTMLDocument have had their JS objects instantiated and hence their prototype chains (probably a safe assumption.  I'll also assume none of the elements have JS objects instantiated yet.

Since HTMLDocument and Window are not on WebIDL bindings yet, I would expect all the EventTarget members to live on EventTarget.prototype, HTMLDocument.prototype, and Window.prototype.  That would be the three copies of add/removeEventListener and dispatchEvent.  Everything on the Document interface I'd expect to see on Document.prototype and HTMLDocument.prototype (and everything on Node would be on both NOde.prototype and HTMLDocument.prototype), which explains all the things there are "2" of in comment 81.

Looking at comment 79, it looks like WebIDL methods (which go through JS_DefineFunctions) will be parented to the proto they live on.  WebIDL property getters/setters (which go through JS_DefineProperties) will be parented to the global (so in this case the Window).  Not sure where XPConnect things are parented, offhand.
Depends on: 840809
After landing bug 839973 and bug 840809, we're now down to this.  (Note that I had to change SUNDRIES_THRESHOLD's value to 1 to get this level of detail, with any aggregation into "sundries" entries.)

│   └─────196,200 B (00.19%) -- top(about:blank, id=18)/active/window(about:blank)
│         ├──189,256 B (00.18%) -- js/compartment(about:blank)
│         │  ├──143,360 B (00.14%) -- gc-heap
│         │  │  ├───62,496 B (00.06%) ── unused-gc-things
│         │  │  ├───38,632 B (00.04%) -- shapes
│         │  │  │   ├──16,912 B (00.02%) ── base
│         │  │  │   ├──14,040 B (00.01%) ── dict
│         │  │  │   └───7,680 B (00.01%) -- tree
│         │  │  │       ├──6,400 B (00.01%) ── global-parented
│         │  │  │       └──1,280 B (00.00%) ── non-global-parented
│         │  │  ├───38,464 B (00.04%) -- objects
│         │  │  │   ├──37,088 B (00.04%) ── function
│         │  │  │   ├───1,280 B (00.00%) ── ordinary
│         │  │  │   └──────96 B (00.00%) ── cross-compartment-wrapper
│         │  │  ├────2,176 B (00.00%) ── arena-admin
│         │  │  ├────1,400 B (00.00%) ── type-objects
│         │  │  └──────192 B (00.00%) ── scripts
│         │  ├───29,344 B (00.03%) -- shapes-extra
│         │  │   ├──21,056 B (00.02%) ── compartment-tables
│         │  │   ├───6,496 B (00.01%) ── dict-tables
│         │  │   ├───1,600 B (00.00%) ── tree-tables
│         │  │   └─────192 B (00.00%) ── tree-shape-kids
│         │  ├────8,192 B (00.01%) ── type-inference/type-pool
│         │  ├────5,888 B (00.01%) -- objects-extra
│         │  │    ├──5,696 B (00.01%) ── slots
│         │  │    └────192 B (00.00%) ── regexp-statics
│         │  ├────2,048 B (00.00%) ── compartment-object
│         │  ├──────192 B (00.00%) ── regexp-compartment
│         │  ├──────160 B (00.00%) ── cross-compartment-wrapper-table
│         │  ├───────64 B (00.00%) ── debuggees-set
│         │  └────────8 B (00.00%) ── script-data
│         ├────6,640 B (00.01%) -- dom
│         │    ├──5,616 B (00.01%) ── other [2]
│         │    ├────512 B (00.00%) ── element-nodes
│         │    └────512 B (00.00%) ── event-targets [2]
│         └──────304 B (00.00%) ── style-sheets

BTW, in comment 80 I measured a debug build, which makes some things (e.g. "compartment-object") slightly larger.  This measurement is with an optimized build, which is more realistic.
I'm downgrading this to MemShrink:P3, for the following reasons.

- We've reduced the overhead per not-yet-loaded tab by a factor of 5 (1 MiB --> 200 KiB) since the bug was opened.

- Further reductions are unlikely to have any noticeable effect (excluding the satisfaction of people who are measuring it).  This is only significant for people with many (e.g. 100s) of unopened tabs, and such people are likely to be on machines with lots of RAM, and the memory used by an unloaded tab is dwarfed by that of a loaded tab.
Whiteboard: [MemShrink:P2][Snappy:P2] → [MemShrink:P3][Snappy:P2]
(In reply to Nicholas Nethercote [:njn] from comment #86)
> - Further reductions are unlikely to have any noticeable effect (excluding
> the satisfaction of people who are measuring it).  This is only significant
> for people with many (e.g. 100s) of unopened tabs, and such people are
> likely to be on machines with lots of RAM, and the memory used by an
> unloaded tab is dwarfed by that of a loaded tab.

I wonder, however, how corrolated it is to startup time. Startup time with 100s tabs is abysmal. And surely, going through the process of doing the stuff that allocates those 200k participates.
> I wonder, however, how corrolated it is to startup time. Startup time with
> 100s tabs is abysmal. And surely, going through the process of doing the
> stuff that allocates those 200k participates.

Perhaps.  That's not relevant for MemShrink prioritization, though.
> comment #75:
> There are a few frame scripts that are currently loaded through the message
> manager: the content.js one, a form history one, and an add-ons manager one
> (there's also a panorama one that is only lazily loaded so I assume it's not
> there).

> comment #78
> So I disabled all the content scripts that are currently in use but it made
> a very small difference, 6k per tab.

It appears that these frame scripts are accounted for separately -- under explicit/js-non-window/compartments/non-window-global/compartment([System Principal], inProcessTabChildGlobal?ownedBy=chrome://browser/content/browser.xul)

They take up another 170+ KB per (unloaded) tab.

I filed bug 844661 about that.
Depends on: 844661
> They take up another 170+ KB per (unloaded) tab.
> 
> I filed bug 844661 about that.

Nice catch!

---

In reference to comment 86:  one problem with this bug is that is lacks a clear success criterion.  I.e. when will this bug be closed?  We're already fairly deep into diminishing returns territory (though bug 844661 looks to have a very high cost:benefit ratio, which is nice).
> In reference to comment 86:  one problem with this bug is that is lacks a clear success criterion.  
> I.e. when will this bug be closed?

I think a programmer of average skill who is unfamiliar with the internals of Gecko would probably assert that a not-yet-restored tab shouldn't take up significantly more memory than required to store its URL and <title> and draw its UI.

I'm not sure that's the right criterion, but just throwing it out there.

Maybe we should say: We should close this bug iff we have no more reasonable ideas for reducing memory usage here.  I don't think at this point the memory usage is so worrisome that we need to spend a lot of time coming up with additional ways to reduce it.
> I think a programmer of average skill who is unfamiliar with the internals
> of Gecko would probably assert that a not-yet-restored tab shouldn't take up
> significantly more memory than required to store its URL and <title> and
> draw its UI.

I'm even more pessimistic;  I figure as long as it's more than zero someone will complain.  I'm fine with closing this bug.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
(In reply to Nicholas Nethercote [:njn] from comment #92)
> I'm even more pessimistic;  I figure as long as it's more than zero someone
> will complain.  I'm fine with closing this bug.

Could you shortly answer why a tab eats up ~200KB instead of the size of memory required to store tab's URL, favicon and <title> and it's position on tab-bar (+ all it's DOM attributes, like [pinned])?

I'm just curious.
(In reply to Firefox Portable user from comment #93)
> (In reply to Nicholas Nethercote [:njn] from comment #92)
> > I'm even more pessimistic;  I figure as long as it's more than zero someone
> > will complain.  I'm fine with closing this bug.
> 
> Could you shortly answer why a tab eats up ~200KB instead of the size of
> memory required to store tab's URL, favicon and <title> and it's position on
> tab-bar (+ all it's DOM attributes, like [pinned])?
> 
> I'm just curious.

I agree.  This seems a bit disingenuous to close a bug because 'people will complain anyway' or it's 'good enough'.  Why has no one been able to precisely describing the contents of the memory used by an unloaded tab?  Surely, if we don't plan on decreasing memory usage further, this should at least be KNOWN for posterity.
"It's good enough" is a very common reason for closing a bug.  We have limited resources for working on Firefox, and we're well into diminishing returns territory for this bug.  If anyone wants to keep looking into this they're welcome, but as an example, my efforts are better spent elsewhere at this point.

> Surely, if we don't plan on decreasing memory usage further, this should at
> least be KNOWN for posterity.

I closed the bug.  Consider that your signal.
(In reply to Firefox Portable user from comment #93)
> (In reply to Nicholas Nethercote [:njn] from comment #92)
> > I'm even more pessimistic;  I figure as long as it's more than zero someone
> > will complain.  I'm fine with closing this bug.
> 
> Could you shortly answer why a tab eats up ~200KB instead of the size of
> memory required to store tab's URL, favicon and <title> and it's position on
> tab-bar (+ all it's DOM attributes, like [pinned])?
> 
> I'm just curious.

Because we create a JS "compartment" to store any script we run that in that tab, and the size of a compartment for about:blank is about 200 KB.  If you look at comment 85 you can see that the compartment is 96.5% of the total memory usage.

(In reply to tyaremco from comment #94)
> I agree.  This seems a bit disingenuous to close a bug because 'people will
> complain anyway' or it's 'good enough'.  Why has no one been able to
> precisely describing the contents of the memory used by an unloaded tab? 
> Surely, if we don't plan on decreasing memory usage further, this should at
> least be KNOWN for posterity.

There's a byte by byte accounting in comment 85.  Comments 10, 12, 16-21 explain why we're not putting significant effort into getting that last 200 KB.
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #96)
> Because we create a JS "compartment" to store any script we run that in that
> tab, and the size of a compartment for about:blank is about 200 KB.  If you
> look at comment 85 you can see that the compartment is 96.5% of the total
> memory usage.

Am I right to I assume this 200KB is nearly identical amongst all unloaded tabs?  Can you please explain why _each_ tab needs this 200KB as apposed to them simply sharing (some of) this JS "compartment" until they become loaded tabs?  I have no idea if this is practical or if this is related to 'lazy loading'; I just want to understand the situation better.
(In reply to tyaremco from comment #97)
> (In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #96)
> > Because we create a JS "compartment" to store any script we run that in that
> > tab, and the size of a compartment for about:blank is about 200 KB.  If you
> > look at comment 85 you can see that the compartment is 96.5% of the total
> > memory usage.
> 
> Am I right to I assume this 200KB is nearly identical amongst all unloaded
> tabs?

Kind of.  The lazy loaded tabs are just tabs set to about:blank instead of whatever was loaded before/will be loaded again.  As far as the browser is concerned two tabs set to about:blank are as separate as two tabs set to facebook.com are.  Compartment boundaries are security boundaries, and it's possible for web pages to do things like include an about:blank iframe and manipulate it from script, so we can't just dump all the about:blank pages in the same compartment because there would be no security boundary between an about:blank page facebook.com can touch and an about:blank page stealmyprivateinformation.shadysite can touch.

In practice with about:blank a lot of the space is identical because it's unused.  We allocate separate chunks of memory within a compartment for different kinds of garbage-collected things, so if you have just a handful of each type of things (such as about:blank is likely to) you end up with a lot of unused space.  We have work underway elsewhere to reduce unused space in compartments by sharing these chunks between compartments.  Once that stuff exists it's possible we could share chunks between lazy-loaded about:blank tabs, but that's well in the future and we have to make sure that it's safe for compartments to share chunks with pages from totally separate sites and it's not clear that it will be.
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #96)
> (In reply to Firefox Portable user from comment #93)
> > Could you shortly answer why a tab eats up ~200KB instead of the size of
> > memory required to store tab's URL, favicon and <title> and it's position on
> > tab-bar (+ all it's DOM attributes, like [pinned])?
> 
> Because we create a JS "compartment" to store any script we run that in that
> tab, and the size of a compartment for about:blank is about 200 KB.  If you
> look at comment 85 you can see that the compartment is 96.5% of the total
> memory usage.
> 
> There's a byte by byte accounting in comment 85.  Comments 10, 12, 16-21
> explain why we're not putting significant effort into getting that last 200
> KB.

Either i am completely stupid or i am just totally unable to replicate that unloaded tabs consume just 200KB... all i am seeing is ~666KB.

i have attached the complete testcase with all 900 tabs over 11 windows. additionally the about:memory output for all 900 tabs with 17 loaded (11 shown(1 for each window) + 6 pinned) and just the 17 loaded in the same configuration. both in a fresh profile.

So what do i do wrong? thanks.
aw. forgot build info: ffx_22a1_osx_x86_64
You're ignoring fragmentation.

With the 900 tabs you have:

475.63 MB (100.0%) -- js-main-runtime-gc-heap-committed
├──291.10 MB (61.20%) -- used

and

189.33 MB (100.0%) -- window-objects

and

├────126.57 MB (12.11%) ── heap-unclassified

291.1 + 189.33 + 126.57 = 606.90

with 17 you have:

135.93 MB (100.0%) -- js-main-runtime-gc-heap-committed
├───96.47 MB (70.97%) -- used

109.62 MB (100.0%) -- window-objects

├───62.77 MB (13.51%) ── heap-unclassified

109.62 + 96.47 + 62.77 = 269.06

(606.90-269.06)/(900-17) = 382 KB/tab

The size of that above 200 KB/tab is mostly bug 844661 which accounts for ~170 KB/tab.
If sharing about:blank is unsafe, maybe a new special URL should be created for unloaded tabs, that we could then forbid from being embedded in a webpage?
(In reply to Eric Williams from comment #102)
> If sharing about:blank is unsafe, maybe a new special URL should be created
> for unloaded tabs, that we could then forbid from being embedded in a
> webpage?

That might be possible, but it might also mean creating a lot of special cases in our security infrastructure for this one case.
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #103)
> (In reply to Eric Williams from comment #102)
> > If sharing about:blank is unsafe, maybe a new special URL should be created
> > for unloaded tabs, that we could then forbid from being embedded in a
> > webpage?
> 
> That might be possible, but it might also mean creating a lot of special
> cases in our security infrastructure for this one case.

Actually, this is right along the lines that I was about to suggest. 
If there are zero expectations of interactions on unrestored tabs (other than actually being restored) then we make them a special case/state.
The assumption that every docshell starts with an about:blank content viewer is deeply embedded in docshell, dom/base, and appshell. Changing that would be an insane amount of engineering effort.

Maybe this could be fixed on the frontend side (not creating <browser> elements for not-yet-loaded iframes), but maybe they have similar assumptions about tabs that we have for docshells.
(In reply to Bobby Holley (:bholley) from comment #105)
> Maybe this could be fixed on the frontend side (not creating <browser>
> elements for not-yet-loaded iframes), but maybe they have similar
> assumptions about tabs that we have for docshells.

And comments 16-21 explain that they have exactly those problems ...

... which is why we decided to work on other bugs.  As comment 90 says, we can get much higher return for the effort elsewhere.
Is there a bug on making a JS compartment that has not ever run any JS take up less than 190KB?
> If there are zero expectations of interactions on unrestored tabs 

There are, if "interactions" includes extension code.
> Is there a bug on making a JS compartment that has not ever run any JS take
> up less than 190KB?

Zones helped a lot with that.  In my current about:memory on 64-bit Linux I have numerous compartments that are only 17,120 bytes.
njn: do you mean this is not normal?

│  ├─────160,408 B (00.24%) -- top(about:blank, id=14)
│  │     ├──114,320 B (00.17%) -- active/window(about:blank)
│  │     │  ├──107,296 B (00.16%) -- js-compartment(about:blank)
│  │     │  │  ├───76,792 B (00.12%) -- gc-heap
│  │     │  │  │   ├──41,456 B (00.06%) -- shapes
│  │     │  │  │   │  ├──21,080 B (00.03%) ── tree/global-parented
│  │     │  │  │   │  └──20,376 B (00.03%) ── base
│  │     │  │  │   ├──28,288 B (00.04%) ── objects/function
│  │     │  │  │   └───7,048 B (00.01%) ── sundries
│  │     │  │  ├───16,680 B (00.03%) ── other-sundries
│  │     │  │  └───13,824 B (00.02%) ── shapes-extra/compartment-tables
│  │     │  ├────6,720 B (00.01%) -- dom
│  │     │  │    ├──5,648 B (00.01%) ── other [2]
│  │     │  │    ├────560 B (00.00%) ── element-nodes
│  │     │  │    └────512 B (00.00%) ── event-targets [2]
│  │     │  └──────304 B (00.00%) ── style-sheets
│  │     └───46,088 B (00.07%) -- js-zone(0x116797000)
│  │         ├──37,896 B (00.06%) -- gc-heap
│  │         │  ├──32,128 B (00.05%) ── unused-gc-things
│  │         │  └───5,768 B (00.01%) ── sundries
│  │         └───8,192 B (00.01%) ── type-pool
│  ├─────160,408 B (00.24%) -- top(about:blank, id=16)
│  │     ├──114,320 B (00.17%) -- active/window(about:blank)
│  │     │  ├──107,296 B (00.16%) -- js-compartment(about:blank)
│  │     │  │  ├───76,792 B (00.12%) -- gc-heap
│  │     │  │  │   ├──41,456 B (00.06%) -- shapes
│  │     │  │  │   │  ├──21,080 B (00.03%) ── tree/global-parented
│  │     │  │  │   │  └──20,376 B (00.03%) ── base
│  │     │  │  │   ├──28,288 B (00.04%) ── objects/function
│  │     │  │  │   └───7,048 B (00.01%) ── sundries
│  │     │  │  ├───16,680 B (00.03%) ── other-sundries
│  │     │  │  └───13,824 B (00.02%) ── shapes-extra/compartment-tables
│  │     │  ├────6,720 B (00.01%) -- dom
│  │     │  │    ├──5,648 B (00.01%) ── other [2]
│  │     │  │    ├────560 B (00.00%) ── element-nodes
│  │     │  │    └────512 B (00.00%) ── event-targets [2]
│  │     │  └──────304 B (00.00%) ── style-sheets
│  │     └───46,088 B (00.07%) -- js-zone(0x11679b800)
│  │         ├──37,896 B (00.06%) -- gc-heap
│  │         │  ├──32,128 B (00.05%) ── unused-gc-things
│  │         │  └───5,768 B (00.01%) ── sundries
│  │         └───8,192 B (00.01%) ── type-pool

This is http://hg.mozilla.org/mozilla-central/rev/f691f7abfe33, 64-bit, OS X 10.7
> njn: do you mean this is not normal?
> 
> │  ├─────160,408 B (00.24%) -- top(about:blank, id=14)
> │  │     ├──114,320 B (00.17%) -- active/window(about:blank)
> │  │     │  ├──107,296 B (00.16%) -- js-compartment(about:blank)

That's quite normal for compartments that belong to blank tabs.  However, if you look at the compartments in the "js-non-window" sub-tree you should see some compartments with smaller sizes.
Sorry if I'm missing something, I thought the issues remaining here were:
1) 196 KB (now down to 160 KiB, yay!) per explicit/window-objects/top(about:blank...) - comment 85.

2) 170 KB per tab under in explicit/js-non-window/.../compartment(...inProcessTabChildGlobal...) - bug 844661.

#2 still takes around 170 KB per tab for me:
│  │  │  ├───3,581,512 B (05.17%) -- compartment([System Principal], inProcessTabChildGlobal?ownedBy=chrome://browser/content/browser.xul)
│  │  │  │   ├──2,155,848 B (03.11%) -- gc-heap
│  │  │  │   │  ├────939,264 B (01.36%) -- objects
│  │  │  │   │  │    ├──782,688 B (01.13%) ── function [20]
│  │  │  │   │  │    └──156,576 B (00.23%) ── ordinary [19]
│  │  │  │   │  ├────899,264 B (01.30%) -- shapes
│  │  │  │   │  │    ├──556,760 B (00.80%) ── tree/global-parented [20]
│  │  │  │   │  │    └──342,504 B (00.49%) ── base [20]
│  │  │  │   │  └────317,320 B (00.46%) ── sundries [20]
│  │  │  │   ├────655,360 B (00.95%) ── type-inference/analysis-pool [20]
│  │  │  │   ├────368,640 B (00.53%) ── shapes-extra/compartment-tables [20]
│  │  │  │   ├────207,360 B (00.30%) ── objects-extra/slots [20]
│  │  │  │   └────194,304 B (00.28%) ── other-sundries [20]
The only remaining issue is the inProcessTabChildGlobal, and that's covered by bug 844661.  No need to keep commenting here.
Blocks: 1054660
Summary: Not-yet-restored empty tabs take more memory than they should → [meta] Not-yet-restored empty tabs take more memory than they should
You need to log in before you can comment on or make changes to this bug.