1427594 - Memory consumption on idle

Status: UNCONFIRMED

(Firefox :: New Tab Page, defect, P3)

Description

[alexander]

Attachment: memory_ff.png

User Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:57.0) Gecko/20100101 Firefox/57.0
Build ID: 20171208103543

Steps to reproduce:

Good day.

I did simple memory consumption measurement for 3 days in Firefox and Cliqz (FF based) browsers in "idle" work, with only one empty tab opened (about:blank). OS - Windows 8.1. Firefox 64 bit without any additional extensions or plugins - just pure installation with new profile.

You can check results on images or look into raw data (format - timestamp, process id, memory in bytes):
Firefox: https://drive.google.com/file/d/1jahA8urKCuTw8tDGWQCdBfySuue6TqxA/view?usp=sharing
Cliqz: https://drive.google.com/file/d/1apPHwdtKm_qMLpqc1APpIuZLR1hxh3ji/view?usp=sharing

Main difference between browsers - there are no "internal" FF extensions in Cliqz browser, only our own Cliqz extensions. For me it's looks like memory leaking in one of internal FF extension, like activity-stream.

I'm not sure, is it important or not, but want to share my finding with FF team.


Actual results:

Looks like memory leak happened somewhere in FF extension.


Expected results:

Not consuming memory in idle work?

Comment 1

[alexander]

Attachment: memory_cliqz.png

Comment 2

[Marius Coman [:mcoman], Ecosystem QA]

Hi Alexander,

Could you please repeat your experiment, but on a shorter period of time (~3 or 4 hours) and provide two memory reports (from the "about:memory" page), one from the beginning of the experiment and one from the end of it?

Comment 3

[alexander]

(In reply to Marius Coman [:cmarius] from comment #2)
> Hi Alexander,
> 
> Could you please repeat your experiment, but on a shorter period of time (~3
> or 4 hours) and provide two memory reports (from the "about:memory" page),
> one from the beginning of the experiment and one from the end of it?

Hi, Marlus. Of course, I will do on next week

Comment 4

[alexander]

(In reply to Marius Coman [:cmarius] from comment #2)
> Hi Alexander,
> 
> Could you please repeat your experiment, but on a shorter period of time (~3
> or 4 hours) and provide two memory reports (from the "about:memory" page),
> one from the beginning of the experiment and one from the end of it?

Hi, Marlus. Please find logs here: https://drive.google.com/drive/folders/1N7QGdV0l6pTHp77nzUjGm7I7zLxZ9FQG?usp=sharing

Comment 5

[Marius Coman [:mcoman], Ecosystem QA]

Eric, could you please take a look over the memory reports and see if something stands out?

Comment 6

[Eric Rahm [:erahm]]

The diff shows ~40MiB increase in the main process, we actually see a reduction in the content processes. 

Mostly this is showing up as strings which I believe are primarily shavar [1] related (tracking protection / phishing updates etc). In the small string category there's a ton of "versioncheck-bg.addons.mozilla.org" which I'm going to guess is related to checking for new versions of add-ons. We might want to ping them about why those strings are sticking around. The larger smaller items appear to be base64 encoded images (probably for data urls); I'm not sure what the source is as they're truncated but it's quite possible they're activity stream related. There's also some crash reporting items.

Next largest we see a ~14MiB chunk in heap-unclassified, this could be leaks but odds are it's just tables we're not accounting for. A DMD run [2] could help identify the sources.

After that there's 2.77 MB for activity streams, it's just overhead for all the jsm's it has loaded.

So:
- updates for phishing
- updates for addons
- activity stream stuff

Basically things we expect the browser to do while idle, but we might want to loop people in to determine if these are expected.

And some better details:

> 32.99 MB (100.0%) -- explicit
> ├──27.49 MB (83.33%) -- js-non-window
> │  ├──20.82 MB (63.13%) -- zones/zone(0xNNN)
> │  │  ├──11.84 MB (35.88%) -- strings
> │  │  │  ├───5.35 MB (16.22%) ++ (223 tiny)
> │  │  │  ├───3.16 MB (09.59%) ++ string(<non-notable strings>)
> │  │  │  ├───0.86 MB (02.61%) ++ string(length=43676, copies=20, "iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAYAAABccqhmAAAgAElEQVR4nOx9d5hV1dX+FlE00S+/LwGGmblze29TAEWwILZYEo2KKKifMYoliFGj0djQqGgiYsOa2CuKCmOLvSAdRQRBmrRhmLn1nFtO3+/vj1PmznDvzKDAvcBZz/M+MyjM3L3PWu9699pr70OIaaaZZppppplmmmmmmWaaaaaZZpppFW0A9tHQB8C+BegDYJ8d9Dv0n9234GfvsJ9vmmmm9WBdg7wX/6QPIeQAQsj/EEJ+7XK5Bo4YMaLm+OOPrzv99NNtF1xwgf2iiy5yXHjhhc7zzjvPcfbZZ9tHjx5tHTVqVO3gwYOrDz744P6EkP9HCPklIWT/Xny+PjuaeEwzba+0LsFeKpgOCofDVRMnTvS8/PLLR3z66adjlixZMmH16tW3btmy5ZH29vZXE4nEu6lUanYmk/kmm83+wHHcBp7nW0VRTEiSlJZlmZUkKSNJEitJUlIUxXaO4zbn8/m1mUxmGcuy81Op1MexWOyttra2Z9avX//PZcuWXTdv3rw/vvfeeydNmTKl8dRTT60jhPyaELJfibH0NA7TTNu7rSDg+xT53weOGDGiZsqUKY2ff/75WcuWLftbS0vLk4lE4sNsNvudKIpxRVEk7GKjlFJJkvL5fH49wzBz29vbX1+7du3dixcvvuS11147Zvz48S6iEsM+RcZqKgXT9l7rJuD3qaqqGjh16tSGL7744ty1a9feHYvFmjOZzPeSJGV6EZcKAIlSKmlfZQAyAJlSqmj/XwGgUEopAKrFMi38f9r/V7R/Jxf+TO3n0W4/hKLIHMe1pFKpr1pbW59asmTJ1W+88cZx5513noMQ8osSc2GSgWl7rpVydJfLNfDJJ58ctmjRostbWlqeYhhmoSRJ6W7iSw/IwqCmRYCeArWX1vXndYVS+JlK/U5FUeR8Pv9jLBZ7d9WqVXf897//PW3ixIkeQsiBXeaplBIyzbTdy1A86A+47rrrfJ9" (truncated))
> │  │  │  ├───0.82 MB (02.49%) ++ string(length=284991, copies=3, "allow-flashallow-digest256;a:1490633678/nbase-track-digest256;a:1512580265/nblock-flash-digest256;a:1496263270/nblock-flashsubdoc-digest256;a:1512160865/nexcept-flash-digest256;a:1494877265/nexcept-flashallow-digest256;a:1490633678/nexcept-flashinfobar-digest256;a:1497379266/nexcept-flashsubdoc-digest256;a:1490633678/ngoog-badbinurl-shavar;a:138620-140037:s:142680,142706,142713,142716,142770,142776,142783,142785,142789-142791,142796-142797,142802-142803,142806-142807,142811,142813,142815-142816,142822,142830-142831,142833,142835-142836,142841-142842,142845-142846,142854,142859,142868,142872,142875,142885-142886,142889-142891,142896-142897,142903-142904,142906-142907,142912-142916,142919-142921,142937,142950,142952-142953,142955-142956,142961,142972,142974,142977,142982-142983,142986,142990,142993-142994,142998,143001-143002,143009,143020,143022,143024-143025,143027-143028,143035,143037,143048,143051,143056,143058-143060,143069,143071,143074,143081,143084,143109,143113,143115-143117,143119,143122,143125,14312" (truncated))
> │  │  │  ├───0.59 MB (01.80%) ++ string(length=32756, copies=19, "/x89PNG/r/n/x1A/n/x00/x00/x00/rIHDR/x00/x00/x01/x00/x00/x00/x01/x00/b/x06/x00/x00/x00//r/xA8f/x00/x00 /x00IDATx/x9C/xEC}w/x98U/xD5/xD5/xFE/x16Q4/xD1//xBF//x01/x86/x99/xB9s{oS/x00E/xB0 /xB6X/x12/x8D/x8A(/xA8/x9F1/x8A%/x88Q/xA3/xD1/xD8/xD0/xA8h"b/xC3/x9A/xD8+/x8A/nc/x8B/xBD /x1DE/x04A/x9A/xB4a/x98/xB9/xF5/x9C[N/xDF/xEF/xEF/x8FS/xE6/xCEp/xEF/xCC/xA0/xC0/xBD/xC0Y/xCF/xF3>3(/xCC/xDC/xBD/xCFZ/xEFz/xF7/xDAk/xEFC/x88i/xA6/x99f/x9Ai/xA6/x99f/x9Ai/xA6/x99f/x9Ai/xA6/x99f/x9Ai/x15m/x00/xF6/xD1/xD0/x07/xC0/xBE/x05/xE8/x03`/x9F/x1D/xF4;/xF4/x9F/xDD/xB7/xE0g/xEF/xB0/x9Fo/x9Ai/xA6/xF5`]/x83/xBC/x17/xFF/xA4/x0F!/xE4/x00B/xC8/xFF/x10B~/xEDr/xB9/x06/x8E/x181/xA2/xE6/xF8/xE3/x8F/xAF;/xFD/xF4/xD3m/x17//p/x81/xFD/xA2/x8B.r//x/xE1/x85/xCE/xF3/xCE;/xCFq/xF6/xD9g/xDBG/x8F/x1Em/x1D5jT/xED/xE0/xC1/x83/xAB/x0F>/xF8/xE0/xFE/x84/x90/xFFG/b/xF9%!d/xFF^|/xBE>;/x9AxL3m/xAF/xB4./xC1^*/x98/x0E/n/x87/xC3U/x13'N/xF4/xBC/xFC/xF2/xCBG|/xFA/xE9/xA7c/x96,Y2a/xF5/xEA/xD5/xB7n/xD9/xB2/xE5/x91/xF6/xF6/xF6W/x13/x89/xC4/xBB/xA9Tjv&/x93/xF9&/x9B/xCD/xFE/" (truncated))
> │  │  │  ├───0.55 MB (01.66%) ++ string(length=284991, copies=1, "allow-flashallow-digest256;a:1490633678/nbase-track-digest256;a:1512580265/nblock-flash-digest256;a:1496263270/nblock-flashsubdoc-digest256;a:1512160865/nexcept-flash-digest256;a:1494877265/nexcept-flashallow-digest256;a:1490633678/nexcept-flashinfobar-digest256;a:1497379266/nexcept-flashsubdoc-digest256;a:1490633678/ngoog-badbinurl-shavar;a:138620-140037:s:142680,142706,142713,142716,142770,142776,142783,142785,142789-142791,142796-142797,142802-142803,142806-142807,142811,142813,142815-142816,142822,142830-142831,142833,142835-142836,142841-142842,142845-142846,142854,142859,142868,142872,142875,142885-142886,142889-142891,142896-142897,142903-142904,142906-142907,142912-142916,142919-142921,142937,142950,142952-142953,142955-142956,142961,142972,142974,142977,142982-142983,142986,142990,142993-142994,142998,143001-143002,143009,143020,143022,143024-143025,143027-143028,143035,143037,143048,143051,143056,143058-143060,143069,143071,143074,143081,143084,143109,143113,143115-143117,143119,143122,143125,14312" (truncated))
> │  │  │  └───0.50 MB (01.52%) -- string(length=271492, copies=1, "goog-malware-shavar;a:260843-279834:s:256208-256217,256219-256238,256240-256278,256280-256285,256287-256398,256400-256426,256428-256477,256479-256481,256483-256544,256546-256563,256565-256567,256569-256575,256577-256579,256581-256613,256615-256618,256620-256625,256627,256629-256644,256646-256648,256650-256656,256658-256662,256664-256667,256670-256678,256681,256683-256686,256690-256691,256693-256718,256720-256727,256729,256731-256736,256738-256780,256782,256784-256803,256805-256883,256885,256888-256899,256901-256963,256965-256974,256976-256977,256979,256981-256986,256988-257000,257002-257025,257027-257057,257059-257134,257136-257165,257167-257193,257195-257203,257205-257209,257211-257213,257216-257229,257231-257261,257263-257265,257267-257282,257286,257288-257297,257299-257300,257302-257377,257379-257399,257401-257413,257416-257417,257419-257444,257446-257548,257550-257557,257559-257560,257562-257714,257716-257798,257800-257854,257857-257860,257862-257934,257936-257942,257944-258090,258092-258103,258105-25818" (truncated))
> │  │  │      ├──0.50 MB (01.52%) ── malloc-heap/latin1
> │  │  │      └──0.00 MB (00.00%) ── gc-heap/latin1
> │  │  ├───9.16 MB (27.78%) ++ compartment([System Principal], shared JSM global)
> │  │  ├──-3.27 MB (-9.90%) ── unused-gc-things [4]
> │  │  ├───0.53 MB (01.62%) ++ shapes
> │  │  ├───0.62 MB (01.89%) ++ object-groups
> │  │  ├───0.43 MB (01.30%) ── type-pool [3]
> │  │  ├───0.38 MB (01.15%) ++ (13 tiny)
> │  │  ├───0.38 MB (01.14%) ── baseline/optimized-stubs
> │  │  ├───0.38 MB (01.14%) ── shape-tables [2]
> │  │  └───0.38 MB (01.14%) ── unique-id-map
> │  ├───6.63 MB (20.10%) ++ runtime
> │  └───0.03 MB (00.09%) ── gc-heap/chunk-admin
> ├──13.91 MB (42.16%) ── heap-unclassified
> ├──-6.43 MB (-19.50%) ++ startup-cache
> ├──-6.84 MB (-20.73%) ++ heap-overhead
> ├───3.91 MB (11.85%) ++ xpconnect
> ├───2.71 MB (08.23%) -- add-ons
> │   ├──2.77 MB (08.38%) -- [e-mail]/js-non-window/zones/zone(0xNNN)
> │   │  ├──1.14 MB (03.46%) ++ (24 tiny)
> │   │  ├──0.65 MB (01.98%) ++ compartment([System Principal], resource://activity-stream/common/Dedupe.jsm)
> │   │  ├──0.53 MB (01.62%) ++ compartment([System Principal], resource://activity-stream/lib/LinksCache.jsm)
> │   │  └──0.44 MB (01.32%) ++ compartment([System Principal], resource://activity-stream/lib/FilterAdult.jsm)
> │   └──-0.05 MB (-0.16%) ++ (6 tiny)
> ├──-1.09 MB (-3.30%) ++ workers/workers(chrome)
> ├──-1.11 MB (-3.37%) ++ window-objects
> ├───0.38 MB (01.15%) ── atom-tables/main
> └───0.06 MB (00.19%) ++ (7 tiny)

Comment 7

[Eric Rahm [:erahm]]

Sorry, my footnotes:

[1] https://wiki.mozilla.org/Services/Shavar
[2] https://developer.mozilla.org/docs/Mozilla/Performance/DMD

Comment 8

[Marius Coman [:mcoman], Ecosystem QA]

Andy, Luke, and Ed, could you guys give us your opinion regarding Eric's assessment? Is this expected in this case?

Comment 9

[Luke Crouch [:groovecoder]]

Yes, those flash block lists are served by shavar. :cpeterson knows more about the client-side implementation I think.

Comment 10

[Chris Peterson [:cpeterson]]

Kicking this needinfo to Francois who would know more about the Shavar client code and its memory usage.

Comment 11

[Andy McKay]

versioncheck-bg.addons.mozilla.org is the checking of add-on versions. Strings from that sticking around seems undesireable though.

Comment 12

[Andrew Swan [:aswan]]

(In reply to Eric Rahm [:erahm] (please no mozreview requests) from comment #6)
> In the small
> string category there's a ton of "versioncheck-bg.addons.mozilla.org" which
> I'm going to guess is related to checking for new versions of add-ons. We
> might want to ping them about why those strings are sticking around.

I don't see these sorts of strings in my regular Nightly profile.  Is it possible that the memory report was gathered during a moment while an extension update check was happening?

Comment 13

[François Marier [:francois]]

Those strings look like what the URL classifier list manager uses when requesting list updates from the server.

Comment 14

[Ed Lee :Mardak]

(In reply to Eric Rahm [:erahm] (please no mozreview requests) from comment #6)
> > ├───2.71 MB (08.23%) -- add-ons
> > │   ├──2.77 MB (08.38%) -- [e-mail]/js-non-window/zones/zone(0xNNN)
> > │   │  ├──1.14 MB (03.46%) ++ (24 tiny)
> > │   │  ├──0.65 MB (01.98%) ++ compartment([System Principal], resource://activity-stream/common/Dedupe.jsm)
> > │   │  ├──0.53 MB (01.62%) ++ compartment([System Principal], resource://activity-stream/lib/LinksCache.jsm)
> > │   │  └──0.44 MB (01.32%) ++ compartment([System Principal], resource://activity-stream/lib/FilterAdult.jsm)
There are indeed 23 jsms in lib/ and 5 in common/ so the total count mostly matches up. Dedupe.jsm being the largest one does seem a bit suspicious especially given that it's a relatively small / simple class:

https://searchfox.org/mozilla-central/source/browser/extensions/activity-stream/common/Dedupe.jsm

We actually have looked into webpacking the jsms into a single file, but I believe it was frowned upon for checking in that generated module to mozilla-central. We do convert the common/ jsms to esmodules to be webpack with our other content js files (esmodules) for an optimizing export of our content bundle.

Although, if we assume the 1.14MB for 24 jsms is all overhead, then it's about 50KB per jsm that we could maybe save with a single file?