658922 - High images/content/used/uncompressed values on mangafox.com?

Status: RESOLVED DUPLICATE of bug 683284

(Core :: Graphics: ImageLib, defect)

Description

[Nicholas Nethercote [inactive]]

Spun off from bug 639129.

jakobi reported seeing high values (up to several GB) for
images/content/used/uncompressed for comics at http://mangafox.com.  Also,
this memory was not being released when the tab was closed.

Apparently this add-on helped:
https://addons.mozilla.org/en-us/firefox/addon/empty-cache-button/
but the high memory usage was still visible in about:memory is in
gfx/surface/image.

Furthermore, changing these settings in about:config:

image.mem.decodeondraw: true            # unknown if really required
image.mem.min_discard_timeout_ms: 10000 # back to 10s from 120s aka "never"!?

Seemed to pretty much fix the problem.

jakobi also said that NoScript, Adblock Plus and GreaseMonkey were installed.

----

I tried to reproduce on my Linux64 build.  I tried this comic:

http://www.mangafox.com/manga/are_you_alice/v01/c010/1.html

and clicked "next" to step through all 26 pages, waiting on each page until it
had finished loading.  Memory usage seemed very normal.  The output of about:memory is below (about:memory has changed a lot in dev versions recently, which is why it looks different).  images/content/used/uncompressed only got up to 7.44MB which seems reasonable for 26 large images.

----

jakobi:  what happens if you follow my steps to reproduce?  Are there other specific steps I should try instead?  I wonder if your add-ons are affecting the problem... you could try disabling them all to see if that helps.

----

Main Process

Explicit Allocations
82.93 MB (100.0%) -- explicit
├──39.35 MB (47.45%) -- js
│  ├──25.00 MB (30.14%) -- gc-heap
│  ├───4.81 MB (05.80%) -- mjit-code
│  ├───4.35 MB (05.24%) -- scripts
│  ├───2.43 MB (02.93%) -- tjit-data
│  │   ├──1.83 MB (02.21%) -- allocators-reserve
│  │   └──0.59 MB (00.71%) -- allocators-main
│  ├───0.83 MB (01.01%) -- object-slots
│  ├───0.74 MB (00.89%) -- mjit-data
│  ├───0.63 MB (00.75%) -- tjit-code
│  └───0.57 MB (00.68%) -- string-chars
├──26.12 MB (31.49%) -- heap-unclassified
├───9.33 MB (11.25%) -- images
│   ├──9.25 MB (11.16%) -- content
│   │  ├──7.75 MB (09.35%) -- used
│   │  │  ├──7.44 MB (08.97%) -- uncompressed
│   │  │  └──0.31 MB (00.38%) -- (1 omitted)
│   │  └──1.50 MB (01.81%) -- unused
│   │     ├──1.39 MB (01.67%) -- uncompressed
│   │     └──0.12 MB (00.14%) -- (1 omitted)
│   └──0.08 MB (00.10%) -- (1 omitted)
├───4.50 MB (05.42%) -- storage
│   └──4.50 MB (05.42%) -- sqlite
│      ├──2.46 MB (02.97%) -- places.sqlite
│      │  ├──2.05 MB (02.47%) -- cache-used
│      │  └──0.41 MB (00.50%) -- (2 omitted)
│      ├──0.64 MB (00.77%) -- other
│      ├──0.51 MB (00.62%) -- urlclassifier3.sqlite
│      │  └──0.51 MB (00.62%) -- (3 omitted)
│      └──0.89 MB (01.07%) -- (9 omitted)
├───2.16 MB (02.61%) -- gfx
│   └──2.16 MB (02.61%) -- surface
│      └──2.16 MB (02.61%) -- image
└───1.48 MB (01.78%) -- layout
    ├──1.48 MB (01.78%) -- all
    └──0.00 MB (00.00%) -- (1 omitted)

Other Measurements
669.25 MB -- vsize
137.23 MB -- resident
 92.00 MB -- heap-committed
 77.50 MB -- heap-used
 14.50 MB -- heap-unused
  2.65 MB -- heap-dirty

Comment 1

[jakobi]

update for the last few days:

> * Apparently this add-on helped:
> https://addons.mozilla.org/en-us/firefox/addon/empty-cache-button/
> but the high memory usage was still visible in about:memory is in
> gfx/surface/image.

if freeing was successful, the overall memory usage did shrink
(seems to clear images/content/*). However, part of the time, the
"released memory" wasn't reflected in gfx/surface/* AND/OR content/canvas*.

Maybe my use of this was more a problem-detection rather than problem-workaround

Currently I'm doing this instead:


> * image.mem.min_discard_timeout_ms: 10000

possibly touches the same underlying set of memory bugs and 
reduces the likelihood of loosing a huge chunk of memory at once. Maybe 
also a bit cosmetic:)


* changing to a new profile with the usual addons (esp. adb+, request policy, noscript). This reduced frequency of bloated firefox processes to once every
few days when NOT visiting mangafox.


* Adding a restart button and having system-monitor display a buffer/cache/swap
widget also helps to notice and restart a memory-greedy firefox before performance or even X11 suffers.


wrt mangafox: 
- it's not required to trigger the behaviour, but its combination of 
  images cum javascript seems to be a good trigger for some of the
  memory bugs in question
- it's not THAT image heavy, as it's primary content is one image per page,
  plus a bit more javascript than I'd like. However during a visit you'd 
  load a few dozens of pages in a fairly short moment, esp. with a greasemonkey
  script like mangafox-ajax-preloader.

==

I'll have a look later today at mangafox.com later (how much faster will the firefox memory bloat be triggered?).

Comment 2

[Nicholas Nethercote [inactive]]

Bug 664290 changed image.mem.min_discard_timeout_ms to 10000.  I wonder if that's enough to mark this bug as fixed.

Comment 3

[Justin Lebar (not reading bugmail)]

> I tried to reproduce on my Linux64 build [but reported memory usage didn't grow].

This is probably bug 665952.

Comment 4

[jakobi]

@3: given the about:memory values noticed in images/* with my original profile, interaction of transfering image data to the xserver might be part of the code path in question.

Note that I've set MOZ_DISABLE_IMAGE_OPTIMIZE=1 to avoid X growth due to still cached, but no longer used image data. At least that's what I dimly remember about the workaround concerning X growth from ages ago. 


===

I had some weeks in the meantime to try the setup from #1 against mangafox. Pretty boring, i.e. nothing of interest to report:

I think I noticed some slow growth, but nothing to affect usability or to frequently close and restart firefox like with my original profile (despite similar addons incl. adblockplus, noscript and greasemonkey (mangafox ajax preloader)). Memory usage is still pretty high.

Example URL for testing: 
http://www.mangafox.com/manga/someday_s_dreamers_spellbound/v01/c000/1.html

Switching from a normal tab (say leo.org) to a mangafox tab is interesting:
Growth by 0.6GB to 2.1GB memory mapped and 1.3GB in content/used/uncompressed with my settings. Most of which is released after a few minutes downto 1.5GB/0.6GB.

This increase/decrease is visible also in top. (wrt. Justin's comment 3)

This is a huge and not-really-necessary memory jump, considering that the mangafox page displays only a single main image at once of maybe 100K, with about 90 images of similar size being preloaded by the userscript.

But as far as I've seen, this memory was NOT leaked afterwards.

Switching to Firefox 5 shows the same pattern. 

In a week or so, I'll switch to firefox-trunk as packaged with ubuntu. Maybe then this bug will turn more interesting for debugging :).


===

Ad those strange multiple-monitor firefox-X11 hangs:

However, I was bitten with one reoccurrence of the strange linux nvidia/twinview/X-Org freeze with a mouse pointer being stuck between two heads (some X tweaks and the new profile got rid of the issue in firefox4).
 
As usual just after tab switching in firefox. X frozen, with the system slowly
using more and more swap, but acc. to a quick ps that was  visible in memory sizes for neither X11 nor firefox5... 

Possibly a hopefully _rare_ racecondition somehow related to our leak? Something talking to X11 but being paged out?

Comment 5

[Nicholas Nethercote [inactive]]

I'm not sure what to do with this bug.  Close it?  Sounds like the worst of the problems have gone away, and general image handling is covered by 683284 and its dependents.

Comment 6

[Justin Lebar (not reading bugmail)]

(In reply to Nicholas Nethercote [:njn] from comment #5)
> I'm not sure what to do with this bug.  Close it?  Sounds like the worst of
> the problems have gone away, and general image handling is covered by 683284
> and its dependents.

Agreed.