1149893 - Add a pref to decode all images immediately, to reduce "pop in" at the expense of memory use

Status: RESOLVED FIXED

(Core :: Graphics: ImageLib, defect)

Description

[her34her34]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0
Build ID: 20150122214805

Steps to reproduce:

Create new profile.

Make sure setting is [image.mem.decodeondraw;false]

Load tabs:
1) http://www.neogaf.com/forum/showthread.php?t=996617

2) http://www.neogaf.com/forum/showthread.php?t=996617&page=2

Wait for tabs to finish loading.

If you check memory usage from windows task manager firefox is using ~320mb.

After tabs finished loading, grab scroll bar and try scrolling down to various new positions.

After you reach bottom of each page memory usage is ~800mb indicating now images are actually decoded.


Actual results:

As you scroll down to a new position you will see text and a blank area where the image should be. The blank area only shows website background. After a second the image will then appear/pop in.

This pop in keeps happening as you scroll down


Expected results:

There should be no pop in, text and images should be visible immediately and at the same time.

After tabs finish loading, the images should already be decoded without scrolling and memory usage should be ~800mb without having to scroll.

Comment 1

[Seth Fowler [:seth] [:s2h]]

(In reply to her34her34 from comment #0)
> Make sure setting is [image.mem.decodeondraw;false]

image.mem.decodeondraw literally does nothing in desktop Firefox, and it hasn't for a long time. Please don't use it. The pref has been removed in Firefox 39. I suspect you might want to set "image.mem.discardable" to false instead.

As far as the rest of the report, are you saying to load those two tabs in the background?

If so, that would make sense. With "image.mem.discardable" set to true, which is the default, images in background tabs will be discarded after a while. Since those pages, at least for me, take a *very* long time to load, it's likely that most of the images in them are discarded by the time they've finished loading completely.

Can you confirm if setting "image.mem.discardable" to false eliminates the problem?

Comment 2

[her34her34]

(In reply to Seth Fowler [:seth] from comment #1)

> image.mem.decodeondraw literally does nothing in desktop Firefox, and it
> hasn't for a long time.

The setting "image.mem.decodeondraw" definitely works in Firefox 35. It decodes all images without having to scroll into view first.

Step to reproduce:
1) install firefox 35 and new profile
2) set "image.mem.decodeondraw" to false
3) set "image.mem.discardable" to false (just to make sure decoded images aren't discarded)
4) open the 2 url and wait for finish loading
5) see that windows task manager says firefox using ~1100mb
6) scroll down either tab. all images show immediately, no pop in
7) set "image.mem.decodeondraw" to true
8) close firefox, start firefox, open 2 url again and wait for finish loading
9) see that windows task manager says firefox using ~300mb
10) scroll down either tab. images pop in



> As far as the rest of the report, are you saying to load those two tabs in
> the background?

No, it doesn't matter. Open 2 url and keep focus on either tab. Just don't scroll while tabs are loading.



> If so, that would make sense. With "image.mem.discardable" set to true,
> which is the default, images in background tabs will be discarded after a
> while. Since those pages, at least for me, take a *very* long time to load,
> it's likely that most of the images in them are discarded by the time
> they've finished loading completely.
> 
> Can you confirm if setting "image.mem.discardable" to false eliminates the
> problem?

In firefox 37 with "image.mem.discardable" set to false, pop in still happens.


Do you not see images pop in when you scroll down?

Comment 3

[Seth Fowler [:seth] [:s2h]]

(In reply to her34her34 from comment #2)
> The setting "image.mem.decodeondraw" definitely works in Firefox 35. It
> decodes all images without having to scroll into view first.

Yes, you're absolutely right. I thought that Firefox 35 was when that pref stopped doing anything, but it appears to have been somewhat more recently.

Nonetheless, "image.mem.decodeondraw" is unfortunately not something we can support anymore. That pref controlled a lot of machinery, not just the behavior you want. Since that other machinery is gone, we can't really support "image.mem.decodeondraw". We could consider adding a new pref targeted at just the behavior you want, though - it sounds like you'd like Firefox to decode images automatically as soon as they are downloaded.

Before we get to that point, though, let's see if we can actually fix the problem. Ideally you'd get an experience that you are happy with, without changing any prefs. That's the best possible outcome.

> In firefox 37 with "image.mem.discardable" set to false, pop in still
> happens.

OK, thanks. That suggests it's not just image discarding that is the cause of the issue.

> Do you not see images pop in when you scroll down?

These pages load very slowly for me. (That's part of why I asked about discarding.) But if I let them load completely, then no, I can't reproduce the problem. I tried on both Nightly and 36.

What kind of processor and memory does your machine have?

Comment 4

[her34her34]

(In reply to Seth Fowler [:seth] from comment #3)

> Nonetheless, "image.mem.decodeondraw" is unfortunately not something we can
> support anymore. That pref controlled a lot of machinery, not just the
> behavior you want. Since that other machinery is gone, we can't really
> support "image.mem.decodeondraw".

Thank you for the explanation.


> We could consider adding a new pref
> targeted at just the behavior you want, though - it sounds like you'd like
> Firefox to decode images automatically as soon as they are downloaded.

Yes. The goal is exactly as you said, decode all images immediately and keep decoded image until tab is closed.



> What kind of processor and memory does your machine have?

I have 2 systems: older core 2 duo with 4gb ram and newer core i5 with 8gb ram. Pop in happens on both. After you said it didn't happen for you I was worried I was doing something on my systems to cause the issue so I checked various computers of other people I know and pop in happened on their systems and they could all clearly see it. So i think either you have a super fast system that can decode faster than perceivable or I'm missing something in my steps to reproduce. Maybe the way we are scrolling is different.

Can you please try these steps:

1) set "image.mem.decodeondraw" to false
2) set "image.mem.discardable" to false (just to make sure decoded images aren't discarded)
3) open the 2 url and wait for finish loading
4) open scratchpad and paste: "window.scrollTo(0,window.pageYOffset+1500);"
5) in scratchpad keep clicking "run" button to scroll down page



> These pages load very slowly for me. (That's part of why I asked about
> discarding.) But if I let them load completely, then no, I can't reproduce
> the problem. I tried on both Nightly and 36.

Do you at least see the memory behavior I describe? After pages finish loading ram is low and then after scrolling down pages ram is much higher, indicating images weren't decoded until scrolled into view?

Comment 5

[Seth Fowler [:seth] [:s2h]]

(In reply to her34her34 from comment #4)
> Can you please try these steps:
> 
> 1) set "image.mem.decodeondraw" to false
> 2) set "image.mem.discardable" to false (just to make sure decoded images
> aren't discarded)
> 3) open the 2 url and wait for finish loading
> 4) open scratchpad and paste: "window.scrollTo(0,window.pageYOffset+1500);"
> 5) in scratchpad keep clicking "run" button to scroll down page

Yes, I can reproduce some pop-in when scrolling in this way. Which makes sense, I suppose, since scrolling in this way causes a very large and sudden jump in page position. I still can't reproduce this when scrolling using the touchpad, no matter how fast I scroll. (It's possible that it actually is happening, but it's just imperceptible in the general rapid visual change of scrolling.)

> Do you at least see the memory behavior I describe? After pages finish
> loading ram is low and then after scrolling down pages ram is much higher,
> indicating images weren't decoded until scrolled into view?

Yeah, I absolutely can. This part I could always reproduce.

Comment 6

[Seth Fowler [:seth] [:s2h]]

her34her34, do you see less pop-in when scrolling via other methods? For example, the touchpad or the keyboard?

When I scroll by hitting the spacebar (even holding it down), or using the touchpad, I see either no pop-in or so little it's not noticeable. To reproduce this, I have to make a big jump in scroll position by either grabbing the scrollbar or using the JavaScript you provided. Is it the same for you?

Comment 7

[karlsenk]

I just wanted to say that this is a real problem. The developer is saying that it's barely noticeable, which it very well might be for him, but on my computer it is very noticeable. The 'steps to reproduce' by OP do not capture real world. With a new install and no extensions and loading the first tab you are getting best possible performance which is not real world where people install multiple extensions and browse multiple websites which all cause some slow down in firefox and causes more pop in.

Another thing is that developer is using touchpad to scroll and I think has smooth scrolling turned on. I turn off smooth scrolling because I don't like it but yeah if it's turned on then it will hide pop in. The other thing is that if you scroll with a mouse then you are likely using scroll wheel and everyone I know does not limit themselves to scrolling one page at a time. Sometimes they scroll a little while reading other times they scroll a lot and flick the wheel to get down page and there will be a pop in.

I try to keep all software up to date for security but I also really hate pop in. #firstworldproblems

Comment 8

[Seth Fowler [:seth] [:s2h]]

(In reply to karlsenk from comment #7)
> Another thing is that developer is using touchpad to scroll and I think has
> smooth scrolling turned on.

Yes, I'm using the default preferences except where her34her34 suggested otherwise.

> I turn off smooth scrolling because I don't like
> it but yeah if it's turned on then it will hide pop in.

So you don't see pop in with smooth scrolling off, but do with it on?

> The other thing is
> that if you scroll with a mouse then you are likely using scroll wheel and
> everyone I know does not limit themselves to scrolling one page at a time.
> Sometimes they scroll a little while reading other times they scroll a lot
> and flick the wheel to get down page and there will be a pop in.

I don't have a mouse wheel that works that way (mine click, they don't rotate freely) so I can't test that particular case, but as I mentioned above, I don't get noticeable pop in while holding down the spacebar to scroll rapidly down the page. I also don't get noticeable pop in when using a flick gesture on the touchpad to scroll down the page quickly.

What I'm trying to do here is understand the circumstances in which people are and are not getting pop in.

So far I've heard that people are getting pop in when:

- Grabbing the scrollbar and moving it manually. (I can reproduce this.)
- Flicking a mouse wheel that rotates freely to move rapidly down the page. (I don't have one; can't test this.)
- Scrolling when smooth scrolling is disabled. (Haven't had a chance to test this yet.)

These other scroll methods don't seem to produce pop in:

- Scrolling via the touchpad.
- Scrolling via the keyboard. (At least the spacebar and arrow keys seem to be fine.)
- Scrolling using the scrollbar arrows.
- Scrolling via a mouse wheel that clicks.

I'd appreciate help confirming these lists and adding anything that's missing from them.

Comment 9

[Seth Fowler [:seth] [:s2h]]

(In reply to Seth Fowler [:seth] from comment #8)
> So you don't see pop in with smooth scrolling off, but do with it on?

Sorry, I meant the opposite of this. You *do* see pop in with smooth scrolling off, but you *don't* see pop in with smooth scrolling on, correct?

Comment 10

[her34her34]

(In reply to Seth Fowler [:seth] from comment #6)
> her34her34, do you see less pop-in when scrolling via other methods? For
> example, the touchpad or the keyboard?
> 
> When I scroll by hitting the spacebar (even holding it down), or using the
> touchpad, I see either no pop-in or so little it's not noticeable. To
> reproduce this, I have to make a big jump in scroll position by either
> grabbing the scrollbar or using the JavaScript you provided. Is it the same
> for you?

I'm wondering what kind of computer are you using?


If I press spacebar once, firefox scrolls down 1 page, I've only seen pop in a few times.

If I hold down spacebar for a few seconds, firefox scrolls down 4-6 pages, I release spacebar, firefox stops scrolling, pop in happens most of the time.

And let me say that the steps I wrote out were simplest way I could reproduce problem, the idea was to find simplest way for people at firefox to see the problem on their computers so they can fix it. During my normal browsing pop in happens sometimes when I scroll just 1 page but explaining all the tabs to open and then close, how much to scroll each tab, how long to wait, etc, is not easily reproducible.

The real problem doesn't seem to be this scroll method or that scroll method. The real underlying problem seems to be that firefox cannot always decode an image fast enough which is why the old option was good because all images were decoded. So will you add an option to do that? "image.mem.decodeAllAfterDownload"?

Comment 11

[Seth Fowler [:seth] [:s2h]]

(In reply to her34her34 from comment #10)
> I'm wondering what kind of computer are you using?

I don't think my computer is significantly faster than yours, though it's hard to tell for sure because a Core i5 processor could be anything from a Nehalem to a Broadwell. It certainly could be the case that CPU speed is an issue here. I'm not on Windows, and platform-specific differences may be a factor here as well.
But I think it's most likely just that our heuristics need tweaking - we're not starting image decodes early enough on your machine.

> The real problem doesn't seem to be this scroll method or that scroll
> method.

These different scroll methods imply different rates of movement through the document. That difference  matters since we need to predict which images are going to be in view soon and start decoding them. Most users aren't going to set a non-default preference, so we need things to work well for them in the default configuration, and understanding where we're failing is important.

Unfortunately I can't think of any heuristics that will work when you make huge jumps in the document by suddenly moving the scrollbar a large distance. The STR you gave in comment 0 involve a situation that is very hard for us to improve. So one thing I'm trying to understand here is if this happens for you in other situations where we *can* improve the experience.

> The real underlying problem seems to be that firefox cannot always
> decode an image fast enough which is why the old option was good because all
> images were decoded. So will you add an option to do that?
> "image.mem.decodeAllAfterDownload"?

I don't see any problem with adding an option like that.

Comment 12

[Seth Fowler [:seth] [:s2h]]

I've investigated more based on what you've told me so far. The case where you scroll by holding down the spacebar is one that I'm really concerned about, because it's very aggressive scrolling but we should still be able to make it work on most machines without second-long pauses.

Initially, I couldn't reproduce significant levels of pop in by holding down the spacebar, even on Windows. So I decided to try reproducing in a VM where I could vary the amount of resources. I experimented with a lot of things, and what seem to matter the most by far is the number of cores. I'm easily able to reproduce severe pop in by only giving the VM access to 2 cores. That makes sense, as image decoding is highly parallel and performance really depends on the number of cores in your machine.

I checked if this is a recent regression and it doesn't seem to be. I scrolling the sites in comment 0 by holding down the spacebar in Firefox 35 and saw the same behavior. (The "image.mem.decodeondraw" changes, of course, *are* recent.)

So here's the plan. I'm going to attack this from two directions.

1. In this bug, I'm going to add a pref that will decode all images as soon as they've loaded, as requested above.

2. I'm going to file a new bug to try to improve our scrolling performance in this test case. I think there are some things we can do here.

Comment 13

[Seth Fowler [:seth] [:s2h]]

Attachment: Add a pref that makes us decode all images immediately

Here's the patch that adds the new pref, "image.decode-immediately.enabled".

Comment 14

[Seth Fowler [:seth] [:s2h]]

Attachment: Add a pref that makes us decode all images immediately

Thanks for the review!

I made a trivial tweak in this version of the patch: the C++ default value is
now consistent with the value specified in all.js.

Comment 15

[Seth Fowler [:seth] [:s2h]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c8a395aa2fe1

Comment 16

[Seth Fowler [:seth] [:s2h]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6950dddc0c5c

Comment 17

[johnl15278]

Will the new preference be in Firefox 38?

Comment 18

[Seth Fowler [:seth] [:s2h]]

(In reply to johnl15278 from comment #17)
> Will the new preference be in Firefox 38?

Once it lands in Firefox 40, I can request uplift to 38 and 39. From there a release manager will make a decision as to whether to take the change in those versions.

In other words, we don't know yet. =)

Comment 19

[Seth Fowler [:seth] [:s2h]]

I've filed bug 1152147 to try to reduce pop in on the page in comment 0.

Comment 20

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/6950dddc0c5c

Comment 21

[johnl15278]

At the top of this page it says 
Target Milestone: mozilla40 

Does that mean it landed in Firefox 40 and so uplift to 38 can be requested?

Comment 22

[Seth Fowler [:seth] [:s2h]]

Comment on attachment 8589302 [details] [diff] [review]
Add a pref that makes us decode all images immediately

Approval Request Comment
[Feature/regressing bug #]: Tough to assign a specific bug number. As we've increasingly relied on image visibility tracking heuristics to trigger decoding, "image.mem.decodeondraw" has been less and less meaningful. At this point the pref has totally been removed. Unfortunately, some users relied on it to make us decode all images on a page immediately, which helps perceived scrolling performance on some systems.
[User impact if declined]: Users will lose a pref that gave them better perceived scrolling performance. (Until Firefox 40 gets released, at least.)
[Describe test coverage new/current, TreeHerder]: On m-c for a week.
[Risks and why]: Virtually zero risk. The new pref is disabled by default.
[String/UUID change made/needed]: None.

Comment 23

[Seth Fowler [:seth] [:s2h]]

(In reply to johnl15278 from comment #21)
> Does that mean it landed in Firefox 40 and so uplift to 38 can be requested?

Yep, just did it.

Comment 24

[Sylvestre Ledru [:Sylvestre]]

If I understand correctly, this change landed with 35.
If I am correct, there is no rush for 38 but we can take it for 39.

Comment 25

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/654d90737b68

Comment 26

[Bogdan Maris, Desktop QA]

Tried to reproduce this issue on older build (FF 35.0.1) using steps from comment 0, comment 2 and comment 4 but without success. 
Can you please check if the issue is fixed in Firefox 39 beta 7 and latest Aurora?
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/39.0b7-candidates/build1/win32/
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/

Comment 27

[alexgumbe]

I am seeing image flashing in firefox 41.0.1

I made new profile and only made 2 setting changes:
image.decode-immediately.enabled;true
general.smoothScroll;false

When I scroll I see an image and after half a second the images flashes to a new image that is the same image but at a different image quality.

This happens on all websites but only sometimes. It seems related to how slow the browser is running so if there are other tabs that are busy loading then I am more likely to see image flashing while scrolling down current tab.

I have attached screenshot samples.
a1.png is what I see when I first scroll, then after half a second I see a2.png
b1.png is what I see when I first scroll, then after half a second I see b2.png

Comment 28

[alexgumbe]

Attachment: a1

Comment 29

[alexgumbe]

Attachment: a2.png

Comment 30

[alexgumbe]

Attachment: b1.png

Comment 31

[alexgumbe]

Attachment: b2.png

Comment 32

[karlsenk]

Will this problem be fixed? 

I have noticed this problem.

It looks like a progressive jpeg image where it starts off with one quality and then changes to another quality.

Firefox is unintentionally decoding the same image twice.