1140619 - [Gallery] Double tapping an image to zoom will cause the image to flicker.

Status: VERIFIED FIXED

(Firefox OS Graveyard :: Gaia::Gallery, defect, P3)

Description

[Martin Shuman [:Marty]]

Attachment: logcat_gallery_flicker.txt

Description:
After opening an image in the Gallery, double tapping the image to zoom the first time will cause a black box to flicker.  This does not occur subsequent times, unless the user first returns to the thumbnail gallery, then returns to fullscreen mode.


Repro Steps:
1) Update a Flame to 20150306010207
2) Open the camera app and take a picture.
3) Close the camera app and open the gallery app.
4) Select the picture that was just taken
5) Double-tap the image to zoom in


Actual:
A black flicker occurs as the image zooms in.

Expected:
The image zooms in smoothly, without flickering.

Environmental Variables:
Device: Flame 3.0 (319MB)(Full Flash)
Build ID: 20150306010207
Gaia: 7a91c16bfa348be8b25e09719178efa051512988
Gecko: 0189941a3fd5
Gonk: e7c90613521145db090dd24147afd5ceb5703190
Version: 39.0a1 (3.0)
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:39.0) Gecko/39.0 Firefox/39.0

Repro frequency: 8/10
See attached: video clip (URL), logcat

Comment 1

[Martin Shuman [:Marty]]

This issue does NOT occur in Flame 2.2.
The image zooms without flickering.

Environmental Variables:
Device: Flame 2.2 (319MB)(Full Flash)
Build ID: 20150305002528
Gaia: 89af288bad6751248ff84504fa898206fee127fe
Gecko: 6d8d294aa8f3
Gonk: e7c90613521145db090dd24147afd5ceb5703190
Version: 37.0 (2.2)
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:37.0) Gecko/37.0 Firefox/37.0

Comment 2

[KTucker [:KTucker][Inactive 3/4/2016]]

[Blocking Requested - why for this release]:

This looks pretty bad and it's a regression so nominating 3.0?

Comment 3

[Pi Wei Cheng [:piwei] (inactive)]

mozilla-inbound regression window:

Last Working Environmental Variables:
Device: Flame
BuildID: 20150210141156
Gaia: 8c7865486a1b11076b849bbf8f7fccbaffbfafe7
Gecko: 7fed536bd689
Version: 38.0a1 (3.0 Master)
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:38.0) Gecko/38.0 Firefox/38.0

First Broken Environmental Variables:
Device: Flame
BuildID: 20150210143255
Gaia: 8c7865486a1b11076b849bbf8f7fccbaffbfafe7
Gecko: 6f42cfe37f5c
Version: 38.0a1 (3.0 Master)
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:38.0) Gecko/38.0 Firefox/38.0

Gaia is the same so it's a Gecko issue.

Gecko pushlog:
http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=7fed536bd689&tochange=6f42cfe37f5c

Possibly caused by Bug 1129804.

Comment 4

[KTucker [:KTucker][Inactive 3/4/2016]]

Seth, can you take a look at this please? This looks to be caused by the landing for bug 1129804.

Comment 5

[Seth Fowler [:seth] [:s2h]]

David, what's the Gallery code doing here?

If this was caused by flipping on decode-on-draw, then I suspect it means that the Gallery code is displaying an image that it has never displayed before when you zoom in. (Probably the zoomed-out version is a thumbnail - i.e., it's actually a different image?) We can probably fix this with either a heuristic or by changing the gallery code.

Comment 6

[David Flanagan [:djf]]

Seth: this zoom code in gallery is exactly the code that I predicted would see the biggest impact from your decode-on-draw patch. I have not gotten around to modifying that code, but I have bug 1121648 filed for it.  From our previous discussions, I expect I can just take all the tricky stuff out of gallery and have gecko just do the right thing when the user zooms. Gallery is still using #moz-samplesize to downsample the originally displayed image, then when the user zooms, that fragment is removed and we decode the full size of the image.

Is the decode-on-draw patch going to end up in 2.2? If so, we should make this a blocker.

I'm taking the bug, but setting needinfo for Seth to get a definitive answer to the 2.2 question.

Comment 7

[Seth Fowler [:seth] [:s2h]]

Both decode-on-draw and downscale-during-decode are targeted for 2.2. They are pref'ed off in 2.2 right now, though. (The code is there.)

As far as bug 1121648, that will certainly change this behavior. There are actually two features that are interacting here:

- Decode-on-draw makes us wait to decode an image until someone tries to draw it.

- Downscale-during-decode makes us downscale images to the size they are going to be drawn at, as they are being decoded.

(Both of these are only enabled for JPEG images currently.)

Decode-on-draw is pref'ed on on trunk, and it's the cause of this bug. I have not pref'ed on downscale-during-decode yet, but I intend to very soon.

Here's how things will work if you move away from moz-samplesize in the Gallery code and use the same image in both cases, relying on downscale-during-decode:

When you display the smaller version, we will decode the image directly to that size. Note that so far we have used *much less* memory than the existing solution, because we've only decoded at the smaller size. (And downscale-during-decode is also a bit more memory-efficient than moz-samplesize in some cases.)

When the user zooms, we will try to display a version of the image decoded to its full size, but we won't find one. We'll substitute the smaller version we decoded before, and start redecoding. When the new version is available, we'll redraw the image. This means that the effect will be that the zoomed in image will be pixelated for a moment and then the higher resolution version will snap into place. The memory usage here will be about the same as the existing solution.

The question is: is displaying the image pixelated for a moment (replacing the "flash" being complained about in this bug) an acceptable user experience?

If so, then just moving from moz-samplesize to downscale-during-decode is enough to solve this bug.

If not, then we'll need to spend the additional CPU and memory to decode the higher resolution version of the image eagerly, just as we do now. That means that we'll need some heuristic to trigger decoding at full resolution on the Gecko side. (Drawing the image to a canvas would work, but that's wasteful if you don't need the canvas for anything.) If that the direction we need to go, we can start discussing what such a heuristic might look like.

Let me know what you think.

Comment 8

[David Flanagan [:djf]]

I discussed this with Seth on IRC. Once he enables downsample-on-decode (bug 1124084), then I'll work to fix this bug.

If that gets enabled on 2.2, then we'll need to uplift this to 2.2

Comment 9

[David Flanagan [:djf]]

I just noticed another potential issue. It looks like the gecko image changes cause decoded images to be discarded when an app goes to the background. So if I zoom in on an image and then sleep the phone and then come back it takes 2-3 seconds to redisplay the image, presumably because it is being decoded again. This is with today's nightly which has some known bugs, so maybe the situation will improve.

But if it doesn't improve, then we'll probably want to revert to the preview image (undoing any zoom) when the app is hidden. That will allow us to at least display something quickly when we become visible again.

Comment 10

[David Flanagan [:djf]]

And here's another problem with gecko's new image behavior:

The gallery app allows the user to swipe left and right to browse through their images. In order to make this work well, I use three "MediaFrame" objects to display the current image, the previous image and the next image. The current image is on screen. The previous image is offscreen to the left and the next image is offscreen to the right.  The next and previous images need to be decoded in adavnce so that as the user begins to swipe left or right, they see the next or previous image already decoded.

I used to implement MediaFrame with an img element. But then gecko got smart about releasing decoded image memory when it was offscreen, and so my image swiping didn't work anymore... as the user swiped, they'd see an empty rectangle moving on to the screen, and then they'd see the new image appear. It wasn't smooth anymore.

So I switched MediaFrame to use background-image, because gecko wasn't smart enough to know when a background image could be released, and it worked fine.

Now gecko has changed again, and swiping between images is broken again.

This issue is a little tricky to verify because Gallery has a lot of optimizations for photographs with EXIF previews.  To reproduce this:

1) Take a photo
2) Use gallery to edit the photo and save the edited version. Don't crop it: keep it a big photo.
3) Repeat so that you have 5 edited photos. None of these will have EXIF previews.
4) Tap on the first edited photo, then swipe to see the next and the next and the next

You'll see that there is a noticeable delay before the edited photos appear on the screen, and in particular you'll see that they are not decoded when you first begin the swipe.

If, after swiping to a new photo you quickly swipe back to the previously displayed one, you'll see that it displays right away, because gecko hasn't discarded it yet.  But if you wait before swiping back, then that will take some time to appear as well.

So I'm going to need some way to force gecko to decode an image before it is displayed on screen, and keep that decoded image ready to display instantly.

Comment 11

[David Flanagan [:djf]]

Needinfo Seth for help with the issues described in comments 9 and 10.

Seth: I'm sorry I didn't build gecko and start trying out your patches weeks ago. This seems like it is going to be a lot more complicated than I expected.

With the number of changes that are going to be required for Gallery, I don't think either decode-on-demand or downsample-during-decode will be appropriate to enable in 2.2.

Comment 12

[Seth Fowler [:seth] [:s2h]]

(In reply to David Flanagan [:djf] from comment #9)
> I just noticed another potential issue. It looks like the gecko image
> changes cause decoded images to be discarded when an app goes to the
> background.

That's nothing new actually. Should be unrelated to downscale-during-decode, but of course this is stuff is new and has bugs, so who knows.

> But if it doesn't improve, then we'll probably want to revert to the preview
> image (undoing any zoom) when the app is hidden. That will allow us to at
> least display something quickly when we become visible again.

That seems reasonable to me.

Comment 13

[Seth Fowler [:seth] [:s2h]]

(In reply to David Flanagan [:djf] from comment #10)
> So I'm going to need some way to force gecko to decode an image before it is
> displayed on screen, and keep that decoded image ready to display instantly.

Yep, that is definitely necessary. This is actually exactly the sort of issue I wanted to expose by having you update the Gallery app. We need to have some combination of good heuristics and a way to explicitly request decoding here.

I need to look into this a little bit to see what might be appropriate, but we'll get something in place. (One possibility is using "will-change" as a hint, but I need to consult with others about this to make sure this won't have unintended consequences elsewhere.)

Comment 14

[Seth Fowler [:seth] [:s2h]]

(In reply to David Flanagan [:djf] from comment #11)
> With the number of changes that are going to be required for Gallery, I
> don't think either decode-on-demand or downsample-during-decode will be
> appropriate to enable in 2.2.

I'm starting to feel that way as well. =\

Comment 15

[Seth Fowler [:seth] [:s2h]]

After further consideration, we may be able to avoid the need for any hint here just by considering images just outside the viewport horizontally as visible. (Apparently we already do so if they're just outside the viewport vertically.)

Comment 16

[David Flanagan [:djf]]

Ignoring the issues in comments 9 and 10 for now, here is a summary of
the changes that need to be made to address this bug and take
advantage of gecko's new downsample-during-decode feature. I'm writing
this down both to get it clear in my own head, and so that Seth will
know what Gallery is doing with images.

First, the code that displays images and handles zooming for Gallery
is shared/js/media/media_frame.js

When we display an image, we almost always start off displaying a
preview image instead of the full-size image so that the image is
displayed more quickly. Most of the time the user will not zoom in on
the image, and this is a great optimization. Preview images come from
EXIF metadata when it exists and when that image is at least as big as
the screen. For JPEG images that do not have an EXIF preview (or that
don't have a big enough preview) we use the original image but add a
#-moz-samplesize media fragment to force it to be decoded at a smaller
size. For large non-JPEG images, we manually create a preview image
when we first scan the image and save it on the sdcard in a hidden
.gallery/previews/ directory. The only time we do not display a
preview image is if the full-size image is less than half a megapixel.

Let's assume that we've got a 6mp image that is 2048px wide and 3072px
high, and that it has as 320x480 exif preview. The way MediaFrame
currently works is it starts off displaying a preview image with
something like this:

  background-image: url(previewurl)
  width: 320px;
  height: 480px;

If the user zooms in, we switch to displaying the full-size image. To
prevent a flash as we transition from one image to another, we do
this:

  background-image: url(fullsizeurl), url(previewurl)

So while the new image is being decoded, gecko keeps displaying the old image.

In addition to changing the background-image, when the user zooms, we
also have to change the size of the image obviously. As an example,
suppose that the user zoomed in by a factor of 2. We'd end up with CSS
like this:

  background-image: url(fullsizeurl), url(previewurl)
  width: 2048px;
  height: 3072px;
  transform: scale(.3125) translate(some amount for panning)

The full size image is 2048px wide, but the scale of .3125 means that
we display it 640px wide, which is the zoom factor of 2. When we first
make this change, the fullsize image isn't displayed yet, so the
preview image gets resized (and pixelated) to be 640px wide instead of
320.

After the initial zoom, the background-image never changes. If the
user zooms some more we just change the scale, and if the user pans we
change the translate.

So that's how it works now.  For performance reasons, I think I'm
going to have to continue to use EXIF previews when they're
available. And until downsample-during-decode works for non-JPEG
images, I'm going to have to continue to generate external previews
for large non-JPEG images for the same performance reasons. But in the
case where I have a JPEG image without an EXIF preview, I need to
change this code so that it does not use #-moz-samplesize and just
starts out with the full-size image scaled down to the size of hte
screen. (I suppose that it might be simplest to actually use the full image
size and the scale() right from the start even with a preview image so
that the only thing that needs to change on the first zoom is the
background-image)

It is actually a little more complicated. For *really* big JPEGs, we
want to be sure not to decode them at competely full size. In that
case, the current code uses #-moz-samplesize for the preview and for
the full size image.  Then change to make here is to set the full size
to something smaller than the real full size.

We want the pinch-to-zoom gesture to be really smooth, so ideally we
won't need to trigger multiple decodes. If we start off with the image
displayed at 1/8th size and then start zooming in, it seems bad if we
end up decoding at 1/4 size then half size then full-size. It might be
nice if there was some way to hint to gecko that it should just go
ahead and decode it at full size when the user begins to zoom in.

Could I do that with an off-screen image with its size set to the
desired size? Or would that cause gecko to hold on to two decoded
copies of the image?

Comment 17

[Seth Fowler [:seth] [:s2h]]

Thanks for the detailed explanation! This is very helpful.


(In reply to David Flanagan [:djf] from comment #16)
> And until downsample-during-decode works for non-JPEG
> images, I'm going to have to continue to generate external previews
> for large non-JPEG images for the same performance reasons.

That should be pretty soon, but I want to make sure everything works well with JPEG before flipping it on for other formats.

> But in the
> case where I have a JPEG image without an EXIF preview, I need to
> change this code so that it does not use #-moz-samplesize and just
> starts out with the full-size image scaled down to the size of hte
> screen. (I suppose that it might be simplest to actually use the full image
> size and the scale() right from the start even with a preview image so
> that the only thing that needs to change on the first zoom is the
> background-image)

For the case where you don't have an EXIF preview, you really just need to change the scale.

> It is actually a little more complicated. For *really* big JPEGs, we
> want to be sure not to decode them at competely full size. In that
> case, the current code uses #-moz-samplesize for the preview and for
> the full size image.  Then change to make here is to set the full size
> to something smaller than the real full size.

Yep. Just limit how large you're willing to scale it. (Note that *in theory* you may not have to do that, as attempts to allocate surfaces that are too large will simply fail and we'll continue to scale up a previously-decoded smaller version, but I foresee some edge cases here that might lead to a bad user experience. Eventually the platform could become smart enough to avoid the issue, though.)

> We want the pinch-to-zoom gesture to be really smooth, so ideally we
> won't need to trigger multiple decodes. If we start off with the image
> displayed at 1/8th size and then start zooming in, it seems bad if we
> end up decoding at 1/4 size then half size then full-size. It might be
> nice if there was some way to hint to gecko that it should just go
> ahead and decode it at full size when the user begins to zoom in.

I'd prefer to handle such issues automatically in the platform if possible. This stuff is happening for all web content, not just the Gallery app, and we need to provide a good experience everywhere.

> Could I do that with an off-screen image with its size set to the
> desired size? Or would that cause gecko to hold on to two decoded
> copies of the image?

Right now there's no way that doesn't have downsides.

I think in the short term, if we can't get acceptable behavior with heuristics, we'll probably offer control via 'will-change' or something akin to that. But as I mentioned before, we can't expect all web content to start using properties like that, so we need to make sure that the default behavior is as good as possible.

Comment 18

[David Flanagan [:djf]]

Time for me to get going on this, so that Seth has something to test his gecko changes against.

Comment 19

[GitHub Autolander Bot]

Attachment: [gaia] davidflanagan:bug1140619 > mozilla-b2g:master

Comment 20

[David Flanagan [:djf]]

Attachment: work-in-progress patch for Seth to test with

Seth: if you pull this PR into your own branch of Gaia, you'll get a version of the Gallery app that no longer uses #moz-samplesize to downsample jpegs.

When I test this with a gecko nightly that is a couple of days old, I see two main performance issues:

1) When I take photos with the camera and then edit them (which produces JPEG images with no EXIF preview) then flicking from one image to the next causes a noticeable delay before the image is displayed. This is presumably because gecko is not decoding the image until it comes on screen. I know you have a plan to deal with this for image that are near to the screen.  But let me know if you need me to set will-change or something.

2) For images that do have an EXIF preview, pinching to zoom in is relatively responsive, but sometimes causes flashes and tiling issues where a chunk of the image will be missing for a few seconds. Sometimes the entire screen just goes blank for a few seconds.

I did this testing on a flame with 319mb of memory.

Comment 21

[David Flanagan [:djf]]

Seth suggests that it would also be good to switch back to <img> instead of background-image:

[1:28pm] seth: djf: i recall you mentioned before that MediaFrame used to use <img> elements instead of CSS background images
[1:28pm] seth: djf: do you think there are any barriers to switching back to that approach?
[1:43pm] djf: seth: was in a meeting, sorry.  For MediaFrame, I'm still going to be displaying photo EXIF previews, when those are available and switching to the full-size image when the user zooms in.  Using background-image makes the switch really easy because I can just specify two images and the small one will be displayed until the big one is decoded.  When I was using img tags, I just had to guess at how long it would take to decode the full-size image and use a timeout.  So unless you can think of a way to switch from one image to the other without a flash, I'd like to keep using background-image.
[1:44pm] seth: djf: so here's the thing
[1:44pm] seth: djf: i've been investigating the severe "pop in" that the Gallery app suffers from
[1:45pm] seth: djf: and beyond the thing we talked about yesterday about blobs, the biggest issue is that CSS background images are basically decode-on-draw
[1:46pm] seth: djf: that's true regardless of downscale-during-decode or any of the other work i'm doing. getting a good user experience while swiping between images in the gallery is really hard, because we don't even start decoding them until the swipe has already started
[1:47pm] seth: djf: i do have plans to fix this, but it a whole 'nother project. for content images, by contrast, we are currently much more intelligent about decoding them when they're near the viewport but not yet visible
[1:48pm] seth: djf: i haven't tried this, but what happens if you display the full-size image in an <img> element, and set background-image on the <img> element to the EXIF preview?
[1:49pm] seth: djf: i think that would work just fine, and it'd perform *much* better with gecko as it is today
[1:49pm] djf: seth: I've got no idea what happens if you set background-image on an img element. That might be perfect.  Thanks for that idea.
[1:51pm] djf: seth: the thing that motivated the switch from img to background-image was that gecko got too smart about not decoding off-screen img elements, and we had performance issues. But gecko was not smart enough to do that with background-image, so I switched to that.  Now you're telling me that gecko's intellegence has advanced to a new level and I really ought to switch back.   This does not affect 2.2, I assume?
[1:55pm] seth: djf: see, that's what i don't understand
[1:55pm] seth: djf: background-image has been decode-on-draw for years
[1:55pm] seth: djf: i suspect that you may have been hitting a bug that happened to do what you wanted
[1:56pm] seth: djf: perhaps we weren't as smart about culling stuff that had a transform on it at one point
[1:56pm] djf: seth: IIRC, I had specific advice from someone on the graphics team that I should use it.
[1:57pm] djf: seth: right, the issue was that it wasn't good about knowing whether it was onscreen or not...
[1:58pm] djf: seth: but around the same time I switched from using background-image to img for displaying thumbnails because gecko got better about knowing when the thumbnails needed to be decoded and when the image memory could be freed.
[1:58pm] seth: djf: yeah, there has been tons of churn in this code and the optimal approach keeps changing 
[1:59pm] seth: djf: i'm trying to get us to the point where it doesn't matter and things work well no matter what you do, but we're not there yet, unfortunately
[1:59pm] seth: djf: i am actually a bit concerned now, though, that maybe the truth is that *nothing* works well for the gallery use-case right now
[2:00pm] seth: djf: i am sure that background-image does not work well, but i haven't tried <img> locally here
[2:00pm] djf: seth: as long as it doesn't affect 2.2, we've got time to get it sorted out!
[2:00pm] seth: djf: hopefully in the time since you switched, we've become more intelligent about detecting when transformed elements are near the viewport

Comment 22

[Seth Fowler [:seth] [:s2h]]

Just noticed I never uploaded my experiment version of media_frame.js that uses <img> elements. You can find a copy here:

https://github.com/sethfowler/gaia/tree/use-img-elements-for-mediaframe

Comment 24

[No-Jun Park [:njpark]]

To Qanalysts: once this is resolved fixed, could you also check whether Bug 1168168 (in 512MB config) is fixed as well?  Thanks!

Comment 25

[David Flanagan [:djf]]

I've updated the patch attached to this bug to rebase it now that we're using <img> instead of background-image in media frame.

It looks like this is not going to be easy to remove #-moz-samplesize. It looks to me like there are still platform bugs in downsample-during-decode and I'll have to work with Seth on that. In particular, the images in gallery/test/images/ will not all display with this patch. The 64mp jpeg is one that will not display, so I think it is not actually being decoded at the size it is displayed at.  Even on a 1024mb flame, I have trouble here. Also, when panning through 5 megapixel jpegs without EXIF previews, it sometimes takes way too long to display them, even with plenty of ram. I don't know if decodes are just slow, or if the images off to the left and right of the screen are not getting decoded in advance.

Also it is not clear that the flashing when zooming is caused by #-moz-samplesize. So it could be that my plan to remove moz-samplesize will not actually fix this particular bug.

Now that I can decode at any size, I'm always trying to decode images right at the memory limit. With #-moz-samplesize we always reduced them by at least 1/2, so the decode size was often comfortably below the memory limit. So it could be that I just have to set lower limits in computeMaxDecodeSize() to get back to the performance I see with moz-samplesize.

Comment 26

[David Flanagan [:djf]]

Actually, it looks like sometimes it is sometimes taking almost a second to get the onload event for the 5mp images (1.1mb) jpeg. The image is loading while offscreen, and seems to be decoding off screen, but I can easily get ahead of it because of the long time to the load event...

Comment 27

[David Flanagan [:djf]]

I have not actually been testing with normal photos from the flame, and I should probably do that as well.

Comment 28

[David Flanagan [:djf]]

I can probably tell from an about:memory report what size the images are actually being decoded at. Seth has said that DDD takes the CSS transform into account, but I should verify that it is actually happening in Gallery. If the CSS transform isn't being accounted for, then this patch means that gecko will be decoding those 5mp no-preview jpegs at full 5mp size, and that would be bad.

Comment 29

[David Flanagan [:djf]]

I guess this is all premature. about:memory reports show that DDD is not working, either in Nightly or in yesterday's nightly build for Flame.  The image.downsample-during-decode.enabled pref is set to true in both cases, but images are being decoded at their full size regardless of the size they are displayed at.

I obviously can't remove #-moz-samplesize until gecko is doing the downsampling on its own, so I'm going to stop working on this for now.

Needinfo Seth to find out what's going on here... Had DDD been backed out?

Comment 30

[David Flanagan [:djf]]

Looking at this again, I see that the image is apparently decoded full size and then again at the displayed size. Is this expected, or is something going wrong? What do I need to do in gaia to decode this image only at the small size?

│  ├──21.68 MB (32.20%) -- image(1968x2624, blob:app://gallery.gaiamobile.org/7637621f-c6ca-4e2b-a051-0170de21f9bd)
│  │  ├──19.70 MB (29.25%) ++ unlocked/surface(1968x2624@0)
│  │  ├───1.17 MB (01.74%) -- locked/surface(480x640@0)
│  │  │   ├──1.17 MB (01.74%) ── decoded-nonheap
│  │  │   └──0.00 MB (00.00%) ── decoded-heap
│  │  └───0.81 MB (01.21%) ── source

Comment 31

[David Flanagan [:djf]]

I've just checked again in Firefox (38) (with the ddd pref turned on), Firefox Developer Edition (40) and Nightly (41) and DDD does not seem to work in any of them.  I'm loading this HTML file:

<style>
img {
  transform-origin: top left;
  transform: scale(.5);
}
</style>
<img width=1024 height=1024 src="64mp.jpg">

and then doing an about:memory report.  The 64mp.jpg image is a 8192x8192 pixel jpeg.

about:memory shows that 256mb is used to decode this image at full-size. Then, there is another allocation for the 512x512 pixel surface to display it on the screen. If the browser window is on an external monitor, this takes 1mb of memory. If the brower window is on the retina display of my macbook (2 device pixels per CSS pixel) then the surface takes 4mb instead.

Comment 32

[David Flanagan [:djf]]

Here's the relevant excerpt for the about:memory report described above, using developer edition

│    │  │  ├──257.58 MB (25.06%) -- image(8192x8192, file:///tmp/ddd/64mp.jpg)
│    │  │  │  ├──257.00 MB (25.00%) -- unlocked
│    │  │  │  │  ├──256.00 MB (24.90%) -- surface(8192x8192@0)
│    │  │  │  │  │  ├──256.00 MB (24.90%) ── decoded-nonheap
│    │  │  │  │  │  └────0.00 MB (00.00%) ── decoded-heap
│    │  │  │  │  └────1.00 MB (00.10%) ++ surface(512x512@0)
│    │  │  │  └────0.58 MB (00.06%) ── source

Comment 33

[Seth Fowler [:seth] [:s2h]]

(In reply to David Flanagan [:djf] from comment #30)
> Looking at this again, I see that the image is apparently decoded full size
> and then again at the displayed size. Is this expected, or is something
> going wrong? What do I need to do in gaia to decode this image only at the
> small size?

So this is actually a platform-level issue where we decode at full size as well as at the size we display on the screen. It's caused by the fact that the predictive image decoding we do in Gecko right now just decodes at the image's intrinsic size.

Bug 1151359 will fix this. (Well, for <img> elements it will.) I've been trying really hard to push that through, and I'm hopeful that we'll get it landed relatively soon.

Comment 34

[Hema Koka [:hema]]

Please re-test with platform fixes. 

Thanks
Hema

Comment 35

[Pi Wei Cheng [:piwei] (inactive)]

This issue appears to no longer occur on latest. No flickering occurs following the STR. Also checked bug 1168168 and wasn't able to repro either.

However I noticed another bug when doing the STR: when double-tapping to zoom, the zoom action seems to lag. It's not an immediate and smooth zooming action, it zooms and then pauses to think for a little (the 'pause' lasts a fraction of a second, but on Flame 512MB I've seen it lasted for about a second which seems pretty bad), and then it finishes zooming. Sometimes the image appears to shift very little after it finishes zooming. I'll look into bugging this separately.

Setting bug as fixed since it is no longer occurring.

Original issue does NOT occur on:

Device: Aries 2.5
BuildID: 20150902124951
Gaia: e2fab8f6ac345ecde10a1350e699be9ceb6987d6
Gecko: 1b687fcb5213153855c7ac0f8392ce0a4a7e3382
Gonk: 2916e2368074b5383c80bf5a0fba3fc83ba310bd
Version: 43.0a1 (2.5) 
Firmware Version: D5803_23.1.A.1.28_NCB.ftf
User Agent: Mozilla/5.0 (Mobile; rv:43.0) Gecko/43.0 Firefox/43.0

Device: Flame 2.5 (319/512MB)
BuildID: 20150902030203
Gaia: b75979ec8862bd5799a7c42e938d3f67be38d6ae
Gecko: fb720c90eb49590ba55bf52a8a4826ffff9f528b
Gonk: c4779d6da0f85894b1f78f0351b43f2949e8decd
Version: 43.0a1 (2.5) 
Firmware Version: v18Dv4
User Agent: Mozilla/5.0 (Mobile; rv:43.0) Gecko/43.0 Firefox/43.0

Comment 36

[Pi Wei Cheng [:piwei] (inactive)]

Setting as verified fixed since there was a verifyme tag.

Comment 37

[Pi Wei Cheng [:piwei] (inactive)]

Filed bug 1201322 for the issue observed at comment 35.