1061268 - Wrong photo gets used in some BBC articles

Status: RESOLVED WORKSFORME

(Firefox for Android Graveyard :: General, defect)

Description

[Karen Rudnitski [:kar]]

Attachment: Chrome top stories correct Screenshot_2014-08-30-22-54-05.png

I *know* I have noticed this before, but here's proof. 

When on the m.bbc.co.uk/news site, I start with Top Stories view (the landing page). You see a corresponding photo with the article on that landing page. 

Sometimes when you tap onto an article, the main corresponding photo changes. To that from ANOTHER article. Doesn't happen all the time, but when it does, it's really quite disconcerting (depending on what the article is about and what photo is chosen).

I checked what Chrome did, and they got the right photo. So something is happening in there that shouldn't.

Screenshots attached.

NOTE: although I have logged this to nightly, I know I have seen this happen even a couple of months back but didn't really pinpoint it. So it's not a 'new' problem.

Comment 1

[Karen Rudnitski [:kar]]

Attachment: Chrome correct main photo Screenshot_2014-08-30-22-53-55.png

Comment 2

[Karen Rudnitski [:kar]]

Attachment: Chrome correct in-article photo Screenshot_2014-08-31-03-47-00.png

Comment 3

[Karen Rudnitski [:kar]]

Attachment: FF top stories correct Screenshot_2014-08-30-22-54-14.png

Comment 4

[Karen Rudnitski [:kar]]

Attachment: FF wrong main photo Screenshot_2014-08-30-22-54-25.png

Comment 5

[Karen Rudnitski [:kar]]

Attachment: FF wrong in-article photo Screenshot_2014-08-31-03-46-31.png

Comment 6

[Karen Rudnitski [:kar]]

And now, of course, it looks like the BBC has taken that article off their site. Trying to locate it....

Comment 7

[Karen Rudnitski [:kar]]

Spammy, but I have an example. Funnily enough, I mixed up two real examples into one article. The King article is no longer on the bbc site, but the main corresponding photo is visibly wrong.

The attachments called 'in-article photo' is a different article: http://m.bbc.co.uk/news/magazine-28881335. If you go down in the article, Chrome has a *correct* photo of the philosopher. Firefox doesn't.

Comment 8

[Aaron Train [:aaronmt]]

What does desktop Firefox do when provided a mobile user-agent and same screen size (for anyone to try this out)?

Comment 9

[Mark Finkle (:mfinkle) (use needinfo?)]

Timothy - Anything sound related to image decoding? What could we look for?

Comment 10

[Timothy Nikkel (:tnikkel)]

I've never seen anything image decoding related result in the wrong image being shown. Not sure where to look next. When it happens does it do so consistently? Comment 8 is worth a shot. When it happens taking the src of the wrong image and seeing what it loads might be useful.

Comment 11

[Timothy Nikkel (:tnikkel)]

The specific problem from comment 7 didn't reproduce for me.

Comment 12

[Karen Rudnitski [:kar]]

(In reply to Timothy Nikkel (:tn) from comment #11)
> The specific problem from comment 7 didn't reproduce for me.

You're right - after clearing private data and reloading the page, the correct picture shows up. It's obviously intermittent. And obviously quite real, as I've come across it several times before (but not always noticeable at first depending on what picture it swapped with).

I've seen that in some instances, it takes the next picture 'down' from the Top Stories listing.

Comment 13

[Karen Rudnitski [:kar]]

I really honestly wish I could find a consistent way to surface this bug. But right now, I just catch it when something looks 'off' while I'm reading m.bbc.co.uk/news

Let me know if there's anything I can do to better catch this :/

Comment 14

[aokoye]

Attachment: Desktop Screenshot_2014-09-17 at 1.48.23 PM.png

Comment 15

[aokoye]

Attachment: FF Screenshot_2014-09-17-13-45-58.png

Comment 16

[aokoye]

Attachment: FF Screenshot_2014-09-17-13-46-18.png

Comment 17

[aokoye]

Attachment: Chrome Screenshot_2014-09-17-15-42-11.png

Comment 18

[aokoye]

I was able to reproduce this (or something related perhaps) version 32.0.1 of Firefox for Android on a Samsung Galaxy S5 running Android 4.4.2. Note that this was the first instillation of Firefox on the phone and the first time that it had been used.

1. From your android phone go to http://m.bbc.com/news/
2. Click on "Ebola can 'ruin W African economies'" or copy and paste http://m.bbc.com/news/world-africa-29239604
3. Watch the page load.
4. Go to http://m.bbc.com/news/world-africa-29239604 on Chrome for Android for reference to the layout and the picture or a desktop browser to reference the picture.

What happens:
When I do this I first get the picture that shows up on a desktop browser or Chrome for Android. Within 10 seconds a second picture loads and the layout changes. The picture that loads is from an earlier story at the bottom of that page under "More Africa stories". Additionally the layout changes to the same layout that is seen when you open the same link in Chrome for Android. I attached screenshots from the desktop, Firefox for Android and Chrome. 

Complicating factors:
I was only able to get this to work with one link that was on http://m.bbc.com/news at the time which is probably why so many people were having issues replicating this. It doesn't appear that this is an issue with all Android browsers as the link loads correctly in Chrome.

Comment 19

[Karen Rudnitski [:kar]]

Attachment: Screenshot_2014-10-02-22-21-20.png

Comment 20

[Karen Rudnitski [:kar]]

Seems to be happening more frequently right now (on nightly)

Today's wrong photo is in this article http://m.bbc.co.uk/news/uk-england-29459516

Comment 21

[Karen Rudnitski [:kar]]

Seems to be happening to me on a more frequent and almost daily basis now. Latest URL http://m.bbc.co.uk/news/entertainment-arts-29474422. Screenshot in attachment

Comment 22

[Karen Rudnitski [:kar]]

Attachment: Screenshot_2014-10-03-22-50-29.png

Comment 23

[Richard Newman [:rnewman]]

FWIW, I spoofed the UA on desktop, and tried pretty hard to repro on a slow connection with devtools up, and could not reproduce.

There's a lot going on in this page, though -- they fetch a super small 6KB version of the image, then they swap in a full-res version later with some JS shenanigans. I could imagine that something like the following is happening:

* The main News page is loading, progressively fetching images for subsequent stories.
* Meanwhile, you click through to one story.
* One of these things happens:
  * Some ID/timestamp/whatever for one of the main page progressive fetches collides with the fetch for the large image for the headline, and we either coalesce or hit the cache, using the wrong image.
  * We kill all of the fetches for the progressive images when you click through. Unfortunately, we then reuse a socket (keepalive is on) to fetch the large image, and we end up reading the wrong bytes off the wire.
  * There's a concurrency bug in the network cache.

Things I would suggest, kar:

* Try turning off keepalive (about:config, network.tcp.keepalive.enabled). Remember to turn it back on when you're done testing.

* Try turning off caching. browser.cache.memory.enable, browser.cache.disk.enable. Remember to turn them back on later.

Tweak each of those three settings *individually*, then see if you can repro (for as long as necessary -- if you're on wifi, just leave them this way until you're confident that you would have seen the issue).

Yes, this might take a week, but it'll be enough to determine if we should be pointing fingers at necko, at the cache, or at something higher up the stack.

Comment 24

[Karen Rudnitski [:kar]]

http://m.bbc.co.uk/news/world-europe-29529793

I did the first thing and after a free days, stumbled upon the issue again (address above). Fwiw, I see a flicker of the image so presumably the wrong one overlays the right one.

Will try the other thing now.

Comment 25

[Karen Rudnitski [:kar]]

Second parameter changed and came across the issue today on http://m.bbc.co.uk/news/uk-29559444

Onto the third.

Comment 26

[Karen Rudnitski [:kar]]

Came across it again on http://m.bbc.co.uk/news/uk-england-kent-29582318

The top photo started as the right one but then changed into the wrong picture. Also the photo further down is also wrong.

Back to square one? What else could or should I try?

Comment 27

[Richard Newman [:rnewman]]

If you've turned off the disk cache, the memory cache, and keepalive, I think we can rule out bad caching.

That leaves us with:

* A JS coding error that we're slow enough/fast enough whatever to trigger.
* A JS or network or other coding error in Gecko or Fennec that does the wrong thing.
* A graphics/layer/image handling error.
* A hardware accel issue, if that's relevant for Fennec.

On to the platform folks!

Comment 28

[Karen Rudnitski [:kar]]

FWIW, it happens one at least one article (always near the top of the 'most read' or news landing page (m.bbc.co.uk/news) every time I read the news (at least once a day).

Comment 29

[Brad Lassey [:blassey] (use needinfo?)]

filter on [mass-p5]

Comment 30

[Kevin Brosnan [Ex-Mozilla]]

Attachment: Non-BBC Screenshot_2014-10-18-07-24-48.png

I may have seen this outside the BBC, see attachment. I was using Aurora 35 at the time.

Comment 31

[Karen Rudnitski [:kar]]

Adding my little video clip to show what I'm seeing (apologies for the graininess). Let me know if more examples would help. Still is happening almost every night.

Comment 32

[Karen Rudnitski [:kar]]

Attachment: VID_20140910_153050.mp4

Comment 33

[Aaron Train [:aaronmt]]

Comment on attachment 8512584 [details]
VID_20140910_153050.mp4

wrong video I guess

Comment 34

[Karen Rudnitski [:kar]]

Attachment: CORRECT video demo VID_20141016_203359.mp4

Comment 35

[James Willcox (:snorp) ([e-mail]) (he/him)]

Seth this looks like it could be some sort of imglib cache key collision? Any ideas?

Comment 36

[Seth Fowler [:seth] [:s2h]]

(In reply to James Willcox (:snorp) ([e-mail]) from comment #35)
> Seth this looks like it could be some sort of imglib cache key collision?
> Any ideas?

Doubtful that it's a collision in the ImageLib source data cache because the keys are URIs. Unless these images have the same URI? I don't know about the BBC, but in the case of Kevin's example I doubt that's true.

It also seems to be unrelated to the ImageLib SurfaceCache (for decoded data) since the bug existed before we used that for anything but SVGs.

I don't see any other obvious ways that ImageLib could cause this, though maybe I'm missing something. Perhaps the issue is in the graphics system. That seems plausible since we can't reproduce it on desktop. It could be an issue with volatile buffers (Android is one of only two platforms we use those on, IIRC) or  an issue with how surfaces are managed.

Comment 37

[Seth Fowler [:seth] [:s2h]]

If someone wants to be absolutely certain that the SurfaceCache is not involved, setting 'image.high_quality_downscaling.enabled' to false would be enough to check, since right now we store HQ scaled frames in the SurfaceCache but not normal frames.

Comment 38

[Timothy Nikkel (:tnikkel)]

(In reply to Seth Fowler [:seth] from comment #37)
> If someone wants to be absolutely certain that the SurfaceCache is not
> involved, setting 'image.high_quality_downscaling.enabled' to false would be
> enough to check, since right now we store HQ scaled frames in the
> SurfaceCache but not normal frames.

That pref should be false by default on android. See http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/mobile.js#69

Comment 39

[Timothy Nikkel (:tnikkel)]

I made a try build that should always by-pass the image cache (so always fetching the image from the network even if we've fetched it before). This would rule out (or rule in) the image cache. Karen could you give it a try? Maybe don't use it over a non-wifi connection for too long if you don't have that much in your data plan.

https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-34017b50d28e/

Comment 40

[Karen Rudnitski [:kar]]

fyi, I loaded the try build 2 days ago and still putting it through its paces. No instance of a swapped photo yet.

Comment 41

[Timothy Nikkel (:tnikkel)]

Thanks for testing. Maybe my hunch was on to something. Keep us posted when you feel confident that the try build definitely doesn't have the bug OR definitely does.

Comment 42

[Karen Rudnitski [:kar]]

using the build, I came across the wrong photo being used in an article. It was deeper down in the article, about the 3rd one down, and that same (wrong) photo was used in another photo spot right under it. It was the low-res version from another story.

I don't know if this matters in trouble shooting, but I can get it to happen here on my home wifi network fairly consistently (although when a new build is tried, it takes a few days before the bug starts to surface). But I haven't yet been able to make it happen in the Mozilla London office wifi. Maybe because it hasn't taken a few days to kick in, but I haven't been able to reproduce it there.

Comment 43

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Karen Rudnitski [:kar] from comment #42)

But you have reproduced it on the office wifi before, right? If it only happens at home, I'm inclined to think your ISP has some type of proxy that is screwing up. You aren't using Janus are you?

Comment 44

[Karen Rudnitski [:kar]]

I'm not using Janus, no. 

What I'll do is switch off wifi and browser over my network data connection next. I haven't been able to spend a ton of time trying to reproduce this at the office yet - maybe I should go in just to turn the network ;-)

Comment 45

[Karen Rudnitski [:kar]]

To keep you updated - I force reloaded the page when on my data network, and the wrong image showed up. I have since cleared history and cache, and will continue browsing over data to see if that has an impact.

Comment 46

[Mark Finkle (:mfinkle) (use needinfo?)]

Not tracking 36

Comment 47

[Jason Duell]

> I force reloaded the page

Does Fennec support shift-reload? I seem to remember (as of a few years ago) that we only provided UI for regular reloads.

Also, Re: comment 12: When you clear private data, I assume you clear both cookies and cache?  And until you do that, you can reload the page and you still get the wrong image, but after clearing the cache you see the right one?

Honza: is the info in comment 23 (disabling browser.cache.memory.enable and browser.cache.disk.enable) the best way to turn off the cache?  It sounds like we've ruled out the cache here since Karen sees the bug even with those two prefs disables (so we shouldn't be caching), but then it's weird that she can clear the cache to make the behavior go away.

Comment 48

[Honza Bambas (:mayhemer)]

This sounds a bit to me like symptoms similar to (one of the duplicates of) a bug we've fixed on 35 - bug 1066726.

Can you reproduce on 35+ ?  If not, I vote for duplicating it.

Comment 49

[Honza Bambas (:mayhemer)]

Ups... it has been backported to 33, on what version are you?  (haven't read the comments in detail.)

Comment 50

[Karen Rudnitski [:kar]]

I use nightly, so right now that would be 38. And I reported this for nightly since I first reported the bug, so riding the latest releases since then (which would be whatever nightly was come Sept 1)

Comment 51

[Kevin Brosnan [Ex-Mozilla]]

The nightly of Sept-01 was 34 the nightly of Sept-02 was 35. It is not clear from your comment, you continue to be able to reproduce this issue in Firefox nightly?

Comment 52

[Karen Rudnitski [:kar]]

Sorry Kevin - you're right, I wasn't clear!

I was able to reproduce on nightly 1.5 weeks ago. I switched over to my iOS device last week but will try again on my Android device today and tomorrow to ensure I can, indeed, still reproduce if we think bugs have landed in the past 1.5 weeks.

Comment 53

[Mark Finkle (:mfinkle) (use needinfo?)]

(In reply to Jason Duell [:jduell] (needinfo? me) from comment #47)
> > I force reloaded the page
> 
> Does Fennec support shift-reload? I seem to remember (as of a few years ago)
> that we only provided UI for regular reloads.

Fennec reloads always bypass the cache:
http://mxr.mozilla.org/mozilla-central/source/mobile/android/chrome/content/browser.js#1610

Comment 54

[Karen Rudnitski [:kar]]

Just went to m.bbc.co.uk/news and tapped on the Taiwan plane crash story and two photos mid-way through the article are of the US mid-term elections from a few months back. 

I will say that's a bit different to what was happening before, where it was clearly a photo being replaced by another photo from the same set of Top Story articles. Now it just appears that this one photo pervades my stream.

Comment 55

[Doug Turner (:dougt)]

karen, can you confirm that you're using the latest gecko?  It isn't clear from the comments above.

Comment 56

[Karen Rudnitski [:kar]]

I am always on the latest fennec nightly.

Comment 57

[Doug Turner (:dougt)]

karen, any addons?  any preferences set beyond what's default?  anything changed in about:config?  have you tried with a new profile?


Regarding the site, I see lots of javascript errors when loading in Fennec.  Karen, do you see this sort of problem on any other site?

Comment 58

[Karen Rudnitski [:kar]]

So I just uninstalled nightly and re- downloaded it, so clean profile. First article on the BBC.co.uk/news top story, first photo, is the wrong photo. Will attach.

I never change about:config and obviously I didn't download add-ons nor turned on sync. All clean. Btw, also over an operator network and not over wifi.

Now I say no add-ons, but I checked just on case and something is listed there which I don't understand. 264 codec, but how is that 'by default'?... Also will attach screenshot

Comment 59

[Karen Rudnitski [:kar]]

Attachment: fresh profile, wrong photo

Comment 60

[Karen Rudnitski [:kar]]

Attachment: add-on by default on clean profile

Comment 61

[Karen Rudnitski [:kar]]

Doug - will switch to the telegraph and guardian to see if I come across the issue. Of course, going to the telegraph will now prompt some auto play bugs for AaronMT's pleasure....

Comment 62

[Emanuel Hoogeveen [:ehoogeveen]]

The OpenH264 plugin is automatically downloaded so you can participate in WebRTC calls.

Uninstalling and reinstalling Nightly also won't give you a fresh profile by itself. The profile is stored in %AppData%\Mozilla\Firefox\Profiles\[random stuff].default. If you don't want to entirely nuke it, you can also click "Refresh Nightly..." in about:support, which will remove any add-ons (though you don't seem to have any) and reset your settings, but keep your browsing history and such.

Comment 63

[Honza Bambas (:mayhemer)]

Karen, please backup your profile (both Roaming and *Local*) [1] while you can reproduce the problem.  Also, note that reinstalling Nightly doesn't delete the profile.

[1] http://kb.mozillazine.org/Profile_folder_-_Firefox

Comment 64

[Aaron Train [:aaronmt]]

(In reply to Honza Bambas (:mayhemer) from comment #63)
> Karen, please backup your profile (both Roaming and *Local*) [1] while you
> can reproduce the problem.  Also, note that reinstalling Nightly doesn't
> delete the profile.
> 
> [1] http://kb.mozillazine.org/Profile_folder_-_Firefox

This is mobile we're talking about. Uninstalling org.mozilla.fennec nukes everything.

Comment 65

[Karen Rudnitski [:kar]]

(In reply to Honza Bambas (:mayhemer) from comment #63)
> Karen, please backup your profile (both Roaming and *Local*) [1] while you
> can reproduce the problem.  Also, note that reinstalling Nightly doesn't
> delete the profile.
> 
> [1] http://kb.mozillazine.org/Profile_folder_-_Firefox

Given this is a problem on fennec, I presume you don't need me to do the above?

Comment 66

[Honza Bambas (:mayhemer)]

(In reply to Karen Rudnitski [:kar] from comment #65)
> (In reply to Honza Bambas (:mayhemer) from comment #63)
> > Karen, please backup your profile (both Roaming and *Local*) [1] while you
> > can reproduce the problem.  Also, note that reinstalling Nightly doesn't
> > delete the profile.
> > 
> > [1] http://kb.mozillazine.org/Profile_folder_-_Firefox
> 
> Given this is a problem on fennec, I presume you don't need me to do the
> above?

There is a profile as well on fennec, just harder to find/backup.

Comment 67

[Richard Newman [:rnewman]]

Karen: you can use Margaret's "Copy Profile" add-on to copy it to your SD card.

Comment 68

[Honza Bambas (:mayhemer)]

It copies just the "roaming" part.  I'm particularly interested in the cache folder (on desktop under the "local" part), which is on Android in the /cache/<profile-name>/cache2 dir.

Comment 69

[Karen Rudnitski [:kar]]

I've been monitoring the past 2 weeks, and I haven't come across the issue in a while. Not sure if *anything* has changed, on our side or even the BBC's, but I'm going to resolve this for now and will re-open if it comes up again.

Comment 70

[Chris Peterson [:cpeterson]]

Removing steps-wanted keyword because this bug has been resolved.