542096 - multipart image/motion jpeg streams flicker and most of image not displayed

Status: RESOLVED FIXED

(Core :: Graphics: ImageLib, defect)

Description

[jlhugo]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6

Since upgrading to Firefox 3.6, my webcam feed, which uses the multipart image or motion jpeg functionality, has most of the image cut off, and what is shown flickers.  Problem happens on Win7, Vista, XP, RHEL 5 and Ubuntu.

Reproducible: Always

Steps to Reproduce:
1. go to http://www.upl.cs.wisc.edu/w/Webcam and watch the webcam stream



Expected Results:  
I expect an entire image to be displayed without flickering, but still updating to show changes (ie live stream).  Basically, the stream should appear as it does on any pre 3.6 version of firefox

Comment 1

[Daniel Kabs, reporting bugs since 2002]

I can confirm this on Firefox 3.6.

Firefox 3.5 and earlier don't exhibit this problem.

Comment 2

[Daniel Kabs, reporting bugs since 2002]

Try this web cam for example:

  http://78.158.111.87/control/faststream.jpg?html&stream=full&fps=25

Works with FF 3.5.7, flickers with FF 3.6

Comment 3

[Daniel Kabs, reporting bugs since 2002]

@all reporter:

  Can you please request blocking1.9.2 ?

@all

We are talking about the Content-Type: multipart/x-mixed-replace

This special kind of content contains multiple parts having the same content type and semantic meaning. Each part invalidates - "replaces" - the previous parts as soon as it is received completely. This is also known as "server push" or "server stream".

In the majority of cases the parts will have document type image/jpeg.

Server Push is often used by IP cameras to simulate a video stream by sending JPEG images consecutively.

The parts of the multipart content are separated by a multipart boundary string. This string is defined in the Content-Type header of the HTTP response (sent from the server to the client).

To define the content type of each "part", there has to be a Content-Type header preceding each document data.

Thus a typical "server stream" may look like:
-----------------------
HTTP/1.0 200 OK
Content-type: multipart/x-mixed-replace;boundary="Boundary-String"

--Boundary-String
Content-Type: image/jpeg

[Data]
--Boundary-String
Content-Type: image/jpeg

[Data]
...
-----------------------

HTTP-Header, Boundary-String, Content-Type and Data are separated by CRLF.

See also RFC 2046. 
See also http://en.wikipedia.org/wiki/Motion_JPEG#M-JPEG_over_HTTP

Comment 4

[Tanner Filip [:tanner]]

I can confirm...
Adding Blocking1.9.2? per request of Comment 3.

Comment 5

[Matt]

Also confirm on Firefox 3.6 but it is fixed on trunk.

Regression range - it regressed and then was fixed on trunk:

works:
2009-08-12-04-mozilla-central
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2a2pre) Gecko/20090812 Minefield/3.6a2pre

broken:
2009-08-13-04-mozilla-central
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2a2pre) Gecko/20090813 Minefield/3.6a2pre

broken:
2009-09-12-04-mozilla-central
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.3a1pre) Gecko/20090912 Minefield/3.7a1pre

works:
2009-09-13-05-mozilla-central
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.3a1pre) Gecko/20090913 Minefield/3.7a1pre

Comment 6

[Ria Klaassen (not reading all bugmail)]

(In reply to comment #5)

That would be:

Regression range: http://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2009-08-12+02%3A00%3A00&enddate=2009-08-13+06%3A00%3A00

"Fixed" range:

http://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2009-09-12+02%3A00%3A00&enddate=2009-09-13+06%3A00%3A00

Comment 8

[Daniel Kabs, reporting bugs since 2002]

Does "regression range" mean that the regression appeared during that day and "fixed range" that the bug disappeared on that day?

If so, bug 435296 (from "fixed range") sounds suspicious as comments in this bug are talking about "flickering". Also bug 435296 comment 30 mentions changes regarding "multipart/x-mixed-replace". However, the context may be completely different.

Comment 9

[Ria Klaassen (not reading all bugmail)]

Yes indeed. The fix was a trunk-only fix, that may be the reason why the bug is still present in Firefox 3.6.

Comment 10

[Mike Beltzner [:beltzner, not reading bugmail]]

Bobby: was this due to some of the work you did during that regression range, perhaps?

Not blocking, but if there's a simple fix that we feel comfortable taking on branch, we should.

Comment 12

[Daniel Kabs, reporting bugs since 2002]

Tried with server sending content length for each image in the stream -
reduced flickering a little bit - it's till flickering at the bottom.

Comment 14

[Daniel Kabs, reporting bugs since 2002]

This bug hurts every webcam user. Thus I recommend people not to upgrade to FF 3.6 until this bug is fixed.

How can I help to resolve this issue?

Comment 15

[Daniel Kabs, reporting bugs since 2002]

It crashes my FF on Mac.

Try http://www.videovalvonta.fi/control/userimage.html in one tab and open another tab showing something else, e.g. this bug. FF does not run longer than a couple of minutes.

Comment 16

[Daniel Kabs, reporting bugs since 2002]

  http://62.40.151.172/control/userimage.html

  ... flickers like crazy

Comment 17

[Daniel Kabs, reporting bugs since 2002]

   http://81.198.187.6/control/userimage.html

   flickers away...

Comment 18

[Daniel Kabs, reporting bugs since 2002]

(In reply to comment #15)
> It crashes my FF on Mac.

This might have been due to bug 547143 on FF 3.6.1 (fixed in 3.6.2).

Comment 19

[Daniel Kabs, reporting bugs since 2002]

It might be fixed in 1.9.3 but as gecko 1.9.2 allegedly will live longer than expected (due to Lorentz) this fix will not hit customers soon. How can we move this forward?

Comment 20

[Daniel Kabs, reporting bugs since 2002]

Obviously the release plan for FF 3.7 has been dropped:

http://www.theregister.co.uk/2010/05/10/firefox_4_dot_o_plan/

Will the former Firefox 3.7 be released as Firefox 3.6.4 and will this solve this bug?

Comment 21

[Joe Drew (not getting mail)]

No - Firefox 3.7 is being renamed Firefox 4.0.

Comment 22

[Daniel Kabs, reporting bugs since 2002]

Please correct me if I am wrong:
- the fix for this bug is a trunk-only fix. 
- This means, it's not in 4.0 (named 3.7 before).
=> the fix won't hit users soon.

If this is correct, how can I help to fix this bug earlier?

Comment 23

[Joe Drew (not getting mail)]

This apparently regressed between Firefox 3.5 and 3.6; it'd be really handy to know what made it worse, so we can know the smallest possible fix for 3.6.x.

Comment 24

[Daniel Kabs, reporting bugs since 2002]

Joe, 
  how can I shrink the regression window given in comment #5?

Comment 25

[Joe Drew (not getting mail)]

Aw crap, I missed that we already had a regression range. Stupid self.

If you've got a building tree of mozilla-central, you can try using hg bisect. Or, if that is gobbledegook to you, you can wait a little while longer until someone does it for you :)

Comment 26

[Daniel Kabs, reporting bugs since 2002]

I am ready to take a learning curve on Mercurial depending on how long "a little while" is. 

Joe, an you guess how long until developer will have time to look into this bug?

Comment 28

[Alice0775 White]

Revert to d8d073092334(Neil Rashbrook — Bug 509242 Don't horizontally compress theme preview images) in local build, then the problem is gone.
so, http://hg.mozilla.org/mozilla-central/rev/707f5ac27ac8 of Bug 507902 causes the problem.

Comment 29

[Boris Zbarsky [:bzbarsky]]

Huh.  Why would that affect things, I wonder...

Comment 30

[Boris Zbarsky [:bzbarsky]]

OK, so the problem is this code from that checkin:

    PRBool haveSize = PR_FALSE;
    PRUint32 imageStatus = 0;
    if (currentRequest)
      currentRequest->GetImageStatus(&imageStatus);
    if (imageStatus & imgIRequest::STATUS_SIZE_AVAILABLE)
      haveSize = PR_TRUE;
....
    if (!imageOK || !haveSize) {
      // No image yet, or image load failed. Draw the alt-text and an icon
      // indicating the status

In particular, imgRequest::OnStartRequest sets mImageStatus to imgIRequest::STATUS_NONE.  This, of course, gets called for every frame of the jpeg.

On m-c, imgRequest::OnStartRequest does _not_ set mImageStatus to STATUS_NONE.  Instead, it carefully removes some imgIRequest status flags for the multipart case but does NOT zero out the status altogether.  Which is why things work on m-c.

Comment 31

[Boris Zbarsky [:bzbarsky]]

Attachment: This seems to be the branch-safe conservative fix

Comment 32

[Joe Drew (not getting mail)]

Comment on attachment 448525 [details] [diff] [review]
This seems to be the branch-safe conservative fix

It's actually probably safe to simply clear those three bits unconditionally, which is what we're going to do on m-c shortly, but I like this fix for branch.

Comment 33

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 448525 [details] [diff] [review]
This seems to be the branch-safe conservative fix

Requesting branch approval for this wanted bug.  This should be a quite safe fix that unbreaks multipart jpeg painting on the branch.  The code change is limited to the multipart case.

Comment 34

[Daniel Kabs, reporting bugs since 2002]

Does approval for 1.9.2.5 mean that the fix will not bei in Firefox 3.6.4 but in Firefox 3.6.5?

Comment 35

[Boris Zbarsky [:bzbarsky]]

If it gets approved, yes (though it looks like the version after 3.6.4 will be 3.6.6, but the key is "the version after 3.6.4").

Comment 37

[Daniel Veditz [:dveditz]]

Comment on attachment 448525 [details] [diff] [review]
This seems to be the branch-safe conservative fix

Unfortunately this release is already "too full" to consider additional non-critical patches, however temptingly simple. Moving to the next release.

Comment 38

[Daniel Kabs, reporting bugs since 2002]

I am glad that we have a patch and I hope this bug will be approved soon as webcam users are already complaining about this problem. However they are attributing its effect to the webcam and not to Firefox.

See this report for example:
http://www.info4security.com/story.asp?sectioncode=17&storycode=4125025

"it only seems to work properly on Internet Explorer (Firefox suffers from an unstable image)."

Comment 39

[Boris Zbarsky [:bzbarsky]]

I do think we should make it a priority to land this on branch once 1.9.2.6 relbranches...

Comment 40

[Daniel Veditz [:dveditz]]

Comment on attachment 448525 [details] [diff] [review]
This seems to be the branch-safe conservative fix

Approved for 1.9.2.6, a=dveditz

Comment 41

[Boris Zbarsky [:bzbarsky]]

Pushed http://hg.mozilla.org/releases/mozilla-1.9.2/rev/80c1ad2db62d

Comment 42

[Daniel Kabs, reporting bugs since 2002]

For everybody waiting - this bug is *not* fixed in Firefox 3.6.6, released June 26th, 2010. Flag "status1.9.2" does reflect this as it has been changed to ".7-fixed"

However, there were no notification mails nor was this changed logged in bug history:
https://bugzilla.mozilla.org/show_activity.cgi?id=542096

Strange.

Comment 43

[Boris Zbarsky [:bzbarsky]]

> However, there were no notification mails nor was this changed logged in bug
> history: https://bugzilla.mozilla.org/show_activity.cgi?id=542096

That's because nothing changed in this bug.  What was changed was the name of the flag value (from ".6-fixed" to ".7-fixed").  Then a new value named ".6-fixed" was added to the database.

Comment 44

[zimbatm]

This bug is not just a "webcam users" bug.

The multipart/x-mixed-replace jpeg streaming, while not optimal, is the only patent-free video format that works across all browser vendors (except IE).

I just wanted you to realize that this is not a small issue. By the way I'm also surprised that nobody from Axis came up to provide some help.

From the last messages, I don't know if it will be fixed in the next release, but I really hope so.

Cheers,
  Jonas

Comment 45

[Daniel Veditz [:dveditz]]

(In reply to comment #44)
> From the last messages, I don't know if it will be fixed in the next release,
> but I really hope so.

You can try the beta release of 3.6.7 if you like. Install the Update Channel Selector, choose the beta channel, and then check for updates.
https://addons.mozilla.org/en-US/firefox/addon/6263/

Or wait two weeks and pick up the release when it's out.

Comment 46

[jason.woods]

Tried the latest 3.6.7 beta and it is much better.
However, when there is motion you can see scanning at the bottom of the image, like it is getting updated in small strips so you can have it where top 90% of the image is real time and the bottom 10% is a frame behind - this can be quite straining on the eye and the image should update fully and not have this lagging strips at the bottom.