1309682 - "Progressive" loading of pages no longer works

Status: UNCONFIRMED

(Core :: DOM: HTML Parser, defect, P3)

Description

[Alain Knaff]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0
Build ID: 20160921204512
Firefox for Android

Steps to reproduce:

1. Set up a CGI script on my web server to display a page with some wait time in the middle

#!/bin/sh

/bin/echo Content-Type: text/plain
/bin/echo
/bin/echo test
sleep 3
/bin/echo ing

2. Disabled mod_deflate on my web server to make sure that the server itself is not holding back/delaying the data.

3. Verified that no buffering is happening inside the server using curl

curl http://localhost/line-by-line-test.cgi

3. Accesses the page using Firefox


Actual results:

- curl showed the line "test", and then, 3 seconds later the line "ing", proving that the web server delivers the result correctly

- Firefox waited for three seconds, and then displayed the two lines together
- Replacing this with an script generating equivalent HTML  (Content-Type: text/html, test<p>, ing<p>) didn't work any better.

Problem happens both on desktop Firefox and on Android Firefox.


Expected results:

- Firefox should first show "test", and then 3 seconds later add "ing"

Curl isn't the only browser doing this correctly. Chrome and Operate also do it right (first show "test" and then "ing"

Comment 1

[Tooru Fujisawa [:arai]]

by "no longer works", do you mean it was working before?
if so, which version do you refer?

I see the actual result on Firefox, Chrome, and Safari.
do you see expected result on any other browser?

perhaps the testcase doesn't match to the actual code?
I guess, the output of the testcase is too short,
and by changing the script to output longer string (more than 1024 bytes), it works as expected.

Comment 2

[Alain Knaff]

(In reply to Tooru Fujisawa [:arai] from comment #1)
> by "no longer works", do you mean it was working before?
> if so, which version do you refer?

Indeed, it used to work, but unfortunately I can't remember which version worked.

> 
> I see the actual result on Firefox, Chrome, and Safari.
> do you see expected result on any other browser?

That's 2 others I tested. Well 3 if you include curl.

> 
> perhaps the testcase doesn't match to the actual code?

Huh? What is _that_ supposed to mean?

> I guess, the output of the testcase is too short,


> and by changing the script to output longer string (more than 1024 bytes),
> it works as expected.

But I expect it to work as is. 
You could just as well say that the sleep is blocking the output? Well, guess what this *is* the actual bug. Firefox should display the data within a reasonable amount of time of arriving, even if it's only one sentence. Padding should not be needed to force an update. (Yes, padding does actually work around the issue, but why should this be needed? Chrome and Opera don't need it either.)

Comment 3

[Tooru Fujisawa [:arai]]

sorry, I checked it again and I confirmed it was working as expected on Firefox 4.0b9pre on linux,
and doesn't work on Firefox 16.0a1.
(most old builds are crashing and it's hard to bisect between them...)

anyway, confirmed this is a regression.


(In reply to Alain Knaff from comment #2)
> > perhaps the testcase doesn't match to the actual code?
> 
> Huh? What is _that_ supposed to mean?

I meant, if it was working as expected until not so long ago, and the testcase is different than actual code you're using on production,
the testcase might be hitting different issue.

Comment 4

[Tooru Fujisawa [:arai]]

here's regression range:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=978002c0b0ad&tochange=b8dd6f6f4207

Comment 5

[Tooru Fujisawa [:arai]]

bug 479959 might be related

Comment 6

[Alain Knaff]

Found something interesting: if a charset is specified, it works correctly:

#!/bin/sh

/bin/echo 'Content-Type: text/plain; charset=UTF-8'
/bin/echo
/bin/echo test
sleep 3
/bin/echo ing

Comment 7

[Andrew Overholt [:overholt]]

Henri, what do you think of this? It's not a recent regression, that's for sure :)

Comment 8

[Henri Sivonen (:hsivonen)]

In the absence of a BOM or HTTP-level charset, waiting for 1024 bytes is by design for text/html.

I don't recall if text/plain waits for 1024 bytes intentionally, too, in order to allow the Japanese or Cyrillic detector do its thing. Or if that's an accident.

Before we decide on action in this area, we should get clarity on where we want to on the topic of content-based detection. Previously, the direction was towards *less* content-based detection, but recently, Chrome moved towards *more* content-based detection outside the stardardization process. :-(

annevk, have you had a chance to examine what exactly Chrome does regarding content-based detection now and whether we should try to talk them into not doing what they are now doing or if we should follow them?

nsHtml5StreamParser::SniffStreamBytes() is the relevant method.

Comment 9

[Anne (:annevk)]

I have not gotten further than https://github.com/whatwg/encoding/issues/68#issuecomment-247527906 thus far. That's mostly because I've been focusing on other standards. If you think we should figure out something here soon needinfo me again and I'll get on it next week (likely starting by asking Safari and Chrome what they want to do).

Comment 10

[Andrew Overholt [:overholt]]

(In reply to Henri Sivonen (:hsivonen) from comment #8)
> In the absence of a BOM or HTTP-level charset, waiting for 1024 bytes is by
> design for text/html.
> 
> I don't recall if text/plain waits for 1024 bytes intentionally, too, in
> order to allow the Japanese or Cyrillic detector do its thing. Or if that's
> an accident.

I spoke with Henri and he now believes this is intention.

Until we have full understanding of what could/will change at the spec level, let's mark this fix-optional for at least 50 and 51 since this regression has been around for so long.

Comment 11

[Andrew Overholt [:overholt]]

Anne, I'm taking you up on comment 9 :) Can you and/or Henri figure out if this was an intentional change and if it is, then let's WONTFIX it. Thanks.

Comment 12

[Anne (:annevk)]

We've made some progress talking with the Chrome folks after I filed https://bugs.chromium.org/p/chromium/issues/detail?id=691985 because of bug 1338797. I think we convinced them that only sniffing 1024 bytes is good though they have not changed their behavior around that as of yet.

Comment 13

[Andrew Overholt [:overholt]]

Thanks. Given comment 8 and comment 12 I'm going to mark this fix-optional for 53 and 54.