474211 - [1.9.0]XMLHttpRequest binary data download problem

Status: RESOLVED FIXED

(Core :: DOM: Core & HTML, defect)

Description

[Alexander]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.20) Gecko/20081217 AdCentriaIM/1.7 Firefox/2.0.0.20
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5

var req = new XMLHttpRequest();  
        var mode = false
        req.open('GET', url, mode);  
        req.overrideMimeType('text/plain; charset=x-user-defined');  
        req.onreadystatechange = function() {
            if(req.readyState == 3) {
                alert(req.responseText.charCodeAt(0) & 0xff);

            }
        }
        req.send(null);
        if(!mode){
            alert('NSYNC');
            alert(req.responseText.charCodeAt(0) & 0xff);
        }    

Url may points to any binary file with first byte > ANSI range
If var mode = false this function dispays the same correct result in Firefox
2.0.0.20 and Firefox 3.0.5.
If var mode = true Firefox 2 works in the same correct way when Firefox 3
dispayes incorrect byte code.

Reproducible: Always

Steps to Reproduce:
1. create test file (with non ANSI character at 0 position) reneme it to file.bin for example.
2. Change url varibale in script to your binary test ffile location and run script from HTML or from XUL.
3. in case of 'var mode = true.' in Firefox 3.0.5 incorrect byte code will be dispayed.



I thing that req.overrideMimeType() does not take any affect.

Comment 1

[Matthias Versen [:Matti]]

Firefox doesn't do any rendering, networking...

Comment 2

[Boris Zbarsky [:bzbarsky]]

Attachment: Text file with a single ISO-8859-1-encoded non-breaking space in it

Comment 3

[Boris Zbarsky [:bzbarsky]]

Attachment: HTML testcase with mode == false (sync request)

Comment 4

[Boris Zbarsky [:bzbarsky]]

Attachment: HTML testcase with mode == true (async request)

Comment 5

[Boris Zbarsky [:bzbarsky]]

Attachment: With mode == false, just output the charcode as hex

Comment 6

[Boris Zbarsky [:bzbarsky]]

Attachment: With mode == true, just output the charcode as hex

Comment 7

[Boris Zbarsky [:bzbarsky]]

Attachment: With mode == true, just output the charcode as hex (correct file)

Comment 8

[Boris Zbarsky [:bzbarsky]]

So with the async request in Firefox 3, the value of charCodeAt(0) is 0xfffd (Unicode replacement character, presumably because we're not decoding this data as ISO-8859-1).  For the sync request it's 0xf7a0 (no clue what the heck that is, even).

In Firefox 2, the sync request shows the same garbage 0xf7a0 value, while the async request shows 0xa0.

Confirming the behavior change.

Comment 9

[Boris Zbarsky [:bzbarsky]]

OK, the change from 0xa0 to 0xfffd happened on trunk between 2008-08-06-02 and 2008-08-07-02.  The pushlog range is:

http://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2008-08-06+02%3A00%3A00&enddate=2008-08-07+02%3A00%3A00

That seems to correspond to the checkin for bug 431701.  Which is odd, since that only changed the codepath when Send() is called on a document...

Comment 10

[Boris Zbarsky [:bzbarsky]]

There is another change on trunk, from 0xfffd to 0xf7a0, so that on trunk both the sync and async testcases produce 0xf7a0.  This happened betwen 2008-10-11-02 and 2008-10-12-02.  The pushlog range:

http://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2008-10-11+02%3A00%3A00&enddate=2008-10-12+02%3A00%3A00

Comment 11

[Boris Zbarsky [:bzbarsky]]

And that last behavior change was due to the check-in for bug 459470.

But now I see what's going on here.  If there is a document that resulted from the XHR we use its charset when creating the responseText.  If not, we use the charset set on the channel (in this case by the override type).

Sicking's change in bug 31701 made it so that we don't set up the document until after we decide whether to parse (and in this case we decide to not parse since the type is not XML).  So there is no document set up and we use the channel charset.  

Before that change, we set up a document, then dropped it at load end.  But the testcase is looking at the data _before_ the load has completed (readyState == 3, not 4), so at that point we're using the document charset of this document that has nothing to do with reality...  This is a document that just got created with createDocument, basically, so its charset switched from ISO-8859-1 to UTF-8 as a result of bug 431701 on trunk and bug 464958 on branch.

Jonas, would it be reasonable to just backport bug 459470 to the 1.9.0 branch?  There's no problem on the 1.9.1 branch or trunk, nor is there a problem on the 1.9.0 if the page waits for all the data to be in before reading responseText (a good idea anyway, since before then the last char of responseText might be corrupt).

I do think it's worth fixing the behavior regression on 1.9.0, though.

Comment 12

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Yup, that should be doable, and probably a good idea for other reasons too.

Removing blocking1.9.1 nomination since this is 1.9.0 only, right?

Comment 13

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Patch in bug 459470 should have fixed this