844007 - Firefox 19 : cross-domain XmlHttpRequest.responseText in chinese

Status: RESOLVED FIXED

(Core :: DOM: HTML Parser, defect)

Description

[christopherlorent]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0
Build ID: 20130201065344

Steps to reproduce:

My firefox updated to 19 version.
I use cross-domain XmlHttpRequest.


Actual results:

Since last update (firefox 19), my cross-domain XmlHttpRequest.responseText answers with chinese characters instead of XML content :

㰿硭氠癥牳楯渽∱⸰∠敮捯摩湧㴢畴昭ㄶ∿㸍਼㽸浬⵳瑹汥獨敥琠瑹灥㴧瑥硴⽸獬✠桲敦㴧⽯扩砯硳搧㼾ഊ㱯扪慭攽≷慴捨㌢⁨牥昽≨瑴瀺⼯ㄲ㜮〮〮ㄺ㠰㠰⽯扩砯睡瑣桓敲癩捥⽷慴捨㌯∠楳㴢潢楸㩗慴捨∾ഊ†㱲敬瑩浥慭攽≬敡獥∠桲敦㴢汥慳支∠睲楴慢汥㴢瑲略∠癡氽≐吵䴢楮㴢偔こ∠⼾ഊ†㱩湴慭攽≡摤䙲敱略湣礢⁨牥昽≡摤䙲敱略湣礯∠睲楴慢汥㴢瑲略∠癡氽∵〰∠浩渽∰∠⼾ഊ†㱯瀠湡浥㴢慤搢⁨牥昽≡摤⼢⁩渽≯扩砺坡瑣桉渢畴㴢潢楸㩗慴捨併琢 㸍ਠ‼潰慭攽≲敭潶攢⁨牥昽≲敭潶支∠楮㴢潢楸㩗慴捨䥮∠潵琽≯扩砺乩氢 㸍ਠ‼潰慭攽≰潬汃桡湧敳∠桲敦㴢灯汬䍨慮来猯∠楮㴢潢楸㩎楬∠潵琽≯扩砺坡瑣桏畴∠⼾ഊ†㱯瀠湡浥㴢灯汬剥晲敳栢⁨牥昽≰潬汒敦牥獨⼢⁩渽≯扩砺乩氢畴㴢潢楸㩗慴捨併琢 㸍ਠ‼潰慭攽≤敬整攢⁨牥昽≤敬整支∠楮㴢潢楸㩎楬∠潵琽≯扩砺乩氢 㸍ਠ‼潰慭攽≳瑡琢⁨牥昽≳瑡琯∠楮㴢潢楸㩎楬∠潵琽≯扩砺坡瑣桓瑡琢 㸍਼⽯扪


Here is my Wireshark trace :

Output :
Interface id: 0
WTAP_ENCAP: 1
Arrival Time: Feb 20, 2013 16:21:38.702533000 Paris, Madrid
Time shift for this packet: 0.000000000 seconds
Epoch Time: 1361373698.702533000 seconds
Time delta from previous captured frame: 0.000441000 seconds
Time delta from previous displayed frame: 0.000441000 seconds
Time since reference or first frame: 607.009440000 seconds
Frame Number: 8384
Frame Length: 533 bytes (4264 bits)
Capture Length: 533 bytes (4264 bits)
Frame is marked: False
Frame is ignored: False
Protocols in frame: eth:ip:tcp:http
Coloring Rule Name: HTTP
Coloring Rule String: http || tcp.port == 80

Internet Protocol Version 4, Src: 192.168.3.115 (192.168.3.115), Dst: 192.168.3.232 (192.168.3.232)
Transmission Control Protocol, Src Port: 55446 (55446), Dst Port: http-alt (8080), Seq: 1, Ack: 1, Len: 479
Source port: 55446 (55446)
Destination port: http-alt (8080)

Hypertext Transfer Protocol
Expert Info (Chat/Sequence): POST /obix/watchService/make/ HTTP/1.1\r\n
Message: POST /obix/watchService/make/ HTTP/1.1\r\n
Request Method: POST
Request URI: /obix/watchService/make/
Request Version: HTTP/1.1

Host: 192.168.3.232:8080\r\n
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0\r\n
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n
Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3\r\n
Accept-Encoding: gzip, deflate\r\n
Referer: http://localhost:59619/testTrends.htm\r\n
Origin: http://localhost:59619\r\n
Connection: keep-alive\r\n
Pragma: no-cache\r\n
Cache-Control: no-cache\r\n
Full request URI: http://192.168.3.232:8080/obix/watchService/make/



Input :
Interface id: 0
WTAP_ENCAP: 1
Arrival Time: Feb 20, 2013 16:21:38.705858000 Paris, Madrid
Time shift for this packet: 0.000000000 seconds
Epoch Time: 1361373698.705858000 seconds
Time delta from previous captured frame: 0.003325000 seconds
Time delta from previous displayed frame: 0.003325000 seconds
Time since reference or first frame: 607.012765000 seconds
Frame Number: 8385
Frame Length: 1152 bytes (9216 bits)
Capture Length: 1152 bytes (9216 bits)
Frame is marked: False
Frame is ignored: False
Protocols in frame: eth:ip:tcp:http:xml
Coloring Rule Name: HTTP
Coloring Rule String: http || tcp.port == 80

Internet Protocol Version 4, Src: 192.168.3.232 (192.168.3.232), Dst: 192.168.3.115 (192.168.3.115)
Transmission Control Protocol, Src Port: http-alt (8080), Dst Port: 55446 (55446), Seq: 1, Ack: 480, Len: 1098
Source port: http-alt (8080)
Destination port: 55446 (55446)

Hypertext Transfer Protocol
HTTP/1.1 200 OK\r\n
Request Version: HTTP/1.1
Status Code: 200
Response Phrase: OK

Content-Length: 814\r\n
Content-Type: text/xml\r\n
Server: Microsoft-HTTPAPI/1.0\r\n
Access-Control-Allow-Origin: *\r\n
Access-Control-Allow-Methods: POST, GET, PUT\r\n
Access-Control-Max-Age: 1000\r\n
Access-Control-Allow-Headers: Content-Type\r\n
Date: Wed, 20 Feb 2013 15:21:35 GMT\r\n

<?xml version="1.0" encoding="utf-16" ?>
<?xml-stylesheet type='text/xsl' href='/obix/xsd' ?>
<obj name="watch112" href="http://192.168.3.232:8080/obix/watchService/watch112/" is="obix:Watch">
<reltime name="lease" href="lease/" writable="true" val="PT1M" min="PT0S"/>
<int name="addFrequency" href="addFrequency/" writable="true" val="5000" min="0"/>
<op name="add" href="add/" in="obix:WatchIn" out="obix:WatchOut"/>
(...)

I don't put all the return XML, because it has not a big interest, but we see that I have a correct XML answer.


Expected results:

I tried with other browsers and I tried to come back to firefox 18.
In both cases the request works. : I have a correct XML in my responseText.

Comment 1

[mjh563]

> <?xml version="1.0" encoding="utf-16" ?>
                                ^^^^^^
At a guess, I'd say it's caused by that. Is your XML actually encoded in UTF-8?

Comment 2

[mjh563]

This seems to be a regression in Firefox 19. Chrome ignores the stated encoding (utf-16) and uses the actual encoding (utf-8), as do previous versions of Firefox.

Comment 3

[mjh563]

Attachment: utf-8-encoded xml file with utf-16 encoding set in xml header

This testcase works in Firefox 18 but fails with "XML Parsing Error: not well-formed" in Firefox 19.

Last good nightly: 2012-11-06
First bad nightly: 2012-11-07

Pushlog:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=f9c2c266e7aa&tochange=e587aa26326e

Comment 4

[christopherlorent]

(In reply to mjh563 from comment #1)
> > <?xml version="1.0" encoding="utf-16" ?>
>                                 ^^^^^^
> At a guess, I'd say it's caused by that. Is your XML actually encoded in
> UTF-8?

It seems that you're right : the server create a "utf-16" xml that is encoded in "utf-8". So there's a bug in our code, but that was not a problem before firefox 19 (and not a problem in all other browsers : this code is tested with opera, chrome, ie and safari on PC and on smartphones...). 

We'll try to fix it on our side, but I agree that it is probably a firefox regression.

Comment 5

[Scoobidiver (away)]

I suspect bug 804635.

Comment 6

[Boris Zbarsky [:bzbarsky]]

Bug 804635 wouldn't have changes XHR behavior in any way, much less XML parser behavior.

The regression range in comment 3 includes bug 716579, though.  And if you look at the attached file, it starts with 0xfe0xff, which is a UTF-16 BOM.

Bugzilla is serving that file with:

  Content-Type: text/xml; name="xml_encoding_test.xml"; charset=

so why is it working at all in other browsers and why did it use to work for us?  Henri?

Christopher, does the file you server also have a UTF-16 BOM before the UTF-8 data?

Comment 7

[Matthias Versen [:Matti]]

I need a faster bisecting system :-(

The first bad revision is:
changeset:   112406:12288a8a5037
user:        Henri Sivonen <[e-mail]>
date:        Tue Nov 06 13:57:51 2012 +0200
summary:     Bug 716579 - Let a BOM override HTTP-level charset in the HTML and
XML parsers. r=smaug.

Comment 8

[Masatoshi Kimura [:emk]]

Attachment: Testcase reading the attached XML with XHR

Comment 9

[Masatoshi Kimura [:emk]]

(In reply to Boris Zbarsky (:bz) from comment #6)
> The regression range in comment 3 includes bug 716579, though.  And if you
> look at the attached file, it starts with 0xfe0xff, which is a UTF-16 BOM.

I did not see the BOM in the attached file.

Comment 10

[Masatoshi Kimura [:emk]]

Attachment: Test case really starting with the BOM

Comment 11

[Masatoshi Kimura [:emk]]

Attachment: Testcase for the second XML

Comment 12

[Boris Zbarsky [:bzbarsky]]

> I did not see the BOM in the attached file.

When I download it with wget, it sure seems to have a BOM, unless emacs is lying to me.

Comment 13

[Masatoshi Kimura [:emk]]

If the BOM is actually present, it reliably fails with other browsers (at least including Chrome and IE10).

Comment 14

[Masatoshi Kimura [:emk]]

(In reply to Boris Zbarsky (:bz) from comment #12)
> > I did not see the BOM in the attached file.
> 
> When I download it with wget, it sure seems to have a BOM, unless emacs is
> lying to me.

I opened it with a binary editor.

Comment 15

[Boris Zbarsky [:bzbarsky]]

Huh.  Maybe emacs hexl-mode _is_ lying to me, then.....  That's highly unfortunate.

Comment 16

[Masatoshi Kimura [:emk]]

Attachment: PL_strcmp works only if the string is null-terminated

Comment 17

[Masatoshi Kimura [:emk]]

This patch should be simple enough for branches.

Comment 18

[Alex Keybl [:akeybl]]

(In reply to Masatoshi Kimura [:emk] from comment #17)
> This patch should be simple enough for branches.

Is this a valid change in behavior or a regression? Comment 4 suggests that this was going to be fixed server-side. We're not sure this is critical enough to track. Please provide more context.

Comment 19

[Masatoshi Kimura [:emk]]

Other unreported sites may be broken because all other browsers tolerate the mislabeled XML.
That said, a mislabled encoding is a fatal error in XML, so the current behavior is correct per the spec.
Another option is simply removing the bogus PL_strcmp call (in this case, no branch landing is needed).

Comment 20

[christopherlorent]

I confirmed that we fixed it on our site with just changing the xml label.

By the way, I agree that you are now correct to the spec, but, as you said, that may causes backward compatibility problems in unreported sites or intranets, and only firefox 19 and later will not be able to tolerate this client bug...

Comment 21

[christopherlorent]

I have to add that after searches on the web, our problem of UTF8 xml file with a UTF16 generated headers seems to be a FAQ with the .NET framework, so we'll probably not be alone to have this regression.

To answer to Borris' BOM question, we generated the XMl with a webservice, so we don't really generate the xml file, but if you really need the answer, I guess I could find a way to know it with an appropriate breakpoint...

Comment 22

[Henri Sivonen (:hsivonen)]

Comment on attachment 717336 [details] [diff] [review]
PL_strcmp works only if the string is null-terminated

r+. Thank you.

Comment 23

[Masatoshi Kimura [:emk]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/7aebc1dec8f1

Comment 24

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/7aebc1dec8f1

Comment 25

[Lukas Blakk [:lsblakk] use ?needinfo]

Based on comment 21 this doesn't need to be a release blocker (has a workaround) but we'll take a low-risk uplift nomination to branches.

Comment 26

[Masatoshi Kimura [:emk]]

Comment on attachment 717336 [details] [diff] [review]
PL_strcmp works only if the string is null-terminated

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 716579
User impact if declined: Some sites using mislabeled XML may be broken
Testing completed (on m-c, etc.): on m-c
Risk to taking this patch (and alternatives if risky): very low
String or UUID changes made by this patch: no

Comment 27

[Masatoshi Kimura [:emk]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/45da257717e0
https://hg.mozilla.org/releases/mozilla-beta/rev/f14b900f450e

Comment 28

[Mihaela Velimiroviciu (:mihaelav)]

Mozilla/5.0 (Windows NT 6.1; rv:20.0) Gecko/20100101 Firefox/20.0

Verified the fix on Firefox 20 beta 2 build: the attached test cases are handled correclty (same as on Firefox 18.0), except " Test case really starting with the BOM" (https://bugzilla.mozilla.org/attachment.cgi?id=717263) which throws the error at a different location: 
XML Parsing Error: not well-formed
Location: https://bug844007.bugzilla.mozilla.org/attachment.cgi?id=717263
Line Number 1, Column 2:￾㼼浸敶獲潩㵮ㄢ〮•湥潣楤杮∽瑵ⵦ㘱•㸿㰊整瑳琾獥㱴琯獥㹴

versus:
XML Parsing Error: not well-formed
Location: https://bug844007.bugzilla.mozilla.org/attachment.cgi?id=717263
Line Number 1, Column 1:㰿硭氠癥牳楯渽∱⸰∠敮捯摩湧㴢畴昭ㄶ∠㼾਼瑥獴㹴敳琼⽴敳琾
^
(on Firefox 18.0)

Is this expected?