79833 - XML:UTF8 characters are not displayed correctly in Page source

Status: RESOLVED WORKSFORME

(Core :: XML, defect, P2)

Description

[Teruko Kobayashi]

This XML page is written in UTF-8, but this does not have encoding declaration.

Japanese characters are displayed correctly, but they are not displayed 
correctly in View|Page source.

This may be related to the bug 41981.

Tested 2001-05-08 build.

Comment 1

[Teruko Kobayashi]

Changed QA contact to [e-mail] and added intl keyword.

Comment 2

[Heikki Toivonen (remove -bugzilla when emailing directly)]

Frank, do you want to have this one? I see you fixed some other intl XML view
source problem earlier...

Comment 3

[Heikki Toivonen (remove -bugzilla when emailing directly)]

nsBranch+ because this is pretty serious handicap for intl developers. Hoping
the fix would be simple.

Comment 4

[Heikki Toivonen (remove -bugzilla when emailing directly)]

This is actually a regression, view source works correctly in NS 6.1.

Comment 5

[Heikki Toivonen (remove -bugzilla when emailing directly)]

Argh, this is bizarre! Occasionally things work, but usually they don't. I have
seen working and broken behaviour both with NS 6.1 and the latest trunk builds.
Removing regression keyword.

Comment 6

[Heikki Toivonen (remove -bugzilla when emailing directly)]

We have DetectByteOrderMark() that can autodetect some charsets, even if the
encoding information is missing. Downloading the URL we get into that function,
and see the first 4 bytes as: 3C 3F 78 6D. According to
http://www.w3.org/TR/2000/REC-xml-20001006#sec-guessing the encoding information
is required in that case. Without it, the DetectByteOrderMark() function does
not detect the encoding at all. 

For normal XML view this happens to work correctly since UTF-8 is the default.
But view source goes through HTML code path, which has a different default.
Since DetectByteOrderMark() was not able to detect the encoding, we are left
with the HTML default which shows the bug.

The weird thing still remains: how does the view source SOMETIMES work correctly?

Teruko, could you provide an additional testcase, which is basically the same as
in URL but that has the encoding explicitly set to UTF-8?

While debugging, I noticed that DetectByteOrderMark() is buggy and slow; I
started fixing it, and will attach what I have done so far.

Comment 7

[Heikki Toivonen (remove -bugzilla when emailing directly)]

Attachment: Some DetectByteOrderMark() fixes

Comment 8

[Teruko Kobayashi]

Heikki, I created the test case which you requested in
http://babel/tests/browser/xml/xml-japan/weekly-utf8.xml

View sourch works fine in the page.

Comment 9

[Heikki Toivonen (remove -bugzilla when emailing directly)]

Er, but now the original is gone? I need both, please, maybe named like:

weekly-utf-8.xml                - the original
weekly-utf-8-with-encoding.xml  - the one with encoding

Also, what kind of program can I use to edit these files? Win2k Notepad displays
the characters as "[]", and Visual C++ 6 as garbage, and saving the file from
either one will change the byte order marks.

Comment 10

[Teruko Kobayashi]

Heikki, I created the test cases as you requested.
http://babel/tests/browser/xml/xml-japan/weekly-utf-8.xml
http://babel/tests/browser/xml/xml-japan/weekly-utf-8-with-encoidng.xml

Comment 11

[Shanjian Li]

The charset of a html page is determined in 
/mozilla/content/html/document/src/nsHTMLDocument.cpp, function 
StartDocumentLoad(). I tried one testcase in  view source, it did go along that 
code path. Yes, for html, the default normally is latin1, but user can change 
that. If your default is set to UTF8, or if you have some of the charset 
autodetector turn on, UTF8 will be detected. That might explained the bizzarre 
behavior you experienced.  
Hope those infor is helpful. Let me know if you have questions. 

Comment 12

[Heikki Toivonen (remove -bugzilla when emailing directly)]

Adding perf keyword as well. DetectByteOrderMark() is called 22 times before the
profile manager comes up, of which 18 times we actually end up in trying to
detect the encoding. The numbers to bring up about:blank are 117 and 83,
respectively. While detecting encoding, the current code first copies the buffer
to a temp variable (usually a memory allocation needed since the buffer is so
big), does a Cut() string operation and some unneeded comparisons etc.

I believe I have now speeded up the encoding detection part quite a bit. There
is no longer any string copying done, and no unneeded comparisons. It could
perhaps be still made a bit faster by relying more on the allowed form and bit
operations.

I'll try to check what Quantify has to say.

Needless to say, I am still trying to track down why things sometimes work
without the encoding and sometimes not.

Btw, Teruko, the encoding sample has different byte order marks from the
original. I finally tried editing with Emacs, and it preserves the byte order
marks so now I can twiddle with my local testcase as well.

I will attach the current patch.

Comment 13

[Heikki Toivonen (remove -bugzilla when emailing directly)]

Attachment: opt

Comment 14

[Heikki Toivonen (remove -bugzilla when emailing directly)]

Attachment: opt2

Comment 15

[Heikki Toivonen (remove -bugzilla when emailing directly)]

Moving. My understanding of the spec is that this could actually be a WONTFIX,
but I really would like to know why things occasionally work. Putting explicit
encoding in the XML declaration is always correct and in the testcase I believe
it is even required per the spec because the first 4 bytes are "3C 3F 78 6D".

Comment 16

[Heikki Toivonen (remove -bugzilla when emailing directly)]

The perf patch has landed, removing keyword. As to the actual bug, I understand
the testcase was incorrect (always requiring encoding in which case we do the
right thing). The weird thing is when it is not present and things sometimes
work and sometimes they don't work. Minor issue, so moving to future.

Comment 17

[Boris Zbarsky [:bzbarsky]]

This worksforme now that view-source defaults to the charset of the page you
opened it from....