233689 - Japanese (possible other languages as well) in UTF-8 is not displayed if mixed with non japanese Characters

Status: RESOLVED INVALID

(Firefox :: General, defect)

Description

[Frank Berger]

User-Agent:       
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040207 Firefox/0.8

Japanese Characters, encoded in UTF-8 are not displayed if they are mixed with
non  japanese characters inside a html-statement.

please see http://www.nosebleed.de/jap/utf8.html 
in the row labeled 'japanese:-              -' is actually text written. You
might see it if you try to mark it with your mouse. You should see the same
japanese text as written a row below. here you have only japanese text between
two <hr>s

for reference how it should look like: http://www.nosebleed.de/jap/euc-jp.html

this is only reproducable for me with firefox 0.8.. in firebird 0.7 and Mozilla
1.6 this works fine..

Reproducible: Always
Steps to Reproduce:
1. view a japanese/english mixed page in utf-8
2.
3.

Actual Results:  
when hightlighting the supposed japanese text, it sometimes appears stays for a
short while, until something else is highlighted

Expected Results:  
displaying the japnese characters

Comment 1

[Frank Berger]

update: it happens as well with german umlauts. entire words with german umlauts
in it won't be displayed... the words around them with english-valid-characters
are displayed.

Comment 2

[Jason Barnabe (np)]

I just see a bunch of question marks at both URIs on Mozilla/5.0 (Windows; U;
Windows NT 5.1; en-US; rv:1.7a) Gecko/20040210 Firebird/0.8.0+

Comment 3

[Frank Berger]

well, you might need to have japanese fonts installed!?

Comment 4

[zankou]

i just came across this very problem when accessing www.google.co.kr.  it
happens in both firefox 0.8 and thunderbird 0.5 in linux, exactly as the
reported has described.  shift-jis and euc-kr encodings seem to work fine, but
utf-8 seems to cause real problems with korean and japanese characters.  just as
a test, i found i am able to view the same page in konqueror without problems.

Comment 5

[Ryuichi KUBUKI]

Sample pages are 404 not found. Can anyone upload testcase?

Comment 6

[Frank Berger]

sorry, old page went down, new test-case (same files):
http://home.bawue.de/~frberger/jap/

Comment 7

[Ryuichi KUBUKI]

(In reply to comment #6)
> sorry, old page went down, new test-case (same files):
> http://home.bawue.de/~frberger/jap/

WFM - I could see the both encodings without a problem, UTF-8 and EUC-JP (two
lines of "japanese: this is Japanese" and "this is Japanese" in Japanese) with
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a2) Gecko/20040622
Firefox/0.8.0+ (trunk) on Windows XP Japanese version. Character Encoding |
Auto-detect is set to Japanese.

Comment 8

[me]

*** Bug 253914 has been marked as a duplicate of this bug. ***

Comment 9

[me]

Similar problem with Chinese UTF-8 or HTML entities on Mozilla/5.0 (X11; U;
Linux i686; en-US; rv:1.7) Gecko/20040626 Firefox/0.9.1.

Steps to Reproduce:
1. display http://en.wikipedia.org/wiki/Main_Page
2. look at other languages on left hand side, locate references to Chinese and
Japanese.

Comment 10

[me]

(From the forums) is this a failure to access all fonts? Something to do with
gtk2 (whatever that is)? Only getting the TrueType fonts and not the other kind?

Mozilla 1.4 displayed Chinese and Japanese characters without any problems; it's
just Firefox that can't.

Comment 11

[me]

I've fixed my problem, I think, by downloading all the TrueType fonts from
Windows and installing them on Linux. At least, Firefox is now displaying
Chinese and Japanese characters for me. (Not to say that this is the same as
Frank Berger's bug).

Comment 12

[Frank Berger]

In Version 1.0 Preview Release 
Mozilla/5.0 (X11; U; Linux i686; rv:1.7.3) Gecko/20040914 Firefox/0.10 
the problem seems to be fixed. 
I'm the original creator of this bug, and I can't reproduce the bug anymore in
the new Browser-version.

Has anybody an idea what happened? I didn't upgrade or change anything else on
my system besides upgrading Firefox.

Maybe this was related to the usage of several fonts from several languages in
one page? something which basically can only happen if a page is in UTF-8?

Comment 13

[Daniel Lublin]

I'm running 1.0PR and still have the problem.
On the page:

  http://en.wikipedia.org/wiki/Deng_Xiaoping

I can see neither Chinese, Japanese or Korean characters, for example
in the "in other languages"-box to the left. What I see is for each
character is box with hexadecimal numbers.

Comment 14

[Daniel Lublin]

Ah, yes. I just tried installing Mozilla 1.7.3, and found out that it
correctly display all characters on the mentioned Wikipedia page.

If I look at the font settings in Mozilla (Preferences->Appearance->Fonts),
I see various jis-fixed-jisx and isas-(fang)song ti-gb2313 fonts
for Japanese and Simplified Chinese. The corresponding fields in
Firefox are empty, but the configuration file grepfs/all.js contains,
for both Mozilla and Firefox, nonetheless these settings.
Does Firefox filter out these fonts somehow?

Also, I had a look at about:buildconfig. For Firefox, the configuration
options contain "--enable-xft --disable-freetype2". I do not know if
this is of any relevance.

(My system btw, is Debian GNU/Linux on IA32, unstable)

Comment 15

[Frank Berger]

I found the reason why it didn't work the versions between 0.5? and 0.91
If a page only sets the UTF-8 information in the <meta http-equiv> header
then UTF-8 characters above the US-ASCII equivialent (maybe ISO-8859-1) were 
invisible.
They were visible if additionaly the Content-type header had the character-set
as well, eq: Content-type: text/html; encoding="UTF-8"
THEN the older versions of Firefox will display all UTF-8 characters.