Closed Bug 20570 Opened 25 years ago Closed 25 years ago

Certain JPN character are not displayed in Address Book Card

Categories

(Core :: Layout, defect, P2)

Product:
Core
Component:
Layout
Platform:
x86
Windows NT
Type:
defect

Tracking

()

Status:
VERIFIED FIXED

People

(Reporter: momoi, Assigned: ftang)

References

(
URL
)

Details

(Whiteboard: [PDT-] have fix)

Attachments

(11 files)

UTF8 file with some entities translated into JPN
2.45 KB, text/dtd
Details
abCardOverlay.dtd file. The problematic one.
1.79 KB, text/plain
Details
abCardOverlay.dtd file corrected by inserting spaces in front of the 3 words translated as workadound.
1.75 KB, text/plain
Details
JPG view of the problem with uncorrected abCardOverlay.dtd.
34.26 KB, image/jpeg
Details
JPG view with the corrected (space inserted) abCardOverlay.dtd file.
36.17 KB, image/jpeg
Details
"abCardOverlay.dtd" file in UTF-8 based on 2/11/2000 Win32 build.
2.44 KB, text/plain
Details
JPG image file illustrating how this dialog window should really look like.
34.48 KB, image/jpeg
Details
XUL example turned into HTML equivalent with DIV tags (UTF-8 encoded)
473 bytes, text/html
Details
An image of the above HTML file as viewed by 2/11/00 Mozilla
39.94 KB, image/jpeg
Details
UTF-8 encoded abCardOverlay.dtd file which works with M14 builds
2.61 KB, text/plain
Details
UTF-8 encoded JPN abCardOverlay.dtd file. Ignore the other 2 file of the same name I uploaded.
2.61 KB, text/plain
Details 
** Observed with 12/1/99 Win32 build **

I was reading through Yuta Taniguchi's M10 translation note (see
above URL and noticed this problem is mentioned. In a msg later,
he asked me to file this bug as it is continuing into the current
M12 build.

The problem is that certain characters cannot be dispalyed either
in UTF-8 or NCR converted .dtd files. The workaround used is
to insert a full-width JPN space before or after the problem
character. But this does not seem to work for all problem
characters. Besides this means that you get to see
the space also where none is needed.

An example .dtd file where this problem is obseved is cited below.
Apparently, it is observed in other files, too. One caution
is that I see that \u540D is displayed without a workaround
in some places. Also the suggested workaround does not
seem to work for all cases.

resource:/chrome/addressbook/locale/en-US/abCardOverlay.dtd

** Try translating "!ENTITY Name.tab" & "!ENTITY Name.box" using
   \u540D at the beginning or middle of a word and see what
   happens. With this character, inserting a ful-width JPN spaces
   before and after the chatracter seem to acts as a workaround
   in some cases, e.g. when the character is in a middle position.

** With \u4E08, this workaround does not seem to work.

   You need to play around a bit with these characters to
   get the hang of a problem and its various manifestations.)


Characters you can use for recreating the problem include:

\u540D -- "name" causes the problem at the beginning or middle
           of a word.
\u5909 -- "change" seems to cause the problem at the beginning of
          a word only.
\u4E08 -- "above" seems to cause the problem at the beginning of
          a word only.
\uFF09 -- "full-width right parenthesis" for an obvious reason
          cause the problem at the word end.
Product: Browser → MailNews
This seems to be that the layout engine fail to compute the width of the
glyph.

CC Erik and Frank.
Product: MailNews → Browser
This shall not be a Mail/News problem; it is a general double-byte
character display problem. It might be either layout or xpwidget. IQA will
conduct more tests.

Changing "Product" category to "Browser".
Assignee: tao → trudelle
Component: Localization → XP Toolkit/Widgets
So far, it looks more like an xpwidget-specific or general layout problem.

IQA, would you perform more tests to isolate which one it might be.

CC ji to find out if this is an XP problem.
Rename the attached file to abCardOverlay.dtd
and use it in place of:

resource:/chrome/addressbook/locale/en-US/abCardOverlay.dtd

3 entities have been translated:

<!ENTITY Name.tab   "名前">
<!ENTITY Name.box   "宛名)">
<!ENTITY Address.tab "名鳥">

Open Task | AdressBook, and click on the "New Card" button.
Check the 2 tabs at the top -- you see nothing instead of the
correct 2 characters.
Check the name of the first section -- it does show.

(The above is true for Win32 12/01/99 build)
BTW, I don't think Product makes much difference in this case.
Address Book is part of Mail/News development. The Component
is what is more important.
Same problem on Linux 1999120208-M12 build.
Summary: Certain JPN character are not displayed in Address Book Card → [BETA] Certain JPN character are not displayed in Address Book Card
Adding [BETA] to the summary
Assignee: trudelle → evaughan
Priority: P3 → P2
Target Milestone: M13
Since there have been several comments that this could be a general layout bug,
has anyone been able to reproduce it outside of a widget?  Since it looks like
the tab widget could be involved, reassigning to evaughan as p2 for m13.
Status: NEW → ASSIGNED
Target Milestone: M13 → M14
I have seen this bug several times. It happens all over the place. As I remember 
is because GFX can't calculate the correct size of the fonts. I don't know the 
bug number but it is a general GFX layout bug. Can someone there please get it 
to the right owner.
Assignee: evaughan → troy
Status: ASSIGNED → NEW
Component: XP Toolkit/Widgets → Layout
QA Contact: teruko → petersen
GFX is not layout...
Assignee: troy → evaughan
Eric, this is not the font height problem. Kat, please test those characters
at the beginning and in the middle of words in normal HTML text (i.e. not in a
XUL widget). If the bug still occurs in HTML documents, then the bug might be
in the GFX area. Otherwise, it's somewhere else.
putting on beta1 radar
The tab control uses a generic area frame which is the same at the "DIV" layout 
component. So this problem should occur in HTML as well eliminating the 
possibility of it being a XUL problem. Could you try what Erik suggests and 
recreate it in a generic HTML file? If happens in HTML reassign it to Erik and 
it will get where it is supposed to go. If it doesn't happen in HTML reassign it 
to me and I will investigate it more.

Thanks.
Assignee: evaughan → momoi
Sorry, this bug somehow slipped through my radar for a while.

I tried these 3 problem words in HTML files (in Shift_JIS and UTF-8 encodings)
and also HTML files using DIV tags (in Shift_JIS and UTF-8 encodings).
These words placed at the beginning of each line or somewhere in the
middle of a Japanese sting.

In all of these files, the words all displayed correctly.
Keywords: beta1
This involves manipulating Japanese data and concerns Japanese L10n. Assigning myself as QA contact.
Needed for Beta1.
QA Contact: petersen → momoi
Back to evaughan, as he suggested.
Assignee: momoi → evaughan
I received a note from M13 Japanese Translation team volunteers who just completed
their work. They say that having to discover a problem like this while doing voluminous
translation work and having to put in a workaround case-by-case is a terrible waste of time.
Why do they even bother with case-by-case workarounds at this point? We've only
shipped M13. It's not even beta yet. Things are still changing, and being fixed.
They cannot expect the product to be perfect at this point in time, so they
cannot complain about that either (though we are grateful for their efforts and
early adoption).
Erik, it seems that I have paraphrased their sentiment somewhat wrong. 
I think the JLP team's wish is to have this kind of problem taken care of soon
as that will help what they do. Blame me for poor translation of the sentiment
expressed.

As to the question of why bother with workarounds at this point, these people
are very competent and as Mozilla envangelizers in Japan, they want each Milestone
to look as best as it can. If they find a workaround, they will most likely to put it in
unless it has an effect on core functionality.
Whiteboard: [pdt+]
Why is this assigned to me? Does this work in html? Tabs are just AreaFrames. 
XUL is not doing any of the rendering. Please don't assign this back to me 
unless this is indeed a XUL problem. I don't think it is.
Assignee: evaughan → erik
I assigned it back to you because you requested that it be assigned back to you
if it was found that the problem did not occur in HTML. momoi tested it in HTML
and reported above that it worked. So I assigned it back to you. It's what you
asked for. See above.

I suspect that the problem isn't in XUL either. So I'm reassigning to momoi,
who may know who the address book owner is. Please assign to AddBook owner.
Assignee: erik → momoi
Assigning to hangas. Paul, can you take a look at this?
Assignee: momoi → hangas
Kat, can you explain the problem in non-i18n languange?  I am not clear on the 
issue here.  It looks like a global problem of some sort that has something to do 
with how dtd strings are displayed for other languages.  Maybe we could sit down 
at my computer and you could demonstrate the issue and I could help you get it 
assigned to someone that could fix it.

Paul, let me update this a bit so that you might see this
problem more visually.

I'll update the problem causing abCardOverlay.dtd file.
Please use the file I'll be uploading next, this is 
the file which JPN translators prepared without using the 
workaround first. You can use just this file among the
US build and you'll probably see the problem immediately.

I'll also upload abCardOverlay.dtd file with the space-inserted
as a workaround.

I'll also atttach 2 jpg images illustrating the problem
and the workaround.

1. JPN translators translated the following 3 entities names using
   the same character, \u540D, beginning each word.
2. All three fail to be displayed.
3. When they inserted a space in front of the firt character of each
   word, they were displayed.

!ENTITY Name.tab
!ENTITY Name.box
!ENTITY FirstName.label
(Note: I think the 3rd tab (whose name fails to display) comes from 
somewhere else other than this file (not part of the Mozilla build)
and so probably is unrelated to this problem -- though this
should not happen.)

Paul, please give me a call Monday (except 3-5pm). I'll be happy to 
sit down wit you to insolate the problem.
To see this problem, you would need a Japanese font.
If you don't have one, there is a Windows font below
which you can install via "Control Panel | Fonts | File |
Install new fonts...".

ftp://ftp.netscape.com/pub/communicator/extras/fonts/windows/

Get the Cyberbit.ZIP file. Unarchive it and install via Control Panel.
Summary: [BETA] Certain JPN character are not displayed in Address Book Card → Certain JPN character are not displayed in Address Book Card
I see the change made to the file and the result, but I do not think that this is 
a problem with the address book xul or any address book code.  This looks like 
some global problem with the display of two byte characters in tabs and 
fieldsets.  Evaughan seems to have made the call here.  On 1-21 he says that this 
bug happens all over the place and it is a GFX layout bug.  I think it should be 
assigned to someone that works with the GFX layout of two byte characters.

That's why I suggested testing this in HTML. Kat tested it in HTML, and the
problem did *not* appear there. So the bug is not in GFX (my area). Somebody
needs to isolate the problem. If it's not in address book, it's probably in XUL
somewhere.
assigning to evaughan to investigate, he'll need help from someone in I18N 
reproducing this.
Assignee: hangas → evaughan
Adding me to cc.
I'll be happy to assist evaughan. Please give me a call.
I meant "for demoing" the problem not fixing it.
Ok I tested this. The font comes up great. All except the first tab. Its blank. 
Not sure if the example was old or this is the bug. But the font comes up 
everywhere else in the dialog.
Status: NEW → ASSIGNED
I uploaded the "abCardOverlay.dtd" file based on 2/11/2000
Win32 build. There was one additional entity definition 
but otherwise it is the same file as the one used for
M13-rtm for Japanese.

In this file I see 3 blanks on the 2/11/2000 Win32 build.
Let me illustrate:

On the very first page when you click on New Card, you see the 
following where "XX" marks the missing words. Note that there are
3 missing words: 1st Tab (Name.tab). Name.box, FirstName.label.

  "XX"   Tab2  Tab3  Tab4

 ----------------------------------------------------------------------
|                                                                      |
|  ----- "XX"  --------------------------------------------------      |
|  |                                                          |        |
|  |       Field 1  "XX" :                                    |        |
|  |       Field 2 (Last Name):                               |        |
|  |       Field 3 (Display Name):                            |        |
If you see the 1st tab empty, then that is definitely part
of this bug as reported originally and confirmed on 
2/11/00 build.
The image file I just uploaded was obtained by 
inserting a full-width Japanese space character
in front of each the 3 problem words. When we do
this, all 3 words show up correctly.

View this image while you're using the 2/11/2000 version
of the abCardOverlay.dtd file with a Japanese font on your
system.
Ok I made an isolated testcase in XUL. You will notice that the first word does 
not show up. Now I tried to convert this to html but I can't seem to get it to 
work. I want to isolate this as within xul or html. How can I creat this exact 
example in html with no xul? Its just a div tag with each word with breaks in 
between.

<!DOCTYPE window SYSTEM "chrome://addressbook/locale/abCardOverlay.dtd">
<?xml-stylesheet href="chrome://editor/skin/" type="text/css"?>

<window
	xmlns:html="http://www.w3.org/TR/REC-html40"
	xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
	xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
	align="vertical">

	      <html:div style="border: 5px inset red">
		       &Name.tab; <html:br/> 
			   &Address.tab; <html:br/> 
			   &Other.tab;
	      </html:div>
</window>

Will the above example do for your purpose, Eric?
I'll next upload an image of how that html file looks like on
Mozilla.
Hi, reviewing all the attachments I uploaded, it seems
that I uploaded the original English abCardOverlay.dtd
when I said:

"------- Additional Comments From momoi@netscape.com  2000-02-12 01:12 -------
Created an attachment (id=5190)
"abCardOverlay.dtd" file in UTF-8 based on 2/11/2000 Win32 build."

Since evaughan already made an example of .xul file that
works, I take it that he coprrected the M13-UTF8 file slightly
to make it with the current builds.

I'll now upload the correct UTF-8 encoded JPN abCardOverlay.dtd 
which will work with M14 builds.
Sorry!
Sorry, I made another error. It seems that you cannot upload a 
file with the same name again. I uploaded abCardOverlay.dtd
sometime ago and even though I corrected the file
to the one needed for M14, it cannot replace or exist side-by-side
with the same name file which I uploaded before. That's why
you get the same English file.

This time, I'll rename the file so that it will not match with the
one that was uploaded before.
Ok I still can't figure this out. Its really weird. But I did get this dialog to 
we fine with a work around. If you have a current build (yesterday) a new xul 
tag has been added called "text". Its syntax is <text value="mytext"/>. It 
displays these character quite well. So if you change the tabs to:

	      <tab selected="true"><text value="&Name.tab;"/></tab>
	      <tab><text value="&Address.tab;"/></tab>
	      <tab><text value="&Other.tab;"/></tab>

You can also do this in the "legend" of fieldsets or anywhere something is not 
showing. In fact displaying text this way is much more efficent in xul than divs 
so if you don't need text wrapping or special font changes you should use these.

What do you think? If this will unblock you we should probably take this bug off 
the PDP+ radar.
Ok I fixed this dialog. It will now display the UTF-8 character. I did what I 
mentioned in the previous message. So I'm marking this fixed. I have opened 
another bug with the simple xul example that doesn't work and mark it as post 
beta. bug # is: 27653
Status: ASSIGNED → RESOLVED
Closed: 25 years ago
Resolution: --- → FIXED
OK. It tried modification of "abCardOverlya.xul" in the 3 
problematic places with 2/11/2000 build:

<text value="&Name.tab;"/>
<text value="&Name.box;"/>
<text value="&FirstName.label;"/>

With these modifications, as evaughan says above, the "name" character
does show correctly. I also tried the other problematical characters
reported for this bug,i.e. \u5909 \u4E0A \uFF09, under UTF-8. They 
all showed correctly with this workaround.

While I agree that this creates a workaround, how do we ensure
that this problem does not crop up elsewhere? 
Are we going to suggest that .xul writers should always use
this style elsewhere? Or is this just a workarond to be used
only until Bug 27653 gets fixed?
Hi, Eric, it does not look like "<text value="&FirstName.label;"/>" 
was not done for "&FirstName.label;". This label was one of 
the original 3 problem cases reported. And it would not display the
character without this change. 

Re-opening ...
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Eric, please take a look at Bug 27699.
Discussion there points to a problem when the 2nd byte of the 
character contains what would correspond to ASCII Control characters.
Look at all the probelm characters I reported on 12/1/99. They all contain
the 2nd byte in ASCII Control characters range.
Yes it does sound like there is some problem with rendering these characters. 
But I can't figure out why xul causes it to happen. A div tag in XUL is the same 
as a div tag in html. Its the same code path. I may just have to step in the 
debugger through both examples and figure out why it does not paint. But that 
will take some time.
I'm guessing that it's not working in XUL because the XML parser is munging
Unicode characters that happen to have a low-order byte in the control
character range (e.g. 0x0D). To rule out XUL, all you have to do is create an
XML file with arbitrary element names (e.g. MYDIV) together with a CSS style
sheet that indicates how to present MYDIV elements. See the following for an
example of XML with CSS:

  http://www.xml.com/1999/03/ie5/first-x.xml
  http://www.xml.com/1999/03/ie5/first-x.css

If you can show that XML itself has the same problem with those characters,
then it's not XUL's fault. It's the XML parser's fault.
Eric, it would probably be best in any case to complete
your workaround by taking care of the: &FirstName.label; 
by changing that sectiotn to include: <text value="&FirstName.label;"/>.
This was missed in the workaround you put in earlier in
abCardOverlya.xul. 
Well I made my fix to the tabs. This fix is for the lable. Unfortunately <text> 
tags can't work as labels (yet). So the work around won't work for this. So I 
guess we will just need fix this bug.
Depends on: 27954
Eric, as I said before, I used <text value="&FirstName.label;"/>
in 2-3 builds after 2/11/00 and it actually let the label
display correctly in Japanese. As a workaround, this should be
OK. 

ftang notes in Bug 27954 what might be causing this and a
few other bugs in different places. Please take a look at it.
troy,
 I'll be on vacation to the 23 so I'm moving this to layout to take a look at 
it in my absence so at least someone is potentially looking at it. Please 
reassign it back to me on the 23. Thanks.
Assignee: evaughan → troy
Status: REOPENED → NEW
I don't think is a show stopper for beta1. Removing PDT+ to have this 
reevaluated
Whiteboard: [pdt+]
Whiteboard: [PDT-]
I'll ask this question again. In abCardOverlay.xul file, if we change

<html:label for="FirstName" class="CardEdit">&FirstName.label;</html:label>

to 

<html:label for="FirstName" class="CardEdit"><text 
value="&FirstName.label;"/></html:label>

then the 3rd problem character shows correctly on 2/18/2000 Win32 build
with the Japanese .dtd file. 
Eric stated above that the text tag does not work for labels. Yet my
test shows that it does work in this case. That is why I said 
above that this workaround would be OK for M14. 

Is there something terribly wrong in using the text tag for labels?
If not, can we at least put in the above workaround for M14 for
the &FirstName.label; string?

By the way, I removed the text tags from the abCardOverlay.xul file
for the 3 problematic characters of this bug to see if rickg's fix in
Bug 27954 works for this bug. Unfortunately, it did not. So we need
this final piece of the workaround for now.

Not critical for beta1, so moving back to Eric's bug list
Assignee: troy → evaughan
The problem is there are html:div tag in dialogs all over mozilla to do wrapping 
text. They might contain problematic characters. So eventually you will run into 
a place where it won't show up and you can't fix it with a <text> tag because 
the don't wrap like html:div does. So if you want to patch a few dialogs you 
can you should go to the PDP team as get permission to patch. But I'm sure you 
will find other dialogs that won't work.
Status: NEW → ASSIGNED
OK. Let's wait for the real solution. L10n project people can
patch or use a workaround of one kind or another as needed.
does the origional problem (not the work around) fixed after rickg check in the
fix for 27954?

Frank, as I mentioned in my  momoi@netscape.com  2000-02-18 15:14 comments,
the answer is no. This problem was not resolved by that fix -- I removed the workaround from the
.xul file and looked at it after rickg fixed the problem.
tao and momoi, can one of you recreate the problem in my local build
so I can help to narrow down the problem. This bug report is out
of control and the information in here is simply too much for any
one to read.

This will reproduce it nicely:

Make sure you download the attached version of abCardOverlay.dtd with the 
special characters in it. And place it in "dist" where there current one is. 
Then open this xul file in mozilla. The first character will not show up.

<!DOCTYPE window SYSTEM "chrome://addressbook/locale/abCardOverlay.dtd">
<?xml-stylesheet href="chrome://editor/skin/" type="text/css"?>

<window
        xmlns:html="http://www.w3.org/TR/REC-html40"
        xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
        xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
        align="vertical">

              <html:div style="border: 5px inset red">
                       &Name.tab; <html:br/> 
                           &Address.tab; <html:br/> 
                           &Other.tab;
              </html:div>
</window>
I have the fix the problem is in the nsExpatTokenizer.cpp
Here is the patch

Nithesis forget to cast the data to PRUnichar* before compare it, and what
happen the comparsion of s[0] take the first BYTE instead of the first character
to compare with 0x0d, etc.
reassign back to ftang and clear PDT-

Index: src/nsExpatTokenizer.cpp
===================================================================
RCS file: /m/pub/mozilla/htmlparser/src/nsExpatTokenizer.cpp,v
retrieving revision 1.57
diff -c -r1.57 nsExpatTokenizer.cpp
*** nsExpatTokenizer.cpp        2000/01/28 08:02:51     1.57
--- nsExpatTokenizer.cpp        2000/03/03 20:52:59
***************
*** 448,454 ****
    } else {
      CToken* newToken = 0;

!     switch(s[0]){
        case kNewLine:
        case CR:
          newToken = state->tokenRecycler->CreateTokenOfType(eToken_newline,eHTM
LTag_unknown);
--- 448,454 ----
    } else {
      CToken* newToken = 0;

!     switch(((PRUnichar*)s)[0]){
        case kNewLine:
        case CR:
          newToken = state->tokenRecycler->CreateTokenOfType(eToken_newline,eHTM
LTag_unknown);
***************
*** 462,468 ****
      }

      if(newToken) {
!       if ((s[0] != (XML_Char)kNewLine) && (s[0] != (XML_Char)kCR)) {
          nsString& theString=newToken->GetStringValueXXX();
          theString.Append((PRUnichar *) s,len);
        }
--- 462,468 ----
      }

      if(newToken) {
!       if ((((PRUnichar*)s)[0] != (XML_Char)kNewLine) && (((PRUnichar*)s)[0] !=
 (XML_Char)kCR)) {
          nsString& theString=newToken->GetStringValueXXX();
          theString.Append((PRUnichar *) s,len);
        }

Z:\mozilla\htmlparser>
Assignee: evaughan → ftang
Status: ASSIGNED → NEW
Whiteboard: [PDT-]
Status: NEW → ASSIGNED
Whiteboard: have fix
Frank, thanks for fixing the bug. As I guessed, it was in the XML parser.
Nisheeth, please take a look at the fix, not only to review it, but to become
aware of this type of Unicode issue.
PDT-
Whiteboard: have fix → [PDT-] have fix
fix and check in.
Status: ASSIGNED → RESOLVED
Closed: 25 years ago → 25 years ago
Resolution: --- → FIXED
** Checked with 3/6/2000 Win32 build **

Of the 3 problems reported poriginally, 2 have been
fixed by this check in.

&Name.tab;
&FirstName.label

but '.box' type does not work. The 2 characters for "name"
still disappear without a workaround.

&Name.box

Re-opening ...
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Corrections:

&FirstName.label  --> &FirstName.label;
&Name.box         --> &Name.box;

Forgot the ";" at the end when writing these two.

It seems we have additional problems.
Status: REOPENED → ASSIGNED
Frank showed me that there was more than a simple
<text value= ...> change made to the &Name.box; part. 
Simply removing the <text...> tag will not work even for 
English.
With M13 abCardoverly.xul file without the workarounds
put in by eric, the fix here worked perfectly for &Name.box;
also. I'm now resolving this as fixed again and verifying it.
Status: ASSIGNED → RESOLVED
Closed: 25 years ago → 25 years ago
Resolution: --- → FIXED
Verified as fixed.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: