159188 - help index sorting is incorrect for international

Status: VERIFIED FIXED

(SeaMonkey :: Help Documentation, defect)

Description

[rchen]

"The current sort function does not working well for double-byte languages like
Japanese, ie, whatever we did sort in the RDF, our app will re-sort them again
when display, but in a wrong way, a messy way." from Yxia'report.

Comment 1

[Michele Carlson]

adding adt1 rtm, l12y, nsbeta1. We really need this fixed for Mach V. It's a
stop ship bug for the int'l versions of the Help.

Comment 2

[rchen]

Attachment: one of the three bitmaps sent from the translator

Comment 3

[rchen]

Attachment: Please ignore the previos bmp. I convert it to jpg. - The first screen shot from the vendor.

Comment 4

[rchen]

Attachment: The second screen shot.

Comment 5

[rchen]

Attachment: The third one.

Comment 6

[rchen]

It is suggested by the vendor to have a flag in the preference to turn on/off
the index sorting, like sortindex="true" or "false".  If the flag is set to off,
the order is based on the order in the rdf file.  

I suppose the strings were read in the a tree (binary tree?). The strings were
sorted  when they were stored. We may have the choice to sort the strings or not
to sort them but in the order of the strings in the rdf file?

Comment 7

[Frank Tang]

is this help stuff in mozilla or netscape tree?

Comment 8

[Frank Tang]

Is this the right place we should look into?
http://lxr.mozilla.org/seamonkey/source/extensions/help/resources/content/help.xul
 <!-- index panel -->

Comment 9

[Frank Tang]

it looks the content is a merge of
http://lxr.mcom.com/commercial/source/extensions/help/resources/locale/en-US/help-index1.rdf
and
http://lxr.mozilla.org/seamonkey/source/extensions/help/resources/locale/en-US/help-index1.rdf

is that what happened ?
Does XUL sort service kick in to sort it? otherwise, how the comparision happen?

Comment 10

[rudman]

This bug should be in Bugscape for now, yes? Though the general issues do apply
to Help translations.

The commercial build of Help stomps on ("overwrites") the Moz build. We're
mainly concerned with the commercial build for now, namely, the source
identified at
http://lxr.mcom.com/commercial/source/extensions/help/resources/locale/en-US/help-index1.rdf.
In other words, the source is entirely within one tree or another, not a merge.

Here's what I think might be happening: Help index sorting does not happen at
build time or run-time. Ian created a script that he used for the sorting. He
ran the help html files through the scripts, which produced the index RDF file.
The scripts look at idx tags and sorts according to those. If those tags are not
translated, then presumably the sorting always occurs according to the English
version. Could that account for the "wrong, messy way"? If so, we'd have to make
the script available to the translators (and post on Mozilla).

Ian is at a conference now but is in email contact. He'll need to comment here
when he's able to check email and access Bugzilla. No ETA for a fix until then.
As I may be wrong on the assessment of the problem, no conclusions should be
drawn on the fix until Ian has a chance to comment.

Comment 11

[Ian Oeschger (gone)]

Attachment: sorting (and creating in the first place) script for help index

Yes. Steve has it (and helped me remember what was going on here): I don't sort
in the XUL because I sorted in the RDF itself as I generated it. The
comparisons I make in line 52 of this Python script (i.e., keylist.sort(lambda
x, y: cmp(string.lower(x), string.lower(y)))) are a homegrown sorting thing.

But the sorting is integral to the script: I have to order my lists of entries
so I can make sure I'm not making redundant entries and so I can do the
subentries that are specified by the ":" in the HTML source. Can't simple strip
this stuff out of the script and sort in the XUL. Which is why some sort of
triggered/preference-wrapped sorting in the viewer itself might be a good idea.


(But still not sure I am the rightful owner of this bug.)

Comment 12

[rchen]

I received the Japanese localized files from the vendor. The order in
help-index1.rdf has been sorted according to the pronunciation of Japanese
translation manually by the vendor. However I saw the order displayed in Help
System index tag is not the same as in the rdf file. If there is no run time
sorting, how can we explain this?  

I think we should have a short term solution and a long term solution for this.
 The short solution is to have a preference for us to turn on/off the run time
sorting so that we can turn it off for some languages to follow the order in rdf
file. We can live with this for RTM release. 

The long term solution can be providing an extra field "sorting index" in the
rdf file to store the extra info for languages like Japanese and Chinese. The
application can use "sorting index" to sort the string in run time.

Comment 13

[rchen]

I found out if I altered the order of records in help-index1.rdf in en-US.jar,
it didn't change the order in display. It proves that the order in
help-index1.rdf is not the order in display. Is it right, Ian?  Or there is
something I am missing.

Comment 14

[Ian Oeschger (gone)]

Ray - yes. sorry. I mispoke in an earlier comment (and then double-checked): I
do sort in the index. The attributes of the tree in this snippet handle the sorting:
http://lxr.mozilla.org/seamonkey/source/extensions/help/resources/content/help.xul#211

and it's obviously sorted by English. But if the IDX tags in the HTML source are
translated and if (and here's the big IF) the tree has sorting for other
languages, then a Japanese RDF file should be sorted in the viewer correctly.
Maybe we can bring a tree person in here--like hewitt (cc'd)?

Comment 15

[rchen]

Ian, the sorting method can not be universal.  The logic for English doesn't
apply to Japanese and Chinese. The sorting for Japanese is based on the
pronunciation which can't be interpreted by the text string itself. That's why
at this moment the  short term solution is to have a flag to turn on/off the
index sorting in application. The vendor has pre-sortted  the Japanese RDF file
manually. We can discuss the long term solution suggested in comment #12
afterwards. 

Comment 16

[Ian Oeschger (gone)]

sounds good, ray. looking into it more now...the English RDF file is pretty well
sorted. I didn't think about doing this explicitly, but it may be an artifact of
the sorting I had to do to get all the entries organized. Maybe we can turn off
the  sorting in the index altogether and see if we still end up with sorted
English? Giving that a whirl now.

Comment 17

[rchen]

I am not so sure about turning it off for all.

cc [e-mail]. They are doing european languages. Richie, could this bug
be a problem for you?


It is better to make it flexible. I hate to see regression at this moment.

Comment 18

[Ian Oeschger (gone)]

Attachment: remove sorting from index (and sort in the RDF)

could it be this easy? I removed the sorting attributes from the index tree and
we still get the index sorted appropriately. We could do it this way and just
have each language do their own hand sorting of the RDF (though that is still a
bit manual).

Comment 19

[rchen]

I am going to have a quick test with you patch. If it works, we might just
extract the "true" value to a dtd file and refers it in the xul. That can be
very simple fix. We don't even need to use preference. 

Comment 20

[rchen]

It works! I just removed one line - sortActive="true".

The easiest way to fix this probably is to put the value in the dtd file and
refers it in the xul file.

See we did for the radiobutton:
in dtd file:
<!ENTITY hidenetscapeRadioButton          "false">
in xul file:
<checkbox id="netscapeRadioButton"
                    label="&netscapeRadioButton.label;"
                    pref="true"
                    preftype="bool"
                    prefstring="browser.toolbars.showbutton.netscaperadio"
                    prefattribute = "checked"
                    hidden="&hidenetscapeRadioButton;"/>

Then, we have the control in localization.

Comment 21

[Ian Oeschger (gone)]

Attachment: updated per rchen's suggestion

great idea, ray! here is that patch. need to get this reviewed now I guess (and
go eat some dinner).

Comment 22

[rchen]

r=rchen. Need to find out sr and driver.

Comment 23

[Ian Oeschger (gone)]

cc'ing blake and trolling for a superreview on this simple update to the help system

Comment 24

[Richard Close]

We have not seen this problem in our builds - probably because we use Roman
characters also.  Unless, of course an accented character is the first letter....

Comment 25

[jag (Peter Annema)]

Comment on attachment 92983 [details] [diff] [review]
updated per rchen's suggestion

sr=jag

Comment 26

[Jaime Rodriguez, Jr.]

adt1.0.1+ (on ADT's behalf) approval for checkin to the 1.0 branch, pending
drivers's approval. pls check this in asap, then replace "fixed1.0.1" with
"verified1.0.1". thanks!

Comment 27

[Ian Oeschger (gone)]

cc'ing asa for driver approval. Hi, Asa - think you can stamp this simple fix?
thanks

Comment 28

[Jaime Rodriguez, Jr.]

valeski or randell jessup would be the right guys for branch checkins. pls email
them to get one of their approvals for the 1.0 branch.

Comment 29

[Judson Valeski]

 please checkin to the MOZILLA_1_0_BRANCH branch. once there, remove the
"mozilla1.0.1+" keyword and add the "fixed1.0.1" keyword. 

Comment 30

[Ian Oeschger (gone)]

checked into branch. updating keywords accordingly.

Comment 31

[Ying-Lin Xia]

> Thanks to Ian, Ray to fix this problem quick, and everyones help.
> I just confirmed with the 08-03 US branch build, the Japanese index
> sorting issue is fixed by adding the "sortflag" setting in help.dtd.
> HTP's current JA index RDF file works good to me for the JA order.
>
> Please go ahead to verify the bugzilla bug:
> http://bugzilla.mozilla.org/show_bug.cgi?id=159188
>
> IQA, in case if you like to check it out, I just made a JA language pack
> (same as last Friday's pack but just add one sortflag in), you can
> install it into the latest US branch build.
> http://jazz/users/yxia/publish/test/20020805_ja_index/

Comment 32

[Kasumi]

Confirmed JA fix on 2002-08-03-08-1.0 +
http://jazz/users/yxia/publish/test/20020805_ja_index/
If happened on Zh, Please reopen. 

Comment 33

[Kasumi]

verified

Comment 34

[Ying-Lin Xia]

And please check the patch into trunk.

Comment 35

[Jessica Blanco]

Did this ever get fixed in the trunk? This patch was never checked into the trunk.