181711 - A pref to not convert Arabic Numerals to Hindi

Status: RESOLVED FIXED

(Core :: Layout: Text and Fonts, enhancement)

Description

[Thamer Mahmoud]

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.3a) Gecko/20021120

Testcase URL is an example from Bug 82347. The bug occurs with the default
mozilla numeral setting (regularcontextnumeralBidi) and the Testcase renders
correctly in IE6.

TO REPRODUCE:

1. Go to http://bugzilla.mozilla.org/attachment.cgi?id=54477&action=view
2. Look at the second example (2nd line)

ACTUAL :  Arabic letters followed by Hindi numerals
EXPECTED :  Arabic letters followed by Arabic numerals

Comment 1

[Andreas Kunz]

Well, the persons involved in that bug seem to think that the screenshot after
their patch (attachment 57329 [details]) is fine - and it looks like what I see.
I don't know what particular set of letters you know better than me (telling
fomr your mail adress maybe arabic?).

I'll just try my guess because I once tried to learn a bit of arabic and I still
know the numbers I was shown there. 
And these numbers looked exactly like the signs on the left side of the second
line. The signs on the right look very like arabic letters, so it seems correct
to me.
IE shows our "usual" letters there: 12345... these are also called "Arabic"
because they are derived from the original arabic letters. (Because the Romans
were too stupid to find a good way of writing down numbers, you know: I, II,
III, IV, V, VI,... - these would be "Roman Numbers").
Maybe mozilla uses the context to decide what "arabic numbers" are in each case:
either the original ones (second line) or the western derivation of them (first
line).

What confuses me a bit is that (in IE and mozilla) Persian numerals, Hindi
decimals and (real) Arabic numerals look just the same. But since I don't know
anything about Persian and Hindi signs, that could be ok, because they might be
very similar....

So please make clear if you are very certain about one of these charsets - and
anybody else who knows more: feel free to state what you knaow about it. 

Comment 2

[Thamer Mahmoud]

Well I'm not saying that the testcase is wrong (Note that the actual example
text is referring to the source code and not the rendering) so there shouldn’t
be any complaints about this in the original (older) bug.

I think the IE6 rendering is more correct than mozilla's own context rendering.
I've seen many anomalies with number rendering in mozilla on arabic pages but
none in IE6. This is while system setting is set to context and standard numbers
is set to Arabic Numerals. (translates to regularcontextnumeralBidi?) 

Is this is the only difference in rendering that is causing all these problems?
I don't know, but fixing this one (i.e. emulating IE) would be a good start for
sure.

As far as I know about Persian numbers, they’re identical to arabic except for 4
and 5. I'm not sure about their other digits though. 


For more info...  http://unicode.org/glossary/index.html#arabic_digits

Comment 3

[Simon Montagu :smontagu]

I disagree with the reporter's expectations. I understand contextual
substitution to mean that any numbers after Arabic text should be displayed as
Hindi numbers, and that also seems to be what Microsoft's documentation is
saying at
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/intl/nls_0to3.asp:
"Context-based substitution. Digits are displayed based on the previous text in
the same output葉hat is, European digits follow Latin scripts, Arabic-Indic
digits follow Arabic text, and Thai digits follow Thai text."

In practice, IE (tested with 5.5) only does this if the "Standard digits" are
set to Hindi numbers.

Comment 4

[Thamer Mahmoud]

Standard Digits is actually a very practical feature, as Hindi Numerals tend
to look smaller and clumsier than Arabic Numerals in some fonts. I do prefer
setting (context + Arabic Numerals) because it doesn't break websites that
hard code their numbers, this is as opposed to something like "arabicnumeralBidi".

I reported this bug thinking that "regularcontextnumeralBidi" is equivalent to
IE's (context + Arabic Numerals). What lead me to that assumption are the
labels of those settings. (i.e Regular vs Hindi)  and the fact that there are
two separate settings dealing with contextual numerals. Now it seems that a
setting like IE's (context + Arabic Numeral) is missing from mozilla and
instead we have "hindicontextnumeralBidi" which is what exactly?

Comment 5

[Thamer Mahmoud]

I think there should be a new pref that emualtes IE's rendering while Standard
Digits is set to Arabic Numerals.

Revising summary and --> enhancement

Comment 6

[Simon Montagu :smontagu]

Making our prefs and behaviour consistent with MS Windows seems reasonable to
me, especially if we also fix bug 151374 and listen to the system setting by
default.

Adding some ccs

Comment 7

[Simon Montagu :smontagu]

I'm changing my position on this issue after being referred to Section 13.3 of
the Unicode Standard (http://www.unicode.org/book/ch13.pdf, p. 320)

By default, we should always display the nominal forms with no substitution.
This doesn't require any semantic change to the code, only changing the default
value of bidi.numeral in all.js and adding a few comments.

Comment 8

[Simon Montagu :smontagu]

Attachment: Add an option not to do numeral conversion and make it the default

Comment 9

[Simon Montagu :smontagu]

BTW, the reason for making this change comes from
http://lists.w3.org/Archives/Public/public-iri/2003May/0010.html

Comment 10

[Simon Montagu :smontagu]

Attachment: Full patch

Oops, missed a file in the first attachment

Comment 11

[rbs]

Comment on attachment 122607 [details] [diff] [review]
Full patch

r+sr=rbs based on comment 7 and comment 9.

Comment 12

[Simon Montagu :smontagu]

Comment on attachment 122607 [details] [diff] [review]
Full patch

Requesting a= for 1.4 final.

Comment 13

[Asa Dotzler [:asa]]

Comment on attachment 122607 [details] [diff] [review]
Full patch

a=asa (on behalf of drivers) for checkin to 1.4.

Comment 14

[Matitiahu Allouche]

I beg to differ with Simon Montagu's understanding of the Unicode Standard TUS 
3.0 section 13.3.  The last sentence says:  "This state (nominal) is the 
default state in the absence of any numeric shape selector or a higher-level 
protocol."  This legitimates reliance on a higher-level protocol, and 
Mozilla's preferences can definitely be considered such a higher-level 
protocol.  So whatever appears there for numeric shaping should be ok from a 
Unicode conformance point of view.
I suggest to set the default to the option that would be favored by the 
largest number of Arabic users.  I guess that would be contextual shaping, but 
Arabic experts may submit different suggestions.

Comment 15

[Simon Montagu :smontagu]

My interpretation of the Unicode Standard implicitly takes into account the
acceptability of using Mozilla's preferences and the system settings as
higher-level protocols: otherwise I would have advocated removing this
preference altogether.

I don't believe that Mozilla should silently impose a higher-level protocol on
the user by setting the default to contextual shaping. Users who want contextual
shaping can still set the preference back, and distributors of localized
versions can change the default in their package. 

We should of course add a release note so that users will be aware of the
change, and to make the transition less painful we should also fix bug 151374
before the next release.

Comment 16

[Simon Montagu :smontagu]

Fix checked in

Comment 17

[Rasha Morgan]

Netscape/Mozilla try to be system independent, so to solve this they defaulted 
to contextual shaping, this might not be the best option in some Arab 
countries, so what can be done is to default to contextual when the default 
browser locale is for one of the Arab countries which use National digits (i.e. 
Hindi Numerics like Egypt). Or optimum to add in the preference an item for the 
Arabic numeric shapes for the user to choose.