78201 - [BiDi]:Arabic 2 byte fonts don't seem to render / Add support for iso8859-6.8x fonts

Status: RESOLVED FIXED

(Core :: Layout: Text and Fonts, defect)

Description

[Prabhat Hegde]

On my bidi builds on both Solaris and Linux [Trunk nightly (04/29) + 
sources from erik@netscape], i am not able to view any arabic output. 
All i get is glyph for invalid char (?). The font i am using is a 2 byte 
font based on iso-8859-6. [Please email me for the font as i am not sure 
if i can attach it here]. The problem may be with 1 byte input-2 byte 
output (just guessing at this point).

Test URL's : www.ayna.com, www.al-jazirah.com

Comment 1

[nhottanscp]

Reassign to [e-mail].

Comment 2

[Prabhat Hegde]

Attachment: Changes against 04/20 trunk obtained from erik needed to build bidi

Comment 3

[Prabhat Hegde]

Attachment: The two byte arabic fonts that i tested with.

Comment 4

[Frank Tang]

What is the encoding of the two bytes font?

Comment 5

[Prabhat Hegde]

Attachment: font encoding is iso-8859-6 + Arabic Presentation Forms A & B (Font Encoding File Attached)

Comment 6

[Andreas Becker]

Changing QA contact to [e-mail]

Comment 7

[Frank Tang]

move to Bidi Hebrew/Arabic component
probat- Pleaes attach your font again. Please specify mime type correctly while
you attach. Is it a zip file?

Is that true the font is encoded in Unicode ? What is the XLFD ?

Comment 8

[Frank Tang]

reassign to [e-mail]
katakai- this is a linux font issue. I think you know the font code well to make
this happen. Please work with bstell for details if you need help.
First qustion we want to know is this font a TrueType font ? a ISO-10646 font?
what is the XLFD? 
If it is a ISO-10646 font, then we will use the ISO-10646 font path in the gtk. 

Comment 9

[Zach Lipton [:zach]]

Mass-move all BiDi Hebrew and Arabic qa to me, [e-mail]. 
Thank you Gilad for your service to this component, and best of luck to you 
in the future.

Sholom. 

Comment 10

[Katsuhiko Momoi]

QA to mahar.

Comment 11

[Roland Mainz]

What is the status of this bug ? Will this be fixed for 1.0 ?

Comment 12

[Roland Mainz]

Comment on attachment 32644 [details]
The two byte arabic fonts that i tested with.

Uhm... this tar.Z misses the matching fonts.dir file... ;-(

Comment 13

[Simon Montagu :smontagu]

*** Bug 136101 has been marked as a duplicate of this bug. ***

Comment 14

[Simon Montagu :smontagu]

It looks as if we have to add some special voodoo to use the font encoding from
attachment 36498 [details].

Comment 15

[Prabhat Hegde]

hi simon,

I believe this is because the font calls itself "iso8859-6" while actually
being unicode encoded "iso10646". 

prabhat.

Comment 16

[Roland Mainz]

prabhat:
Do you know which encoding the following fonts (diestributed with Solaris) have:
1. /usr/openwin/lib/locale/ar/X11/fonts/TrueType
2. /usr/openwin/lib/locale/ar/X11/fonts/Type1
?

Comment 17

[Prabhat Hegde]

Attachment: arabic fonts.dir and ttmap for Solaris

hi roland,

as i mentioned it is iso10646. As mentioned earlier it calls 
itself "iso8859-6". I am attaching arabic_font_info.tar.Z which
has fonts.dir and also ttmap file for your info.

prabhat

Comment 18

[kill this account]

> Do you know which encoding the following fonts (diestributed with Solaris) 
> have:
> 1. /usr/openwin/lib/locale/ar/X11/fonts/TrueType
> 2. /usr/openwin/lib/locale/ar/X11/fonts/Type1

Are these fonts or are these directories?

Comment 19

[Roland Mainz]

Brian Stell wrote:
> > 1. /usr/openwin/lib/locale/ar/X11/fonts/TrueType
> > 2. /usr/openwin/lib/locale/ar/X11/fonts/Type1
>
> Are these fonts or are these directories?

This are dirs which contain fonts for the X11 system (Solaris puts the
X11-related code into /usr/openwin - but that's another story).
Solaris seperates it's fonts per locale and then per font type ("ar" == arabic
locale, "TrueType"/"Type1" are the font types).

Comment 20

[Roland Mainz]

prabhat:
I have still no luck, the '?' do not disappear...

.. and hacking the fonts.dir and renaming the fonts to
-- snip --
NASKHMT.ttf -monotype-naskh-medium-r-normal--0-0-0-0-p-0-iso10646-1
NASKHBD.ttf -monotype-naskh-bold-r-normal--0-0-0-0-p-0-iso10646-1
-- snip --
causes Xsun to ask for a matching encoding file ("Cannot find encoding file for
iso10646-1" in /var/dt/Xerrors).

Looks I have to seek little bit harder for finding a solution...

Comment 21

[Prabhat Hegde]

hi roland,

I don't think hacking font-encoding will help. As you mentioned, you also
need to add ttmap for the encoding, add entry in ttmaps.dir and so on.

One idea is to change the converter currently used in nsFontMetrics for this
font for Solaris only. ie iso8859-6 'SingleByteconverter' to be enhanced to
handle 8859-6 + unicode encoded presentation forms A & B.

prabhat. 

Comment 22

[Roland Mainz]

Prabhat Hegde wrote:
> I don't think hacking font-encoding will help. As you mentioned, you also
> need to add ttmap for the encoding, add entry in ttmaps.dir and so on.
>
> One idea is to change the converter currently used in nsFontMetrics for this
> font for Solaris only. ie iso8859-6 'SingleByteconverter' to be enhanced to
> handle 8859-6 + unicode encoded presentation forms A & B.

I would perfer a solution which is not Solaris-specific since many people use
SPARCs and use Xterminals or copy these fonts around...
Adding special encoder support and treating the ISO-8859-6 fonts always as
doublebyte fonts may be a solution.
Looks we simple need someone who hacks the ISO-8859-6 converter code a little
bit... :)

Comment 23

[Roland Mainz]

prabhat:
Do you know whether the PostScript Type1 fonts in
/usr/openwin/lib/locale/ar/X11/fonts/Type1/ contain the presentation forms a&b,
too ? If yes - how can I use them from the X11 API ?

Comment 24

[kill this account]

> One idea is to change the converter currently used in nsFontMetrics for this
> font for Solaris only. ie iso8859-6 'SingleByteconverter' to be enhanced to
> handle 8859-6 + unicode encoded presentation forms A & B.

How would this work?

Would it be applied to all iso8859-6 fonts?

Would there be a different encoding; eg: "iso8859-6x" ?

Comment 25

[Frank Tang]

Prabhat: please please please please do not attach file as zip or 
application/octet-stream. Attach seperate attachment for the thing you want to
attach- one for each file. please. I won't have the right tool to look at your
attachement from my Mac. 

Comment 26

[Masaki Katakai]

Hmm, I could not still find out how to browse Arabic with iso-8859-6 fonts.
Have anyone succeeded?

- Sun's ar fonts - failed as this bug report
- Tried langbox arabic fonts from     
  http://www.langbox.com/AraZilla/linux/arafontfull-1.2-4.i386.rpm 

Can anyone know the location of correct iso-8869-6 fonts?

There is two way I have succeeded to display arabic characters
without iso-8859-6 fonts,

1. use FreeType as fontpath to /usr/openwin/lib/locale/ar/TrueType
   Great!!

   However, there is not priting solution for now if we use TrueType.
   We's planning to suppport Xprint for Arabic. Xserver should
   be able to load and displayt Arabic glyphs.

2. Use iso10646-1 fonts 

   I tried arabic bdf fonts from 

   http://crl.nmsu.edu/~mleisher/download.html

   It works well.

Comment 27

[Masaki Katakai]

Attachment: not working - langbox 8859-6x fonts from http://www.langbox.com/AraZilla/linux/arafontfull-1.2-4.i386.rpm

Comment 28

[Masaki Katakai]

Attachment: not working - sun's 8859-6 fonts of Solaris

Comment 29

[Masaki Katakai]

Attachment: working - arabic24 10646 fonts from http://crl.nmsu.edu/~mleisher/download.html

Comment 30

[Masaki Katakai]

Attachment: not working - using sun's ar fonts as 10646 fonts by adding some encoding files into Solaris, compare arabic24 10646 fonts, some diffs on 0x06xx range

Comment 31

[Roland Mainz]

Comment on attachment 83639 [details]
not working - langbox 8859-6x fonts from http://www.langbox.com/AraZilla/linux/arafontfull-1.2-4.i386.rpm 

AFAIK we do not have any mapping for "iso-8859-6x" fonts in our fontmetrics
code

Comment 32

[Roland Mainz]

What about treating ISO-8859-6 fonts like iso10646-1 in our fontmetrics system,
e.g. use the X11 API to query which glyphs are present in the font and not try
to "guess" it based on the encoding ?
That seems to work for the Solaris 2.8 fonts from
/usr/openwin/lib/locale/ar/X11/fonts/TrueType (at least "xfd" treats them as
16bit font then...).

Comment 33

[Roland Mainz]

Attachment: Screenshot of "pfaedit" viewing the fonts "SHA1____.PFA" and "SHA2____.PFA" from /usr/openwin/lib/locale/ar/X11/fonts/Type1/

Does anyone know which encoding standard the fonts "SHA1____.PFA" and
"SHA2____.PFA" from /usr/openwin/lib/locale/ar/X11/fonts/Type1/ implement ?

Comment 34

[Simon Montagu :smontagu]

These encodings are new to me and certainly not any of the common standard or
proprietary encodings from ISO, ASMO, Microsoft, IBM or Apple, but each one is a
folding of part of the Unicode set into 8 bits (assuming that they both start
from 0x00 at the top left).

SHA1____.PFA is like a variant of ISO-8859-6: 

0x00-0x7F - equivalent to ASCII
0x80-0xEF - equivalent to Unicode 0x0600-0x066F

SHA2____.PFA maps the presentation forms from Unicode 0xFF70-0xFEFF to 0x70-0xFF

Comment 35

[Roland Mainz]

Simon Montagu wrote:
> These encodings are new to me and certainly not any of the common standard or
> proprietary encodings from ISO, ASMO, Microsoft, IBM or Apple, but each one is 
> a folding of part of the Unicode set into 8 bits 
> (assuming that they both start from 0x00 at the top left).

Mhhh...

... what above creating a new X11 font encoding (we may call it
"sun.unicode.plane" :) which lists the unicode plane and offset being used in
the font (this would be sufficient for 8bit fonts; 16bit fonts would need
multiple entries in fonts.dir) ?

Comment 36

[Roland Mainz]

Clarification:
We are talking about two things here:
Solaris ships with two kinds of Arabic fonts:
1. TrueType fonts in /usr/openwin/lib/locale/ar/X11/fonts/TrueType/ - they are
listed as *-iso-8859-6 fonts in
/usr/openwin/lib/locale/ar/X11/fonts/TrueType/fonts.dir - but the Solaris
TrueType font engine idetifies them correctly as 16bit fonts.
2. PS Type 1 fonts in /usr/openwin/lib/locale/ar/X11/fonts/Type1/ - which seem
to represent the unicode blocks for arabic (see comment #34) squished into 8bit
fonts

For [1] I propose to check whether ISO-8859-6 fonts are 16bit X11 fonts. If they
are 16bit fonts we should assume that they have the arabic glyphs in the
expected places
For [2] I propose the idea listed in comment #35 (new X11 encoding scheme
"sun.unicode.plane")

Comment 37

[Roland Mainz]

I filed a seperate bug (bug 159430 ("[RFE] Add support for X11 fonts which
represent single unicode blocks")) for the discussion around [2] from comment
#36

Comment 38

[Roland Mainz]

BTW: I filed bug 158894 ("RFE: Document how to treat TrueType fonts as
iso10646-1 with Solaris/Xsun") to document how users can use TrueType fonts with
iso10646-1 encoding on Solaris/Xsun to view/print Arabic pages.

Comment 39

[kill this account]

> 2. PS Type 1 fonts in /usr/openwin/lib/locale/ar/X11/fonts/Type1/ - which seem
> to represent the unicode blocks for arabic (see comment #34) squished into 
> 8bit fonts

> For [1] I propose to check whether ISO-8859-6 fonts are 16bit X11 fonts. 

Is it valid to use ios8859-x for a 16 bit font?

> If they are 16bit fonts we should assume that they have the arabic glyphs in 
> the expected places

Is there any standard that describes this? Doing this if no standard exists
makes me very nervous.

> For [2] I propose the idea listed in comment #35 (new X11 encoding scheme
> "sun.unicode.plane")

  0x00-0x7F - equivalent to ASCII
  0x80-0xEF - equivalent to Unicode 0x0600-0x066F

  ...Unicode 0xFF70-0xFEFF to 0x70-0xFF
  (ignore the typo)

These do not have a simple mapping which the suggestion in comment #35 assumes.

Comment 40

[kill this account]

Just to be clear: both ftang and I are happy to support these fonts and if
there is no standard we can just start the encoding-registry name with 
something like "x_" to indicate this is a non standard encoding.

Comment 41

[kill this account]

from email, Prabhat Hegde <[e-mail]> writes:

The arabic language folks at Sun finally fixed their font encoding which 
they say is Stds based (LangBox). This font is called iso8859-6.8x and is
also supported by Gnome/Pango. How do i get this font to appear in the
arabic font selection on my Solaris box?
  Edit->Preferences->Font->Languages->Arabic
Currently only iso8859-6 based font shows up in the selection box.

Comment 42

[kill this account]

blizzard: here is a small task where you could get you feet wet working on
fonts.

Comment 43

[Prabhat Hegde]

Thanks to quick education from ftang, simon and myself found the following -

A> A UnicodeToLangBox converter already exists in ucvlatin written long-long ago
by ftang which can be re-used after synching with latest code-base.
B> This converter needs to be modified to handle Arabic Pres form B.

Finally, the converter is not as complicated as when frank wrote it since arabic
presentation forms are already generated by layout. Hence only mapping but no
shaping logic is needed. I am on it right now and should have a patch by tomorrow.

Comment 44

[Simon Montagu :smontagu]

Attachment: Patch

This is mostly Prabhat's work with some contributions from me. intl/uconv
changed under me while I was working on it, so there may be some oddities.

Comment 45

[Simon Montagu :smontagu]

Attachment: Patch with all the files in it this time

Comment 46

[Roland Mainz]

Over to smontagu...

Comment 47

[Roland Mainz]

Comment on attachment 101339 [details] [diff] [review]
Patch with all the files in it this time

r=[e-mail]

Patch builds&works, I can see the arabic fonts properly and print them (with
Xprint), too (assuming that the matching fonts&*.enc&*.ttmap files are
available).

Comment 48

[Roland Mainz]

I forgot one minor nit (no need to file a new patch for that):
-- snip --
-  NS_IMETHOD FillInfo(PRUint32* aInfo);
-};
-
+static PRUnichar uni2lbox [] = 
+ {
+  0xC1,   /* FE80 */
+  0xC2 , 
+  0xC2 ,
-- snip --

Can you make that array |const|, please ?

Comment 49

[Roland Mainz]

Attachment: Page one of http://ayna.com printed with 2002-09-29-08-trunk+patch from attachment 10133 (colors are wrong, that's a bug of Solaris's PostScript viewer sdtimage)

Comment 50

[Roland Mainz]

Attachment: Page one of http://ayna.com printed with 2002-09-29-08-trunk+patch from attachment 10133 (JPEG to bypass the attachment size limit of bugzilla)

Comment 51

[kill this account]

would you add a font-lang group for arabic?

 nsFontLangGroup FLG_JA =      { "ja", nsnull };
 nsFontLangGroup FLG_KO =      { "ko", nsnull };
+nsFontLangGroup FLG_AR =      { "ar", nsnull };
 nsFontLangGroup FLG_NONE =    { nsnull , nsnull };

Comment 52

[Roland Mainz]

Brian Stell wrote:
> would you add a font-lang group for arabic?

... which reminds me that we didn't add one for indic ("hi-IN") either... ;-(

Comment 53

[Simon Montagu :smontagu]

Comment on attachment 101339 [details] [diff] [review]
Patch with all the files in it this time

Thanks for the r=, but the patch needs polish at the very least. Also, we have
a problem with lam-alef ligatures (as attachment 101369 [details] shows) which must be
fixed.

Comment 54

[Prabhat Hegde]

Yup - i think it still needs some bug-fixing. Its not just combo(LAM+ALEF) case.
I am not a native user or expert so the only way i can tell is by comparing with
Windows version and also IE. I tried the following sites:

assafir.com (MAC-ARABIC) this was the best.
aljazeera.com (Windows)
bbc.co.uk -> Arabic (Unicode encoded)

However, it is worthwhile to integrate it so that arabic testers on Solaris can
start testing. I can tell that selection is badly broken.

Comment 55

[Roland Mainz]

Simon Montagu wrote:
> Thanks for the r=, but the patch needs polish at the very least. Also, we have
> a problem with lam-alef ligatures (as attachment 101369 [details] shows) which must be
> fixed.

I thought this is a problem with the Solaris fonts - or did I understand you
wrong here ?

Comment 56

[Roland Mainz]

Prabhat Hegde wrote:
> However, it is worthwhile to integrate it so that arabic testers on Solaris 
> can start testing. I can tell that selection is badly broken.

That's why I gave my r= for it.
The code works IMHO "good enougth" for trunk and we can't really kill all issues
in one step.
Without it we're completely screwed without the evil iso10646-1 fonts (there's
still the problem that the ISO8859-6.8x encoding files for Solaris+Xfree86
aren't available in the public... ;-( ).

Comment 57

[Prabhat Hegde]

Langbox encodings are publicly available:
http://www.langbox.com/arabic/FontSet_ISO8859-6-8X.html

Comment 58

[Roland Mainz]

Prabhat Hegde wrote:
> Langbox encodings are publicly available:
> http://www.langbox.com/arabic/FontSet_ISO8859-6-8X.html

... I was think about the encodings files (*.enc and *.ttmap) in this case.

Comment 59

[Simon Montagu :smontagu]

Attachment: Patch v.2 with lam-alef handling

I made the changes that Roland and Brian asked for, did some general clean-up,
and added handling for lam-alef.

This exposed a bug in nsRenderingContextGTK::GetTextDimensions: text with
lam-alef was laid out incorrectly and couldn't be selected properly, unless I
set MOZILLA_GFX_DISABLE_FAST_MEASURE in my environment. (Thanks are due to
Roland for help in identifying this problem).

Comment 60

[Simon Montagu :smontagu]

Attachment: Patch v.3

Oops! removed a printf

Comment 61

[Roland Mainz]

Roland Mainz wrote:
> Brian Stell wrote:
> > would you add a font-lang group for arabic?
>
> ... which reminds me that we didn't add one for indic ("hi-IN") either... ;-(

Filed bug 172515 ("Some nsFontLangGroup entries missing in X11 font code") for
that issue...

Comment 62

[kill this account]

The font changes look good and I am qualified to review them.
r=[e-mail] for the font changes.

I don't work in the converter code enough to be qualified to review those changes. 
Perhaps ftang can do that.

Comment 63

[Roland Mainz]

Adding dependicy to bug 172683 ("Problem with layout of Arabic lam-alef
ligatures and nsRenderingContext{GTK|Xlib}::GetTextDimensions()") since lam-alef
using the LangBox iso8859-6.8x is screwed-up due that bug...

Comment 64

[Simon Montagu :smontagu]

Attachment: Patch v.4: merged to tip (with fix for bug 172515) and added Mac build changes

Comment 65

[Simon Montagu :smontagu]

ftang, please review the converter and Mac build changes in attachment 102093 [details] [diff] [review].

Comment 66

[Frank Tang]

Comment on attachment 102093 [details] [diff] [review]
Patch v.4: merged to tip (with fix for bug 172515) and added Mac build changes

r=ftang
only one minor issue. Please  change the return to NS_OK in the following
function:
+NS_IMETHODIMP nsUnicodeToLangBoxArabic8::GetMaxLength(
+const PRUnichar * aSrc, PRInt32 aSrcLength, 
+			    PRInt32 * aDestLength) 
+{
+  *aDestLength = 2*aSrcLength;
+  return NS_OK_UENC_EXACTLENGTH;

from NS_OK_UENC_EXACTLENGTH
then you just update the patch with a has-review with it. 
The rest of the code looks good

Comment 67

[Simon Montagu :smontagu]

Attachment: Patch addressing ftang's review comments

Comment 68

[Simon Montagu :smontagu]

Comment on attachment 102096 [details] [diff] [review]
Patch addressing ftang's review comments

r=ftang per comment 66

Comment 69

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Comment on attachment 102096 [details] [diff] [review]
Patch addressing ftang's review comments

sr=roc+moz

Comment 70

[Simon Montagu :smontagu]

Fix checked in.

Comment 71

[Roland Mainz]

There is one final thing ToDo:
We need an entry in the >=1.2b release notes with the following information:
- iso8859-6.8x support added for Unix/Linux incl. Xprint
- Information where to get the iso8859-6.8x-encoded fonts from

Comment 72

[Roland Mainz]

prabhat:
Is there a way to contribute the iso8859-6.8x *.enc dir file to Xfree86.org and
create a patch for Solaris's "ar"-locale with the fixed arabic fonts and
*.enc/*.ttmap files ?

Comment 73

[Roland Mainz]

Filed release note item under bug 174672 comment #1 ...

Comment 74

[Prabhat Hegde]

Hi roland, OK - i'll look at *.enc file (as you know even minor contribution
needs to go via legal). As to Solaris, i believe Ar locale owners will create a
patch.

Comment 75

[Roland Mainz]

Prabhat Hegde wrote:
> i'll look at *.enc file
> (as you know even minor contribution needs to go via legal).

;-((

> As to Solaris, i believe Ar locale owners will create a patch.

Well, AFAIK you'll have to patch the Solaris "ar"-TrueType fonts, too - per
smontagu some glyphs are missing in these fonts (however, it should be easy to
add them via "pfaedit" (more details on demand) ... :))

Comment 76

[Roland Mainz]

Filed http://bugs.xfree86.org//cgi-bin/bugzilla/show_bug.cgi?id=420 ("RFE: Add
encodings files for Arabic LangBox encodings iso8859-6.8, iso8859-6.8x and
iso8859-6.16") to get support for these encodings in Xfree86...