161137 - MathML renders wrong symbols if TeX fonts are installed

Status: RESOLVED WONTFIX

(Core :: MathML, defect)

Description

[Henri Sivonen (:hsivonen)]

Build ID: 2002080503 FizzillaCFM

Reproducible: Always.

Steps to reproduce:
1) Install the versions of Mathematica fonts. I used the versions that came with
the Classic version of Mathematica.
2) Install the Type 1 versions of Computer Modern (OzTeX or Textures
package--doesn't matter) from AMS.
3) Load http://www.mozilla.org/projects/mathml/start.xml

Expected results:
Expected a rendering similar to the reference screenshots.

Actual results:
The rendering falls apart and wrong glyphs are shown.

Additional info:
The rendering is correct with the Mathematica fonts but without the TeX fonts.

Comment 1

[Henri Sivonen (:hsivonen)]

Attachment: Screenshot: Mathematica fonts installed no TeX fonts

Comment 2

[Henri Sivonen (:hsivonen)]

Attachment: Screenshot: Mathematica fonts and TeX fonts

Comment 3

[Damir Perisa]

MathML-fonts and full LaTeX2e installation (all TeX fonts) installed
-using win32, build 2002080504 (no problem)
-using linux kernel2.4.16 same build (no problem)

i'm not used with MacOSX but i work with linux and the font-management must be
simmilar

try to check if fontnames are simmilar or equal (Mathematica and TeX ones) 

Comment 4

[rbs]

> 2) Install the Type 1 versions of Computer Modern

Ah, if that's what you guys have been doing then it calls for troubles.

The problem with these TeX fonts is that the encoding is different when you use
Type1 or TrueType. The ucvmath module has an #ifdef PLATFORM to pick one of the
two:

On Windows/Mac/OS2, it picks the TrueType encoding.
On Linux/Unix, it picks the Type1 encoding.

So if one subsequently uses a Type1 version instead of TrueType, the mismatch
arises. Conversely, Linux people often reports strange rendering because they
install the TrueType versions instead of Type1 - bug 35236.

The difficulty of fixing the problem lies in the fact that GFX doesn't know
which physical font was actually loaded by the OS. Indeed, if that was possible
to know that the loaded font is Type1 or TrueType, then the right encoding could
be associated on the fly.

In the meantime, try the TrueType versions to see if that solves the problem.
(There is no problem with the Mathematica fonts because the encoding is the
same.)

Comment 5

[rob raguet-schofield]

>Indeed, if that was possible to know that the loaded font
>is Type1 or TrueType, then the right encoding could be 
>associated on the fly.

actually, there is a way to tell.  i'm don't have my code in front of me, but i'll 
post something in the morning.

Comment 6

[rbs]

In that case, the Mac could be protected once for all from this problem.

If that was possible on Linux too, then a protection could be added there as 
well.

Comment 7

[Henri Sivonen (:hsivonen)]

Attachment: Screenshot: Mathematica and BaKoMa fonts

Comment 8

[rbs]

Do you "recover" the character maps of the TrueType TeX fonts when you visit:
http://www.mozilla.org/projects/mathml/fonts/encoding/

Comment 9

[rob raguet-schofield]

Attachment: code to determine at runtime if a given font is a TrueType font or not

perhaps when we create the converter object we could pass in what we know about
the font's format as another parameter...

Comment 10

[rbs]

Attachment: Pick TrueType or Type1 encoding on the fly

Iteration to illustrate how to proceed, missing the CopyCString2Pascal()
business. schofield, something for you to shew upon.

However, it is still not clear to me if installing the TrueType versions fixed
the problems (comment 8). There is something else to note. Even if the TrueType
font is there, there are other internal data in the font file that are meant
for the Mac or_exclusive_ Windows. So, if installing the TrueType versions
didn't fix the problem (e.g., the internal data for the Mac are missing), we
may have to request people to exclusively install Type1. In which case, this
elaborate patch won't be necessary. It will only be necessary to s/.ttf/.t1/g
in fontEncoding.properties and adjust the #ifdef in ucvmath.

Comment 11

[rbs]

Attachment: loose the #ifdef and fix the makefiles

Comment 12

[Henri Sivonen (:hsivonen)]

> Do you "recover" the character maps of the TrueType TeX fonts when you visit:
> http://www.mozilla.org/projects/mathml/fonts/encoding/

With the BaKoMa fonts, some chars show up correctly (mostly the first halves of
the fonts). Greek chars tend to fall back to Lucida Grande. Some chars show up
as rectangles.

Comment 13

[rbs]

Now that bug 161481 is over, it would be great to finish off this one... Are you
willing to do some experiments?

I am going to attach a patch to try with the Type1 TeX fonts. The whole point of
the experiment is to confirm/deny the suitability of these versions on the Mac.
These fonts are already known to work fine on Unix. So if it turns out that they
are also okay out-of-the-box on the Mac, then these could be recommended by
default, and the code might then take step to do the detection properly.

Comment 14

[rbs]

Attachment: patch to try out the Type1 TeX fonts

Comment 15

[rbs]

The patch isn't quite ready for review yet. I am just calling on Mac folks with 
Mac builds, rob, hsivonen, for an initial attempt to hook Type1 TeX fonts, and 
see how they fare.

Comment 16

[Henri Sivonen (:hsivonen)]

I applied the patch and rebuild intl and gfx. I installed Type 1 versions of the
CM fonts and uninstalled the BaKoMa fonts. I rebooted to make sure the BaKoMa
fonts weren't cached anywhere. I'm still seeing gibberish with the CM fonts
installed. :-(

Comment 17

[rbs]

Do you have any utility on the Mac to graphically visualize the character map of 
a font? If so, could you attach a screenshot of what you see on, say, CMSY10?

Comment 18

[rob raguet-schofield]

Attachment: cmsy10 true type glyph screenshot

did you want the true type, type 1, or both?

Comment 19

[rbs]

for completeness, let's see the true type too.

Comment 20

[rbs]

s/true type/type 1/

Comment 21

[rob raguet-schofield]

hmmm.  i downloaded the type 1 fonts from ftp://ftp.ams.org/pub/tex/psfonts/cm/cmps-
unix.tar.gz but i can't seem to get them to work.  could someone who has them working 
point me in the right direction?

Comment 22

[Henri Sivonen (:hsivonen)]

I think OS X doesn't support data fork Type 1 fonts (.pfb, .pfa). The resource
fork-based versions are ftp://ftp.ams.org/pub/tex/psfonts/cm/cmps-oztex.hqx and
ftp://ftp.ams.org/pub/tex/psfonts/cm/cmps-textures.hqx

Comment 23

[rbs]

I am a bit in the dark due to the lack of a Mac build environment. But it could 
be that there is an issue with the handling of the |script|. The following
document describes related problems:
http://www.mactech.com/articles/mactech/Vol.14/14.09/MultiscriptEnvironment/
Maybe it is necessary to "lock" the system into a "symbolic script" mode so that 
the system processes the glyph indices "as-is", and doesn't try to re-resolve 
the glyphs into something else.

===
Also, while re-reading the code, I noted that GetBoundingMetrics() is doing 
something that doesn't look quite right: Looking at 
nsUnicodeRenderingToolkit::GetTextSegmentBoundingMetrics(), it does

  ::TextFont(fontNum);
  ScriptCode script = ::FontToScript(fontNum);
  [...]
        GetScriptTextBoundingMetrics(buf, outLen, script, segBoundingMetrics);

But FontToScript() doesn't have a meaningful value for a symbolic font. So the 
value should be fetched from |fontMapping| which is already caching them. For 
example fontMapping.ConvertUnicodeToGlyphs() could be changed to return the 
script code. Or there could be a function fontmap.GetFontScriptCode(fontnum);

Then, some special-casing will be needed to avoid calling CharacterByteType() 
with BAD_SCRIPT in GetScriptTextBoundingMetrics().

Comment 24

[rbs]

mac folks, any additional insights here?

Comment 25

[rbs]

Attachment: updated: patch to try out the Type1 TeX fonts (change ucvmath.xml)

Comment 26

[rbs]

Attachment: updated: patch to try out the Type1 TeX fonts (change ucvmath.xml)

Comment 27

[Henri Sivonen (:hsivonen)]

Attachment: ATSUI's view of Type 1 CM Plain

Comment 28

[Henri Sivonen (:hsivonen)]

Attachment: ATSUI's view of ASY10

Comment 29

[Henri Sivonen (:hsivonen)]

Attachment: ATSUI's view of CIRCLE10

Comment 30

[Henri Sivonen (:hsivonen)]

Attachment: ATSUI's view of INE10

A couple of points: The glyph-to-character mappings of the LaTeX font family
fonts are completely bogus. I think we shouldn't instruct people to install
fonts like this since they might disturb ATSUI.

The mappings of Computer Modern Plain are partially correct. There are some
mismapped glyps: the ff ligature maps to U+21B5 DOWNWARDS ARROW WITH CORNER
LEFTWARDS, which explains an anomaly I though was a regression in Mozilla. The
ffl ligature maps to U+270f PENCIL. The dotless j maps to PLACE OF INTEREST
SIGN. The grave accent maps tp CHECK MARK. The glyph to the left of te
exclamation mark maps to U+0009. The right curly double and single quote maps
to ASCII  QUOTATION MARK and APOSTROPHE. The upside-down exclamation mark maps
to LESS-THAN sign. The upside-down question mark maps to GREATER-THAN SIGN. the
left double quote maps to REVERSE SOLIDUS. The dot above maps to LOW LINE. The
left single quote maps to GAVE ACCENT. Also the glyphs after z have bogus
mappings.

IMO, it would make more sense to fix the font than to try to special-case it in
Mozilla, because it would disturb ATSUI in other apps anyway.

Comment 31

[Henri Sivonen (:hsivonen)]

And the reason why teh Wolfram fonts don't cause trouble with other apps is that
as far as ATSUI is concerned, all the glyphs of those fonts are mapped to PUA
code points.

Comment 32

[Henri Sivonen (:hsivonen)]

Attachment: CMR TTF encoding page with Jaguar + Mathematica fonts

Comment 33

[Henri Sivonen (:hsivonen)]

Attachment: CMR T1 encoging page with Jaguar + Mathematica fonts

Comment 34

[Henri Sivonen (:hsivonen)]

Attachment: CMMI TTF encoding page with Jaguar + Mathematica fonts

Comment 35

[Henri Sivonen (:hsivonen)]

Attachment: CMMI T1 encoding page with Jaguar + Mathematica fonts

Comment 36

[Henri Sivonen (:hsivonen)]

Attachment: CMSY TTF encoding page with Jaguar + MAthematica fonts

Comment 37

[Henri Sivonen (:hsivonen)]

Attachment: CMSY T1 encoding page with Jaguar + Mathematica fonts

Comment 38

[Vincent Lefevre]

Is it related to bug 147704? I don't know what fonts are installed for TeX on my
machine, but I have the packages from the Debian distribution.

Comment 39

[Henri Sivonen (:hsivonen)]

The screenshots I've attached show two things: The TeX fonts are
ATSUI-incompatible. However, their character repertoire is for the most part
covered by the fonts that come with OS X (at least Jaguar).

Since the fonts could cause trouble with other apps, I think we shouldn't
instruct users to install the fonts and, therefore, shouldn't try to work around
the issues in Mozilla. I think it would make more sense to make the fonts
ATSUI-compatible by importing the Type 1 outlines in a properly coded OpenType
wrapper. If I understood the AMS copyright statement correctly, doing that is OK
if the new font isn't misrepresented as an AMS font. However, I don't know what
tools or skills are needed for making an OpenType fonts out of premade outlines.
ATSUI seems to be aware of code points up to 2FFFF, so coding the non-BMP chars
probably isn't a problem from the ATSUI point of view, although it could be from
the Mozilla point of view.

Would this make sense?
 * Making Mozilla not complain if the TeX fonts are missing.
 * Documenting that the user should not install the old TeX fonts on Mac.
 * Trying to figure out how to convert the fonts to properly mapped 
   OpenType fonts.

Disclaimer: I haven't tested any of this stuff on Mac OS 9.

(This bug and bug 147704 don't seem to be related.)

Comment 40

[rbs]

> A couple of points: The glyph-to-character mappings of the LaTeX font family
> fonts are completely bogus. I think we shouldn't instruct people to install
> fonts like this since they might disturb ATSUI.

These fonts are "symbol" fonts and so they don't have any predefined mapping 
other than as defined by the application. It seems to me that it is ATSUI (like 
the Windows GDI or the X-font server) who should be special-casing these fonts 
by labelling them as "symbol", in the same category as other symbol fonts such 
as Webdings, Wingdings.

To stop Mozilla from popping up the missing font dialog about TeX fonts, just 
set this pref in your prefs.js (or your user.js):

user_pref("font.mathfont-family", "Math1, Math2, Math4, Symbol");

Comment 41

[rbs]

*** Bug 171423 has been marked as a duplicate of this bug. ***

Comment 42

[Paul Libbrecht]

Attachment: 10.1, no TeX fonts, broken notin

I am not sure where this is all but since I've been given the hint to remove
texfonts, so I did... but... fonts are still corrupted.
Attached is a screenshot of 
http://www.w3.org/Math/testsuite/testsuite/Content/TheoryOfSets/notin/notin2.xml
which definitely shows that there a few corrupted things.
Note: I'm not on Jaguar (yet), I have the Math 1 to 5 and not TeX font
installed...
Does 10.2 make the difference ?

Comment 43

[rbs]

Is there a bug in the UMSS? Using the "View MathML Source" feature on Win32 with
the example, I see that the presentation MathML that is fed to Mozilla is:

<math xmlns="http://www.w3.org/1998/Math/MathML">
  <mrow>
    <notin/>
    <mo/>
    <mfenced open="(" close=")" separators=",">
      <mrow>
        <mi>a</mi>
      </mrow>
      <mrow>
        <mi>A</mi>
      </mrow>
    </mfenced>
  </mrow>
</math>

This renders as "(a, A)".

Comment 44

[YAMASHITA Makoto]

Attachment: special casing for TeX fonts to use ATSUI instead of raw QD

This patch does a special casing for TeX fonts to use ATSUI engine instead of raw QuickDraw functions, in GetTextSegmentDimensions, GetTextSegmentBoundingMetrics and DrawTextSegment.
The cause of this problem was that QD's DrawText expects the data to be encoded according to the current font's script system while ucvmath output was UCS2-truncated-to-8bit.
I first thought of using SetScriptVariable to set the script's encoding to smUninterp, but it didn't work.
This patch is not without problems: e.g. performance hurt by fontname comparison, bad layout for some glyphs including u2212 (minus sign, though u002D is not affected) and u00B1 (plusminus)....

Comment 45

[rbs]

> I first thought of using SetScriptVariable to set the script's encoding to
> smUninterp, but it didn't work.

1/ If you do ::FontToScript(fontNum) when fontNum is a Mathematica font, do you get smUninterp as the script? If not, could you try to set the value that you see with from Mathematica fonts on TeX fonts too?

2/ Is there any difference when using the Type1 versions? Note that the font-encoding mappings are very different. So you will need some of the earlier patches here to map the Type1 versions to the right glyphs.
http://www.mozilla.org/projects/mathml/fonts/encoding/cmsy.html

In particular, looking at the listings of TrueType vs Type1 on the page, minus and plusminus have totally different encoding points:

TrueType:                                Type1
  0xA1  0x2212, ...  #Minus sign            0x00  0x2212, ...  #Minus sign
  0xA7  0x00B1  #Plus-minus sign            0x06  0x00B1  #PLUS-MINUS SIGN
  ^^^^                                      ^^^^

Comment 46

[YAMASHITA Makoto]

Both cmsy10 and Math1 have script code 0 and encoding 0, lang 0
Here, "script code" means the result of FontToScript(fontNum), "encoding" the result of
::GetScriptVariable(smCurrentScript, smScriptEncoding);
and "lang" the result of 
::GetScriptVariable(smCurrentScript, smScriptLang);
The rendering does not change when I change these to encoding=32 (smUninterp), lang=32767 (langUnspecified) for both fonts.
please wait a bit for type1s.

Comment 47

[YAMASHITA Makoto]

I've installed the CM-type1-for-texture distribution from ams ftp
ftp://ftp.ams.org/pub/tex/psfonts/cm/
and uninstalled bakoma.
looks like Firefox 1.5 rc1 renders MathML just fine with them, so that the only MathML bug remaining in that release is the "Symbol missing" alert...

Comment 48

[rbs]

So you don't get the gibberish that Henri mentionned in Comment #16? Maybe you guys have different OS versions or something? Anyway, it answers the question that these truetype fonts are not appropriate for the Mac -- perhaps because they miss bits of the internal font data that the MacOS wants as I speculated earlier.

Let's forget about the TrueType, and let's focus now on enabling the Type1. Note that they still cannot work properly without the .t1 converters in ucvmath. You can check this by removing the Mathematica fonts so that they don't kick in, and/or setting the font.mathfont-family to "CMSY10, CMEX10", and visiting the torture test. 

Also, you should "recover" the character maps of the TeX fonts when you visit:
http://www.mozilla.org/projects/mathml/fonts/encoding/

Comment 49

[rbs]

BTW... just to spell it out, be sure not to let your special-casing patch in the way, of course.

Comment 50

[YAMASHITA Makoto]

cm-type1-for-texture fonts are working without Mathematica fonts, once they are recognized by the OS (for reproducers: I had to copy the file "CM/PS screen" from "CM screen" folder, togther with CMEX10...CMSY10 to make them recognized)
cmsy10-type1-for-texture has a cmap table declaring "Mac platform, Roman script, format 0 (256 glyphs array)" with e.g. 0xA1 - Minus glyph correspondence.
IMU this is why it goes well with the truetype conveter (and treating its output as a MacRoman data).

Comment 51

[rbs]

Do you mean that the character map of, say Type1 CMS10, is *exactly* 
attachment 95294 [details]? or
http://www.mozilla.org/projects/mathml/fonts/encoding/cmsy-ttf.gif

Comment 52

[YAMASHITA Makoto]

Attachment: mapping of cmsy for texture

With ATSFontGetTable for CMSY10 I get two cmap tables, one for "Unicode platform" and another for "Mac platform." The Mac table agrees with that of attachment 95294 [details] when you read that as "Minus glyph is in the place of MacRoman-0xA1, or U+00B0, BigCircle glyph is in the place of MacRoman-0xB0 or U+221E..."
I put the whole mapping of Mac table in the attachment. Do you need the code generating it?

Comment 53

[rbs]

It appears therefore that those two tables were deliberately put in the font data to enable using the same font on the Windows platform too.

I assume that it is the Unicode table that is 0-based and maps the type1 converter in ucvmath, right? In the case of CMR10 (which also has a type1 converter in ucvmath), it was reported that it is indeed 0-based, see attachment 96705 [details]. Is it possible to use these Unicode maps instead in the measuring and drawing functions? This way, it won't be necessary to setup brand new font mapping tables in ucvmath to resolve the issues with the minus, big circle, etc.

[I gather from this thread that the additional map in the type1 file must have been added after-the-fact. The initial 0-based mappings cannot work on Windows because this OS reverses some slots in the character map. For example, position 0 is reversed for the replacement glyph, etc.]

Comment 54

[YAMASHITA Makoto]

Attachment: CMSY, Math1 and Symbol mappings

attachment 96705 [details] does not reproduce for me. CMR10's A appears at U+0041 in the Characters Panel.
Unicode map of CMSY is just the ditto of mac table (it's sparse due to MacRoman-Unicode conversion) The "glyph index" (1 based) matches ucvmath-type1 output offset by 1. Glyph 0 means "nonexistent" as in Windows.
Interestingly, Math1 and Symbol do more tricky things: they have 0x03B1->'alpha' mapping in the unicode table and 0x61->'alpha' in the mac table (although their mac table claim to be in Roman script code...), see my attachment.

Comment 55

[rbs]

Why is that things seem to work in your system whereas others reported the gibberish. What is in your system that others do not have?!?

Can other Mac people test the fonts again to confirm that they now works for them too?

Comment 56

[rbs]

jshin, it appears from makoto observations that Type1 TeX now work just fine out-of-the-box on the Mac. Is that your experience too?!?

Comment 57

[Jungshik Shin]

Attachment 96705 [details] seems to have been taken with 'character pallet' with the glyph view selected and I got exactly the same result, which is identical to the visual encoding of CMR10 except for the off-by-one difference as already reported in comment #54 and shown in attachment 96705 [details]. I'm not sure what Yamashita-san meant when he wrote that he couldn't reproduce attachment 96705 [details].

 Latin letters and numbers in the ASCII range have the correct mapping (I can get their Unicode positions when I hover the cursor over), but Greek letters have completely wrong Unicode positions mapped. Presumably, this means that ATSUI would be 'disturbed' with CMR installed as Henri wrote.

I wonder how attachment 95294 [details] was obtained.  

Comment 58

[YAMASHITA Makoto]

Oh, I just didn't know the existence of the glyph view in the pallet.
That surely reproduces in the glyph view, but the view looking that way is not a problem itself, right? It's true that CM fonts have *incorrect* unicode mappings, but I don't think this affects outside-MathML experience. Naturally MacRoman characters are covered by so many fonts, it's less likely CM glyphs are used for them.
Moreover I don't think this "incorrect mapping" theory explains the cause of the original problem... They do match mozilla's expectation (if we interpret the output of ucvmath as the codepoints for the mac table).
So, if CM fonts work for mozilla MahML now, the options are
1) Keep telling people not to install CM fonts, because theoretically they *can* butter the rendering of macroman latter half chars. And look for PUA mapped CM fonts.
2) Close this bug
I definitely prefer 2.

Comment 59

[rbs]

> I wonder how attachment 95294 [details] was obtained.  

It seems from the screenshot to have been produced by a freeware. Maybe (highly speculative maybe), they may have made their code to scan the Windows data in the TrueType font as well. I have been speculating that the Windows TrueType versions may not be woking on the Mac due the same issue that you mentioned in bug 254585 comment 6. 

But I am confused about the conflicting messages from you guys with the Type1 versions. First, the one-off shift creates a difference re: the .t1 converter used in Unix non-Xft builds. Next, the "disturbance" of ATSUI makes me wonder what other applications are doing. Surely, TeXtures and friends do work on the Mac. What are they doing that we _cannot_ do? Next, what makes Makoto see that Mozilla renders MathML fine out-of-the-box for him?

Comment 60

[YAMASHITA Makoto]

To make it sure, we aren't using ATSUI for CM fonts. When CM* is chosen the string data get converted and go to DrawScriptText, l 1202 of nsUnicodeRenderingToolkit.cpp, that in turn calls QD's ::DrawText that refer to the "macroman" cmap table.

Comment 61

[YAMASHITA Makoto]

jshin, it's not clear from your comment what happens to you when you open
http://www.mozilla.org/projects/mathml/start.xhtml
with Firefox rc3 (or whatever recent official 1.5 release you have) and cm-type1-for-texture installed on your system. Now I have two working cases other than my own mac, so please see how it goes. If that's fine, refrain of #58. If not, is it like attachment 94050 [details]?

Comment 62

[Ryan VanderMeulen [:RyanVM]]

Is this WONTFIX at this point?

Comment 63

[Frédéric Wang (:fredw)]

(In reply to comment #62)
> Is this WONTFIX at this point?

I suggest closing this bug, since TeX fonts are no longer the recommended fonts for mozilla 1.9:

http://www.mozilla.org/projects/mathml/fonts/