534970 - (mathml-operator-dict) [MathML3] Update MathML Operator Dictionary

Status: RESOLVED FIXED

Description

[Frédéric Wang (:fredw)]

MathML 3 comes with a new format and updated entries for Operator Dictionary. Bug 519126 is about using this new compact format. The present bug aims at updating the entries in mathfont.properties.

Comment 1

[Frédéric Wang (:fredw)]

Created attachment 442182 [details] [diff] [review]
Add a perl script to compare MathML operator dictionaries

This script allows to compare our version of the dictionary against the official one. Most changes are new operators or lspace/rspace values, so I think we can take them safely. Probably we need more care for changes related to stretching.

The script is able to output the details of the differences as well as an updated version of our dictionary. Entries in differences.txt are marked with "stretching", "conflict", "new", "obsolete" so you can use "grep -A2" to select a particular subset. The updated version takes the values of the W3C dictionary, except that it keeps our "direction", preserve our comment if any (end of line after the sharp "#") and keeps the "obsolete" entries. Entries are sorted with respect to the W3C dictionary, except for the obsolete ones, which are put at the end and sorted by lexical order over the key.

Comment 2

[Frédéric Wang (:fredw)]

After applying the changes of attachment 423544 [details] [diff] [review] and attachment 441533 [details] [diff] [review] (for mathfont.properties) and comparing with http://monet.nag.co.uk/~dpc/draft-spec/appendixc.xml, the script returns:

- 92 obsolete entries (24 of them are related to stretching)
- 196 unchanged entries
- 102 conflicting entries (9 of them are related to stretching)
- 760 new entries (87 of them are related to stretching)

Comment 3

[Frédéric Wang (:fredw)]

Created attachment 442184 [details]
differences between mathfont.properties and the one of the MathML WG

Comment 4

[Frédéric Wang (:fredw)]

Created attachment 442458 [details] [diff] [review]
update some entries whose stretchiness has changed

Comment 5

[Frédéric Wang (:fredw)]

With the patch of comment 4, the differences are:

- 92 obsolete entries (24 of them are related to stretching)
- 214 unchanged entries
- 84 conflicting entries (5 of them are related to stretching)
- 760 new entries (87 of them are related to stretching)

The 5 remaining entries whose stretchiness has changed are:

[conflict][stretching] - operator.\u005F.infix (low line)
-lspace:2 rspace:2 stretchy
+lspace:1 rspace:1
--
[conflict][stretching] - operator.\u2225.infix (parallel to)
-lspace:5 rspace:5 stretchy
+lspace:5 rspace:5
--
[conflict][stretching] - operator.\u2223.infix (divides)
-lspace:5 rspace:5 stretchy
+lspace:5 rspace:5
--
[conflict][stretching] - operator.\u2216.infix (set minus)
-lspace:3 rspace:3 stretchy
+lspace:4 rspace:4
--
[conflict][stretching] - operator.\u002F.infix (solidus)
-lspace:3 rspace:3 stretchy
+lspace:1 rspace:1

Comment 6

[Frédéric Wang (:fredw)]

Created attachment 442466 [details]
differences between mathfont.properties and the one of the MathML WG

Comment 7

[Frédéric Wang (:fredw)]

Created attachment 442636 [details] [diff] [review]
Add a perl script to compare MathML operator dictionaries

There was a bug in my script for multiple char operators. The new values are:

- 54 obsolete entries (24 of them are related to stretching)
- 214 unchanged entries
- 122 conflicting entries (5 of them are related to stretching)
- 722 new entries (87 of them are related to stretching)

Comment 8

[Frédéric Wang (:fredw)]

Among the 24 obsolete entries that are stretchy, there is at least two annoying ones for the "scale strechy" patch:

operator.\u22C2.infix (⋂)
operator.\u22C3.infix (⋃)

In Joe Java's test, in "De Morgan's laws" for sets there is something like:

<mrow><mi>X</mi> <mo>=</mo> <mo>&Intersection;</mo> <mi>Y</mi>  </mrow>

Here, the &Intersection; operator is considered as infix (so stretchy and not largeop) rather than prefix (not stretchy and largeop). Without the patch, we don't see the difference. But with the patch applied, the former stretches the symbol in the vertical direction only (to cover the height of the formula) while the latter simply increases width and height by twice (it's what we want).

To fix the problem, we can:
- make these operators non-strechy. But still they won't be largeop.
- remove them. For infix operators, the MathML WG uses &#x2229; and &#x222A; instead.
- add a rule for determining the form, that takes into account this kind of "equal-like" operators.

Comment 9

[Frédéric Wang (:fredw)]

Created attachment 442638 [details]
bug infix ⋂

Comment 10

[Karl Tomlinson (:karlt)]

(In reply to comment #8)
> - remove them. For infix operators, the MathML WG uses ∩ and ∪
> instead.

Removing the infix operators so that U+22C2 N-ARY INTERSECTION is always treated like its prefix form seems reasonable to me.
i.e. It seems that N-ARY INTERSECTION should not be used for a binary operator.

Comment 11

[Karl Tomlinson (:karlt)]

> [conflict][stretching] - operator.\u2225.infix (parallel to)
> -lspace:5 rspace:5 stretchy
> +lspace:5 rspace:5

Note bug 414294.

Comment 12

[Frédéric Wang (:fredw)]

Created attachment 442729 [details] [diff] [review]
remove infix entries for N-ary Union and Intersection [pushed]

Comment 13

[Frédéric Wang (:fredw)]

Created attachment 442730 [details] [diff] [review]
add entries for vertical and horizontal arrows [pushed]

This patch adds stretchy vertical/horizontal arrows and should fix bug 516292.

Comment 14

[Frédéric Wang (:fredw)]

Created attachment 442735 [details] [diff] [review]
update some entries whose stretchiness has changed

Comment 15

[Frédéric Wang (:fredw)]

Order of application is: attachment 442729 [details] [diff] [review], attachment 442730 [details] [diff] [review] and attachment 442735 [details] [diff] [review].

The two first can be safely taken now. For the third one, we have to wait the "scale stretchy" patch, otherwise integrals will be too small.

Comment 16

[Frédéric Wang (:fredw)]

Created attachment 443008 [details] [diff] [review]
add entries for stretchy fences [pushed]

Comment 17

[Frédéric Wang (:fredw)]

New values:

- 52 obsolete entries (22 of them are related to stretching)
- 293 unchanged entries
- 122 conflicting entries (5 of them are related to stretching) (see comment 5)
- 643 new entries (8 of them are related to stretching)

The 8 remaining "new stretchy entries" are:

operator.\u2044.infix (fraction slash)
operator.\u2215.infix (division slash)
operator.\u21D6.infix (north west double arrow)
operator.\u21D7.infix (north east double arrow)
operator.\u21D8.infix (south east double arrow)
operator.\u21D9.infix (south west double arrow)
operator.\u2921.infix (north west and south east arrow)
operator.\u2922.infix (north east and south west arrow)

For the diagonal arrows, I think we should wait bug 552290. For the two, see bug 414294 comment 15.

Comment 18

[Frédéric Wang (:fredw)]

Created attachment 446278 [details] [diff] [review]
 Add a perl script to compare MathML operator dictionaries and check validity of our private data

Comment 19

[Frédéric Wang (:fredw)]

Created attachment 446279 [details] [diff] [review]
make operator with a stretchy form use the same direction for all their forms [pushed]

Comment 20

[Frédéric Wang (:fredw)]

Created attachment 446281 [details] [diff] [review]
update some entries whose stretchiness has changed

Comment 21

[Frédéric Wang (:fredw)]

I've updated the set of patches to add a constraint on directions.

Order of application of patches is:
1) attachment 442729 [details] [diff] [review]
2) attachment 442730 [details] [diff] [review]
3) attachment 534970 [details] [diff] [review]
4) attachment 446279 [details] [diff] [review]
5) attachment 446281 [details] [diff] [review]

Comment 22

[Frédéric Wang (:fredw)]

1) attachment 442729 [details] [diff] [review]
2) attachment 442730 [details] [diff] [review]
3) attachment 443008 [details] [diff] [review]
4) attachment 446279 [details] [diff] [review]
5) attachment 446281 [details] [diff] [review]

Comment 23

[Karl Tomlinson (:karlt)]

Comment on attachment 446279 [details] [diff] [review]
make operator with a stretchy form use the same direction for all their forms [pushed]

Noting that this is for bug 524275 comment 11.

Comment 24

[Frédéric Wang (:fredw)]

Created attachment 447035 [details] [diff] [review]
Add a perl script to compare MathML operator dictionaries and check validity of our private data

Comment 25

[Frédéric Wang (:fredw)]

Created attachment 447036 [details] [diff] [review]
updateOperatorDictionary.pl: add private data "integral"

This new property is for bug 414277.

Comment 26

[Frédéric Wang (:fredw)]

Created attachment 447037 [details] [diff] [review]
add private data "integral" [pushed]

and this is the patch for mathfont.properties

Comment 27

[Frédéric Wang (:fredw)]

Created attachment 454961 [details] [diff] [review]
update some entries whose stretchiness has changed [pushed]

Comment 28

[Frédéric Wang (:fredw)]

Created attachment 454962 [details] [diff] [review]
Add a perl script to compare MathML operator dictionaries and check validity of our private data

Comment 29

[Frédéric Wang (:fredw)]

Created attachment 454963 [details] [diff] [review]
updateOperatorDictionary.pl: add private data "integral"

Comment 30

[Frédéric Wang (:fredw)]

Maybe we can also take the two patches adding the scripts? They are not used in Gecko code, but rather are just checking the operator dictionary, so I think it would be safe to take them.

The Math WG has updated its dictionary, so the result changed:

- 49 obsolete entries (19 of them are related to stretching)
- 285 unchanged entries
- 155 conflicting entries (7 of them are related to stretching)
- 645 new entries (38 of them are related to stretching)

In a first time, I think I'm going to provide a patch which reorders the entries of mathfont.properties with respect to the script output, but without modifying them. Hence it would be easier to see the future changes. Next, I'll give a patch adding the new entries that do not have a stretchy or largeop property. This should be the case for most of them.

Comment 31

[Frédéric Wang (:fredw)]

Created attachment 463330 [details] [diff] [review]
sort entries of the dictionary

Comment 32

[Frédéric Wang (:fredw)]

Created attachment 463331 [details] [diff] [review]
add new entries which are neither stretchy nor largeop

Comment 33

[Frédéric Wang (:fredw)]

Here are the new values after applying the two patches:

- 49 obsolete entries (19 of them are related to stretching)
- 884 unchanged entries
- 155 conflicting entries (7 of them are related to stretching)
- 46 new entries (38 of them are related to stretching)

The 8 remaining non-stretchy new entries are the following largeops:

[new entry] - operator.\u2A03.prefix (n-ary union operator with dot)
[new entry] - operator.\u2A05.prefix (n-ary square intersection operator)
[new entry] - operator.\u2A07.prefix (two logical and operator)
[new entry] - operator.\u2A08.prefix (two logical or operator)
[new entry] - operator.\u2A09.prefix (n-ary times operator)
[new entry] - operator.\u2A0A.prefix (modulo two sum)
[new entry] - operator.\u2AFC.prefix (large triple vertical bar operator)
[new entry] - operator.\u2AFF.prefix (n-ary white vertical bar)

Comment 34

[Frédéric Wang (:fredw)]

Created attachment 463669 [details] [diff] [review]
fix space conflicts

Comment 35

[Frédéric Wang (:fredw)]

Created attachment 463671 [details] [diff] [review]
add new fence, accent and largeop entries

Comment 36

[Frédéric Wang (:fredw)]

The new values after applying the two last patches:

- 49 obsolete entries (19 of them are related to stretching)
- 996 unchanged entries
- 77 conflicting entries (7 of them are related to stretching)
- 12 new entries (12 of them are related to stretching)

Comment 37

[Frédéric Wang (:fredw)]

Created attachment 463818 [details] [diff] [review]
fix conflicts for entries becoming accent

Comment 38

[Frédéric Wang (:fredw)]

Created attachment 463819 [details] [diff] [review]
fix three conflicts for symmetric or fence

Comment 39

[Frédéric Wang (:fredw)]

- 49 obsolete entries (19 of them are related to stretching)
- 1066 unchanged entries
- 7 conflicting entries (7 of them are related to stretching)
- 12 new entries (12 of them are related to stretching)

Comment 40

[Frédéric Wang (:fredw)]

Created attachment 463829 [details] [diff] [review]
remove obsolete infix entries for coproduct, wedge and vee

For these infix forms, one should rather use

operator.\u2A3F.infix = lspace:4 rspace:4 # amalgamation or coproduct
operator.\u2227.infix = lspace:4 rspace:4 # ∧
operator.\u2228.infix = lspace:4 rspace:4 # ∨

This patch should allow to pass all the tests in Topics/LargeOp/ (from the MathML3 testsuite).

Comment 41

[Karl Tomlinson (:karlt)]

Comment on attachment 463330 [details] [diff] [review]
sort entries of the dictionary

This patch seems to be removing a "mirrorable" flag.
That means it would only apply after attachment 450849 [details] [diff] [review] is applied I guess, but
then it would be undoing much of that attachment.

(There are a couple of minsize=0 attributes removed too, but I think that's
fine because I don't think those values change anything.)

I like the idea of sorting these entries though.

Comment 42

[Karl Tomlinson (:karlt)]

(In reply to comment #30)
> Maybe we can also take the two patches adding the scripts?

Yes, that makes sense.

Comment 43

[Karl Tomlinson (:karlt)]

Comment on attachment 463818 [details] [diff] [review]
fix conflicts for entries becoming accent

Have you checked whether drawing the arrows larger (which accent should cause) looks reasonable?

Comment 44

[Karl Tomlinson (:karlt)]

Created attachment 463997 [details] [diff] [review]
first six patches concatenated for aproval request

Requesting approval2.0.  This patch is necessary in some situations to make the patch in bug 414277 behave as expected.
The patch contains no code but modifies data only.  Much of this data can be set or overridden by the author but these values provide sensible defaults.

Comment 45

[Frédéric Wang (:fredw)]

(In reply to comment #41)
> Comment on attachment 463330 [details] [diff] [review]
> sort entries of the dictionary
> 
> This patch seems to be removing a "mirrorable" flag.
> That means it would only apply after attachment 450849 [details] [diff] [review] is applied I guess, but
> then it would be undoing much of that attachment.

Sorry, it's a mistake. We are likely to take those patches before implementing RTL directionality, so it's better to apply them before attachment 450849 [details] [diff] [review]. Fortunately, the script does not support the mirrorable flag yet, so it has been removed and I finally obtained the expected result.

> 
> (There are a couple of minsize=0 attributes removed too, but I think that's
> fine because I don't think those values change anything.)
> 

There was a bug in the script, minsize was removed when it took the value 0. However, this is also what the MathML dictionary does and anyway I think we don't parse it.

Comment 46

[Frédéric Wang (:fredw)]

Created attachment 464149 [details] [diff] [review]
Add a perl script to compare MathML operator dictionaries and check validity of our private data [pushed]

Comment 47

[Frédéric Wang (:fredw)]

Created attachment 464150 [details] [diff] [review]
sort entries of the dictionary and remove minsize attributes

Comment 48

[Karl Tomlinson (:karlt)]

Comment on attachment 442729 [details] [diff] [review]
remove infix entries for N-ary Union and Intersection [pushed]

http://hg.mozilla.org/mozilla-central/rev/170dc0922c23

Comment 49

[Karl Tomlinson (:karlt)]

Comment on attachment 442730 [details] [diff] [review]
add entries for vertical and horizontal arrows [pushed]

http://hg.mozilla.org/mozilla-central/rev/8ddb819c17fd

Comment 50

[Karl Tomlinson (:karlt)]

Comment on attachment 443008 [details] [diff] [review]
add entries for stretchy fences [pushed]

http://hg.mozilla.org/mozilla-central/rev/7aee654600f2

Comment 51

[Karl Tomlinson (:karlt)]

Comment on attachment 446279 [details] [diff] [review]
make operator with a stretchy form use the same direction for all their forms [pushed]

http://hg.mozilla.org/mozilla-central/rev/4428b9c468cd

Comment 52

[Karl Tomlinson (:karlt)]

Comment on attachment 447037 [details] [diff] [review]
add private data "integral" [pushed]

http://hg.mozilla.org/mozilla-central/rev/8a52e79553b6

Comment 53

[Karl Tomlinson (:karlt)]

Comment on attachment 454961 [details] [diff] [review]
update some entries whose stretchiness has changed [pushed]

http://hg.mozilla.org/mozilla-central/rev/77e3880307ea

Comment 54

[Frédéric Wang (:fredw)]

(In reply to comment #43)
> Comment on attachment 463818 [details] [diff] [review]
> fix conflicts for entries becoming accent
> 
> Have you checked whether drawing the arrows larger (which accent should cause)
> looks reasonable?

The new rendering looks good to me.

Comment 55

[Karl Tomlinson (:karlt)]

Comment on attachment 464149 [details] [diff] [review]
Add a perl script to compare MathML operator dictionaries and check validity of our private data [pushed]

rs=me

Comment 56

[Karl Tomlinson (:karlt)]

Comment on attachment 454963 [details] [diff] [review]
updateOperatorDictionary.pl: add private data "integral"

>+            print $file_syntax_errors "warning: operator contains the term \"intgral\" in its comment, but is not integral\n";

\"integral\"

Comment 57

[Frédéric Wang (:fredw)]

Created attachment 480859 [details] [diff] [review]
updateOperatorDictionary.pl: add private data "integral" (fix typo) [pushed]

Comment 58

[Karl Tomlinson (:karlt)]

Comment on attachment 464149 [details] [diff] [review]
Add a perl script to compare MathML operator dictionaries and check validity of our private data [pushed]

http://hg.mozilla.org/mozilla-central/rev/38bc6e6dbf06

Comment 59

[Karl Tomlinson (:karlt)]

Comment on attachment 480859 [details] [diff] [review]
updateOperatorDictionary.pl: add private data "integral" (fix typo) [pushed]

http://hg.mozilla.org/mozilla-central/rev/0e03f2057b4c

Comment 60

[Frédéric Wang (:fredw)]

Created attachment 500676 [details] [diff] [review]
updateOperatorDictionary.pl: add private data "mirrorable"

this will be used after MathML bidi is implemented

Comment 61

[:Ehsan Akhgari (needinfo please, extremely long backlog, Away 8/18)]

Karl, if there are any more patches to land here, please land them on cedar.

Comment 62

[Karl Tomlinson (:karlt)]

Comment on attachment 463818 [details] [diff] [review]
fix conflicts for entries becoming accent

Making U+21DB an accent broke the scale-stretchy-3 reftest (Bug 605605 comment 22) as its munder didn't expect mo to be rendered differently to mspace because it was an accent.  I've addressed this locally by making all the munders in that test use accentunder=false.

Comment 63

[Frédéric Wang (:fredw)]

Yes, it seems a reasonable fix. Do you plan to attach a patch?

Comment 64

[Karl Tomlinson (:karlt)]

No need.

http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d9c2c2e76a37&tochange=0d20320645d0

Comment 65

[Frédéric Wang (:fredw)]

OK, so I guess we should open a separate bug to fix the scale-stretchy-3 reftest?

Comment 66

[Frédéric Wang (:fredw)]

The remaining differences:

- 47 obsolete entries (19 of them are related to stretching)

- 7 conflicting entries (7 of them are related to stretching)

[conflict][stretching] - operator.\u002F.infix (solidus)
-lspace:3 rspace:3 stretchy
+lspace:1 rspace:1

[conflict][stretching] - operator.\u005F.infix (low line)
-lspace:2 rspace:2 stretchy
+lspace:1 rspace:1

[conflict][stretching] - operator.\u007C\u007C.infix (multiple character operator: ||)
-lspace:4 rspace:4
+lspace:2 rspace:2 stretchy fence symmetric

[conflict][stretching] - operator.\u2216.infix (set minus)
-lspace:3 rspace:3 stretchy
+lspace:4 rspace:4

[conflict][stretching] - operator.\u2223.infix (divides)
-lspace:5 rspace:5 stretchy
+lspace:5 rspace:5

[conflict][stretching] - operator.\u2225.infix (parallel to)
-lspace:5 rspace:5 stretchy
+lspace:5 rspace:5

[conflict][stretching] - operator.\u2758.infix (light vertical bar)
-lspace:5 rspace:5 stretchy symmetric
+lspace:5 rspace:5

- 13 new entries (12 of them are related to stretching)

[new entry][stretching] - operator.\u2044.infix (fraction slash)
-
+lspace:4 rspace:4 stretchy

[new entry][stretching] - operator.\u21D6.infix (north west double arrow)
-
+lspace:5 rspace:5 stretchy

[new entry][stretching] - operator.\u21D7.infix (north east double arrow)
-
+lspace:5 rspace:5 stretchy

[new entry][stretching] - operator.\u21D8.infix (south east double arrow)
-
+lspace:5 rspace:5 stretchy

[new entry][stretching] - operator.\u21D9.infix (south west double arrow)
-
+lspace:5 rspace:5 stretchy

[new entry][stretching] - operator.\u2215.infix (division slash)
+lspace:4 rspace:4 stretchy

[new entry][stretching] - operator.\u27F0.infix (upwards quadruple arrow)
+lspace:5 rspace:5 stretchy

[new entry][stretching] - operator.\u27F1.infix (downwards quadruple arrow)
+lspace:5 rspace:5 stretchy

[new entry][stretching] - operator.\u2921.infix (north west and south east arrow)
+lspace:5 rspace:5 stretchy

[new entry][stretching] - operator.\u2922.infix (north east and south west arrow)
+lspace:5 rspace:5 stretchy

[new entry] - operator.\u2ADD\u0338.infix (nonforking with slash)
+lspace:5 rspace:5

[new entry][stretching] - operator.\u2B45.infix (leftwards quadruple arrow)
+lspace:5 rspace:5 stretchy

[new entry][stretching] - operator.\u2B46.infix (rightwards quadruple arrow)
+lspace:5 rspace:5 stretchy

Comment 67

[Karl Tomlinson (:karlt)]

(In reply to comment #65)
> OK, so I guess we should open a separate bug to fix the scale-stretchy-3
> reftest?

That was fixed here:
http://hg.mozilla.org/mozilla-central/rev/0e519f6626c7

(Please use a separate bug if providing any further operator dictionary patches, though.)