MathML 3 comes with a new format and updated entries for Operator Dictionary. Bug 519126 is about using this new compact format. The present bug aims at updating the entries in mathfont.properties.
Created attachment 442182 [details] [diff] [review] Add a perl script to compare MathML operator dictionaries This script allows to compare our version of the dictionary against the official one. Most changes are new operators or lspace/rspace values, so I think we can take them safely. Probably we need more care for changes related to stretching. The script is able to output the details of the differences as well as an updated version of our dictionary. Entries in differences.txt are marked with "stretching", "conflict", "new", "obsolete" so you can use "grep -A2" to select a particular subset. The updated version takes the values of the W3C dictionary, except that it keeps our "direction", preserve our comment if any (end of line after the sharp "#") and keeps the "obsolete" entries. Entries are sorted with respect to the W3C dictionary, except for the obsolete ones, which are put at the end and sorted by lexical order over the key.
After applying the changes of attachment 423544 [details] [diff] [review] and attachment 441533 [details] [diff] [review] (for mathfont.properties) and comparing with http://monet.nag.co.uk/~dpc/draft-spec/appendixc.xml, the script returns: - 92 obsolete entries (24 of them are related to stretching) - 196 unchanged entries - 102 conflicting entries (9 of them are related to stretching) - 760 new entries (87 of them are related to stretching)
Created attachment 442184 [details] differences between mathfont.properties and the one of the MathML WG
Created attachment 442458 [details] [diff] [review] update some entries whose stretchiness has changed
With the patch of comment 4, the differences are: - 92 obsolete entries (24 of them are related to stretching) - 214 unchanged entries - 84 conflicting entries (5 of them are related to stretching) - 760 new entries (87 of them are related to stretching) The 5 remaining entries whose stretchiness has changed are: [conflict][stretching] - operator.\u005F.infix (low line) -lspace:2 rspace:2 stretchy +lspace:1 rspace:1 -- [conflict][stretching] - operator.\u2225.infix (parallel to) -lspace:5 rspace:5 stretchy +lspace:5 rspace:5 -- [conflict][stretching] - operator.\u2223.infix (divides) -lspace:5 rspace:5 stretchy +lspace:5 rspace:5 -- [conflict][stretching] - operator.\u2216.infix (set minus) -lspace:3 rspace:3 stretchy +lspace:4 rspace:4 -- [conflict][stretching] - operator.\u002F.infix (solidus) -lspace:3 rspace:3 stretchy +lspace:1 rspace:1
Created attachment 442466 [details] differences between mathfont.properties and the one of the MathML WG
Created attachment 442636 [details] [diff] [review] Add a perl script to compare MathML operator dictionaries There was a bug in my script for multiple char operators. The new values are: - 54 obsolete entries (24 of them are related to stretching) - 214 unchanged entries - 122 conflicting entries (5 of them are related to stretching) - 722 new entries (87 of them are related to stretching)
Among the 24 obsolete entries that are stretchy, there is at least two annoying ones for the "scale strechy" patch: operator.\u22C2.infix (⋂) operator.\u22C3.infix (⋃) In Joe Java's test, in "De Morgan's laws" for sets there is something like: <mrow><mi>X</mi> <mo>=</mo> <mo>⋂</mo> <mi>Y</mi> </mrow> Here, the ⋂ operator is considered as infix (so stretchy and not largeop) rather than prefix (not stretchy and largeop). Without the patch, we don't see the difference. But with the patch applied, the former stretches the symbol in the vertical direction only (to cover the height of the formula) while the latter simply increases width and height by twice (it's what we want). To fix the problem, we can: - make these operators non-strechy. But still they won't be largeop. - remove them. For infix operators, the MathML WG uses ∩ and ∪ instead. - add a rule for determining the form, that takes into account this kind of "equal-like" operators.
Created attachment 442638 [details] bug infix ⋂
(In reply to comment #8) > - remove them. For infix operators, the MathML WG uses ∩ and ∪ > instead. Removing the infix operators so that U+22C2 N-ARY INTERSECTION is always treated like its prefix form seems reasonable to me. i.e. It seems that N-ARY INTERSECTION should not be used for a binary operator.
> [conflict][stretching] - operator.\u2225.infix (parallel to) > -lspace:5 rspace:5 stretchy > +lspace:5 rspace:5 Note bug 414294.
Created attachment 442729 [details] [diff] [review] remove infix entries for N-ary Union and Intersection [pushed]
Created attachment 442730 [details] [diff] [review] add entries for vertical and horizontal arrows [pushed] This patch adds stretchy vertical/horizontal arrows and should fix bug 516292.
Created attachment 442735 [details] [diff] [review] update some entries whose stretchiness has changed
Order of application is: attachment 442729 [details] [diff] [review], attachment 442730 [details] [diff] [review] and attachment 442735 [details] [diff] [review]. The two first can be safely taken now. For the third one, we have to wait the "scale stretchy" patch, otherwise integrals will be too small.
Created attachment 443008 [details] [diff] [review] add entries for stretchy fences [pushed]
New values: - 52 obsolete entries (22 of them are related to stretching) - 293 unchanged entries - 122 conflicting entries (5 of them are related to stretching) (see comment 5) - 643 new entries (8 of them are related to stretching) The 8 remaining "new stretchy entries" are: operator.\u2044.infix (fraction slash) operator.\u2215.infix (division slash) operator.\u21D6.infix (north west double arrow) operator.\u21D7.infix (north east double arrow) operator.\u21D8.infix (south east double arrow) operator.\u21D9.infix (south west double arrow) operator.\u2921.infix (north west and south east arrow) operator.\u2922.infix (north east and south west arrow) For the diagonal arrows, I think we should wait bug 552290. For the two, see bug 414294 comment 15.
Created attachment 446278 [details] [diff] [review] Add a perl script to compare MathML operator dictionaries and check validity of our private data
Created attachment 446279 [details] [diff] [review] make operator with a stretchy form use the same direction for all their forms [pushed]
Created attachment 446281 [details] [diff] [review] update some entries whose stretchiness has changed
I've updated the set of patches to add a constraint on directions. Order of application of patches is: 1) attachment 442729 [details] [diff] [review] 2) attachment 442730 [details] [diff] [review] 3) attachment 534970 [details] [diff] [review] 4) attachment 446279 [details] [diff] [review] 5) attachment 446281 [details] [diff] [review]
1) attachment 442729 [details] [diff] [review] 2) attachment 442730 [details] [diff] [review] 3) attachment 443008 [details] [diff] [review] 4) attachment 446279 [details] [diff] [review] 5) attachment 446281 [details] [diff] [review]
Comment on attachment 446279 [details] [diff] [review] make operator with a stretchy form use the same direction for all their forms [pushed] Noting that this is for bug 524275 comment 11.
Created attachment 447035 [details] [diff] [review] Add a perl script to compare MathML operator dictionaries and check validity of our private data
Created attachment 447036 [details] [diff] [review] updateOperatorDictionary.pl: add private data "integral" This new property is for bug 414277.
Created attachment 447037 [details] [diff] [review] add private data "integral" [pushed] and this is the patch for mathfont.properties
Created attachment 454961 [details] [diff] [review] update some entries whose stretchiness has changed [pushed]
Created attachment 454962 [details] [diff] [review] Add a perl script to compare MathML operator dictionaries and check validity of our private data
Created attachment 454963 [details] [diff] [review] updateOperatorDictionary.pl: add private data "integral"
Maybe we can also take the two patches adding the scripts? They are not used in Gecko code, but rather are just checking the operator dictionary, so I think it would be safe to take them. The Math WG has updated its dictionary, so the result changed: - 49 obsolete entries (19 of them are related to stretching) - 285 unchanged entries - 155 conflicting entries (7 of them are related to stretching) - 645 new entries (38 of them are related to stretching) In a first time, I think I'm going to provide a patch which reorders the entries of mathfont.properties with respect to the script output, but without modifying them. Hence it would be easier to see the future changes. Next, I'll give a patch adding the new entries that do not have a stretchy or largeop property. This should be the case for most of them.
Created attachment 463330 [details] [diff] [review] sort entries of the dictionary
Created attachment 463331 [details] [diff] [review] add new entries which are neither stretchy nor largeop
Here are the new values after applying the two patches: - 49 obsolete entries (19 of them are related to stretching) - 884 unchanged entries - 155 conflicting entries (7 of them are related to stretching) - 46 new entries (38 of them are related to stretching) The 8 remaining non-stretchy new entries are the following largeops: [new entry] - operator.\u2A03.prefix (n-ary union operator with dot) [new entry] - operator.\u2A05.prefix (n-ary square intersection operator) [new entry] - operator.\u2A07.prefix (two logical and operator) [new entry] - operator.\u2A08.prefix (two logical or operator) [new entry] - operator.\u2A09.prefix (n-ary times operator) [new entry] - operator.\u2A0A.prefix (modulo two sum) [new entry] - operator.\u2AFC.prefix (large triple vertical bar operator) [new entry] - operator.\u2AFF.prefix (n-ary white vertical bar)
Created attachment 463669 [details] [diff] [review] fix space conflicts
Created attachment 463671 [details] [diff] [review] add new fence, accent and largeop entries
The new values after applying the two last patches: - 49 obsolete entries (19 of them are related to stretching) - 996 unchanged entries - 77 conflicting entries (7 of them are related to stretching) - 12 new entries (12 of them are related to stretching)
Created attachment 463818 [details] [diff] [review] fix conflicts for entries becoming accent
Created attachment 463819 [details] [diff] [review] fix three conflicts for symmetric or fence
- 49 obsolete entries (19 of them are related to stretching) - 1066 unchanged entries - 7 conflicting entries (7 of them are related to stretching) - 12 new entries (12 of them are related to stretching)
Created attachment 463829 [details] [diff] [review] remove obsolete infix entries for coproduct, wedge and vee For these infix forms, one should rather use operator.\u2A3F.infix = lspace:4 rspace:4 # amalgamation or coproduct operator.\u2227.infix = lspace:4 rspace:4 # ∧ operator.\u2228.infix = lspace:4 rspace:4 # ∨ This patch should allow to pass all the tests in Topics/LargeOp/ (from the MathML3 testsuite).
Comment on attachment 463330 [details] [diff] [review] sort entries of the dictionary This patch seems to be removing a "mirrorable" flag. That means it would only apply after attachment 450849 [details] [diff] [review] is applied I guess, but then it would be undoing much of that attachment. (There are a couple of minsize=0 attributes removed too, but I think that's fine because I don't think those values change anything.) I like the idea of sorting these entries though.
(In reply to comment #30) > Maybe we can also take the two patches adding the scripts? Yes, that makes sense.
Comment on attachment 463818 [details] [diff] [review] fix conflicts for entries becoming accent Have you checked whether drawing the arrows larger (which accent should cause) looks reasonable?
Created attachment 463997 [details] [diff] [review] first six patches concatenated for aproval request Requesting approval2.0. This patch is necessary in some situations to make the patch in bug 414277 behave as expected. The patch contains no code but modifies data only. Much of this data can be set or overridden by the author but these values provide sensible defaults.
(In reply to comment #41) > Comment on attachment 463330 [details] [diff] [review] > sort entries of the dictionary > > This patch seems to be removing a "mirrorable" flag. > That means it would only apply after attachment 450849 [details] [diff] [review] is applied I guess, but > then it would be undoing much of that attachment. Sorry, it's a mistake. We are likely to take those patches before implementing RTL directionality, so it's better to apply them before attachment 450849 [details] [diff] [review]. Fortunately, the script does not support the mirrorable flag yet, so it has been removed and I finally obtained the expected result. > > (There are a couple of minsize=0 attributes removed too, but I think that's > fine because I don't think those values change anything.) > There was a bug in the script, minsize was removed when it took the value 0. However, this is also what the MathML dictionary does and anyway I think we don't parse it.
Created attachment 464149 [details] [diff] [review] Add a perl script to compare MathML operator dictionaries and check validity of our private data [pushed]
Created attachment 464150 [details] [diff] [review] sort entries of the dictionary and remove minsize attributes
Comment on attachment 442729 [details] [diff] [review] remove infix entries for N-ary Union and Intersection [pushed] http://hg.mozilla.org/mozilla-central/rev/170dc0922c23
Comment on attachment 442730 [details] [diff] [review] add entries for vertical and horizontal arrows [pushed] http://hg.mozilla.org/mozilla-central/rev/8ddb819c17fd
Comment on attachment 443008 [details] [diff] [review] add entries for stretchy fences [pushed] http://hg.mozilla.org/mozilla-central/rev/7aee654600f2
Comment on attachment 446279 [details] [diff] [review] make operator with a stretchy form use the same direction for all their forms [pushed] http://hg.mozilla.org/mozilla-central/rev/4428b9c468cd
Comment on attachment 447037 [details] [diff] [review] add private data "integral" [pushed] http://hg.mozilla.org/mozilla-central/rev/8a52e79553b6
Comment on attachment 454961 [details] [diff] [review] update some entries whose stretchiness has changed [pushed] http://hg.mozilla.org/mozilla-central/rev/77e3880307ea
(In reply to comment #43) > Comment on attachment 463818 [details] [diff] [review] > fix conflicts for entries becoming accent > > Have you checked whether drawing the arrows larger (which accent should cause) > looks reasonable? The new rendering looks good to me.
Comment on attachment 464149 [details] [diff] [review] Add a perl script to compare MathML operator dictionaries and check validity of our private data [pushed] rs=me
Comment on attachment 454963 [details] [diff] [review] updateOperatorDictionary.pl: add private data "integral" >+ print $file_syntax_errors "warning: operator contains the term \"intgral\" in its comment, but is not integral\n"; \"integral\"
Created attachment 480859 [details] [diff] [review] updateOperatorDictionary.pl: add private data "integral" (fix typo) [pushed]
Comment on attachment 464149 [details] [diff] [review] Add a perl script to compare MathML operator dictionaries and check validity of our private data [pushed] http://hg.mozilla.org/mozilla-central/rev/38bc6e6dbf06
Comment on attachment 480859 [details] [diff] [review] updateOperatorDictionary.pl: add private data "integral" (fix typo) [pushed] http://hg.mozilla.org/mozilla-central/rev/0e03f2057b4c
Created attachment 500676 [details] [diff] [review] updateOperatorDictionary.pl: add private data "mirrorable" this will be used after MathML bidi is implemented
Karl, if there are any more patches to land here, please land them on cedar.
Comment on attachment 463818 [details] [diff] [review] fix conflicts for entries becoming accent Making U+21DB an accent broke the scale-stretchy-3 reftest (Bug 605605 comment 22) as its munder didn't expect mo to be rendered differently to mspace because it was an accent. I've addressed this locally by making all the munders in that test use accentunder=false.
Yes, it seems a reasonable fix. Do you plan to attach a patch?
No need. http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d9c2c2e76a37&tochange=0d20320645d0
OK, so I guess we should open a separate bug to fix the scale-stretchy-3 reftest?
The remaining differences: - 47 obsolete entries (19 of them are related to stretching) - 7 conflicting entries (7 of them are related to stretching) [conflict][stretching] - operator.\u002F.infix (solidus) -lspace:3 rspace:3 stretchy +lspace:1 rspace:1 [conflict][stretching] - operator.\u005F.infix (low line) -lspace:2 rspace:2 stretchy +lspace:1 rspace:1 [conflict][stretching] - operator.\u007C\u007C.infix (multiple character operator: ||) -lspace:4 rspace:4 +lspace:2 rspace:2 stretchy fence symmetric [conflict][stretching] - operator.\u2216.infix (set minus) -lspace:3 rspace:3 stretchy +lspace:4 rspace:4 [conflict][stretching] - operator.\u2223.infix (divides) -lspace:5 rspace:5 stretchy +lspace:5 rspace:5 [conflict][stretching] - operator.\u2225.infix (parallel to) -lspace:5 rspace:5 stretchy +lspace:5 rspace:5 [conflict][stretching] - operator.\u2758.infix (light vertical bar) -lspace:5 rspace:5 stretchy symmetric +lspace:5 rspace:5 - 13 new entries (12 of them are related to stretching) [new entry][stretching] - operator.\u2044.infix (fraction slash) - +lspace:4 rspace:4 stretchy [new entry][stretching] - operator.\u21D6.infix (north west double arrow) - +lspace:5 rspace:5 stretchy [new entry][stretching] - operator.\u21D7.infix (north east double arrow) - +lspace:5 rspace:5 stretchy [new entry][stretching] - operator.\u21D8.infix (south east double arrow) - +lspace:5 rspace:5 stretchy [new entry][stretching] - operator.\u21D9.infix (south west double arrow) - +lspace:5 rspace:5 stretchy [new entry][stretching] - operator.\u2215.infix (division slash) +lspace:4 rspace:4 stretchy [new entry][stretching] - operator.\u27F0.infix (upwards quadruple arrow) +lspace:5 rspace:5 stretchy [new entry][stretching] - operator.\u27F1.infix (downwards quadruple arrow) +lspace:5 rspace:5 stretchy [new entry][stretching] - operator.\u2921.infix (north west and south east arrow) +lspace:5 rspace:5 stretchy [new entry][stretching] - operator.\u2922.infix (north east and south west arrow) +lspace:5 rspace:5 stretchy [new entry] - operator.\u2ADD\u0338.infix (nonforking with slash) +lspace:5 rspace:5 [new entry][stretching] - operator.\u2B45.infix (leftwards quadruple arrow) +lspace:5 rspace:5 stretchy [new entry][stretching] - operator.\u2B46.infix (rightwards quadruple arrow) +lspace:5 rspace:5 stretchy
(In reply to comment #65) > OK, so I guess we should open a separate bug to fix the scale-stretchy-3 > reftest? That was fixed here: http://hg.mozilla.org/mozilla-central/rev/0e519f6626c7 (Please use a separate bug if providing any further operator dictionary patches, though.)
or
