1361631 - [css-writing-modes] 'text-orientation: upright' should render Mongolian in its intrinsic (vertical-only native) orientation

Status: RESOLVED FIXED

(Core :: Layout: Text and Fonts, defect, P3)

Description

[KariP]

Attachment: test-case-mongolian.html

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.1 Safari/603.1.30

Steps to reproduce:

I opened the attached test-case-mongolian.html file.


Actual results:

Mongolian text isn’t rendered correctly in vertical writing mode, if text-orientation is upright.



Expected results:

All three Mongolian texts should look the same.

The Writing Modes 3 specification [1] says for upright: “Typographic character units from vertical scripts are typeset with their intrinsic orientation and shaped normally.” Appendix A lists Mongolian as a vertical script [2].

I also confirmed this on css mailing list [3]. A correct implementation may have to inspect the script tag to recognize vertical scripts.

[1] https://drafts.csswg.org/css-writing-modes-3/#text-orientation
[2] https://drafts.csswg.org/css-writing-modes-3/#script-orientations
[3] https://lists.w3.org/Archives/Public/www-style/2017May/0002.html

Comment 1

[KariP]

Attachment: screenshot-firefox.png

Added screenshot-firefox.png to show how the test case looks like on macOS 10.12, Firefox 53.0.

On Mac, I had to install Mongolian White font to get mixed and sideways correct: http://www.mongolfont.com/en/font/mnglwhiteaat.html

Comment 2

[KariP]

Chrome bug: https://bugs.chromium.org/p/chromium/issues/detail?id=717862

Comment 3

[Gérard Talbot]

I get actual results with Firefox 55.0a1 buildID=20170503091824 and I am using Linux. Therefore setting version to Trunk and platform and OS to All.

Kari, let me know if you want to submit your test to the CSS Writing Modes test suite. I can help you with this... although I still am not able to use GitHub at all right now...

Comment 4

[KariP]

I’m afraid my test can’t be used as such in the w3c test suite. It uses the “Mongolian White” font [1], but I don’t think that its license allows inclusion in w3c tests.

Using system fonts seem to also have issues. Mac system font doesn’t appear to give good results for traditional Mongolian in any text-orientation. I suspect that the font is some Mac special font or has a flaw. Windows seems to have good system fonts for Mongolian and they work as expected for mixed and sideways text-orientation. The default installation of my Ubuntu Linux 16.10 doesn’t have any system font for traditional Mongolian.

Do you have a font with Mongolian glyphs in the test suite or is it ok to just rely on system fonts?

The Mongolian text “ᠮᠣᠩᠭᠣᠯ ᠺᠡᠯᠡ” in my test says “Mongolian Language” and I took it from Wikipedia [2].

I wonder if the other vertical scripts [3] should be included in tests. I might be difficult to find a font for Ogham or Old Turkic. They are also special compared to Han or Mongolian scripts, because they are rotated -90°. I don’t think any browser typesets them right.

Unfortunately, I can't spend a lot of time creating these tests, but I'm happy to submit the Mongolian test.

[1] http://www.mongolfont.com/en/font/mnglwhiteaat.html
[2] https://en.wikipedia.org/wiki/Mongolian_language
[3] https://www.w3.org/TR/css-writing-modes-3/#script-orientations

Comment 5

[Gérard Talbot]

(In reply to KariP from comment #4)
> I’m afraid my test can’t be used as such in the w3c test suite. It uses the
> “Mongolian White” font [1], but I don’t think that its license allows
> inclusion in w3c tests.

Kari,

Do not worry too much about this. I can use "Noto Sans Mongolian" which, I believe, has a GPL license. 

https://www.google.com/get/noto/#sans-mong

"
Noto fonts are open source. All Noto fonts are published under the SIL Open Font License, Version 1.1. Language data and some sample texts are from the Unicode CLDR project.
"
https://www.google.com/get/noto/help/faq/

"
The SIL Open Font License (OFL) is a free, libre and open source license specifically designed for fonts and related software based on our experience in font design and linguistic software engineering. 
"
http://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&id=OFL

I used it as a webfont in this test:

http://www.gtalbot.org/BrowserBugsSection/CSS3WritingModes/Issue484651-keyboard-arrow-navigation-inside-vertical-editable-text.html

You can do that too.
 
> Using system fonts seem to also have issues. Mac system font doesn’t appear
> to give good results for traditional Mongolian in any text-orientation. I
> suspect that the font is some Mac special font or has a flaw. Windows seems
> to have good system fonts for Mongolian and they work as expected for mixed
> and sideways text-orientation. The default installation of my Ubuntu Linux
> 16.10 doesn’t have any system font for traditional Mongolian.


Noto Sans Mongolian is installed by default in Kubuntu 17.04!  I use Kubuntu 17.04.

Let me check if this is also the case for Ubuntu 17.04 ...

I see

/usr/share/fonts/truetype/noto/NotoSansMongolian-Regular.ttf

in the list of files in this page:

http://packages.ubuntu.com/zesty/all/fonts-noto-unhinted/filelist

So, it seems that Ubuntu 17.04 installs by default NotoSansMongolian-Regular.ttf .




> Do you have a font with Mongolian glyphs in the test suite or is it ok to
> just rely on system fonts?
> 
> The Mongolian text “ᠮᠣᠩᠭᠣᠯ ᠺᠡᠯᠡ” in my test says “Mongolian Language” and I
> took it from Wikipedia [2].
> 
> I wonder if the other vertical scripts [3] should be included in tests. I
> might be difficult to find a font for Ogham or Old Turkic. They are also
> special compared to Han or Mongolian scripts, because they are rotated -90°.
> I don’t think any browser typesets them right.
> 
> Unfortunately, I can't spend a lot of time creating these tests, 

I can do that for you. Although, like I said, I still am not able to use GitHub at all right now... and this month (may), I won't have time to study and struggle with GitHub.

> but I'm
> happy to submit the Mongolian test.
> 
> [1] http://www.mongolfont.com/en/font/mnglwhiteaat.html
> [2] https://en.wikipedia.org/wiki/Mongolian_language
> [3] https://www.w3.org/TR/css-writing-modes-3/#script-orientations

Gérard

Comment 6

[Gérard Talbot]

> The default installation of my Ubuntu Linux
> 16.10 doesn’t have any system font for traditional Mongolian.

This file

http://packages.ubuntu.com/yakkety/all/fonts-noto-unhinted/filelist

lists

/usr/share/fonts/truetype/noto/NotoSansMongolian-Regular.ttf

so it should be available in Ubuntu 16.10. Can you verify, make sure that the package fonts-noto-unhinted is installed on your Ubuntu 16.10 ?

Gérard

Comment 7

[KariP]

Yep, installing fonts-noto-unhinted fixes Firefox to display Mongolian, Old Turkic and other scripts on Ubuntu 16.10. However, it wasn't installed by default.

Great that you can create the tests. If it helps, you can include or modify my test anyway you want.

Comment 8

[Gérard Talbot]

I have been assigned to review, create and submit writing-mode tests to the CSS Writing Modes test suite and to file bug reports since October 2014:

https://groups.google.com/forum/#!topic/mozilla.dev.platform/7snBFnFW38g

I will adjust and slightly modify your test so that it complies perfectly with all of the Test writing guidelines and checkpoints. I will create a reference file for your test. You will still be author of your test. The test will still identify you as the author; I will be reviewer+approver of your test. It is best that way.

So, your test is on my to-do-list. The only issue now is that I am not familiar, I do not understand how to use GitHub at all, which is what is now used to review, create, modify and submit tests to the CSS Writing Modes test suite. I will struggle, tackle GitHub in June 2017, next month.

Comment 9

[Gérard Talbot]

"
vertical-only
    Scripts that have vertical, but not horizontal, native orientation. Includes: Mongolian, (...)
"
5. Introduction to Vertical Text Layout
https://drafts.csswg.org/css-writing-modes-3/#intro-text-layout

Marking as NEW

Comment 10

[Gérard Talbot]

"
3.3 Vertical Glyphs in the Code Charts

The Unicode code charts generally show characters in the orientation they take when used in horizontal lines. However, prior to Unicode 7.0, there were a few exceptions, mostly for characters or scripts which are normally [*_and only!_*] written in vertical lines; in those cases, the code charts used to show the characters in the same orientation as in vertical lines. Furthermore, such characters are often rotated when displayed in horizontal lines; figure 3 shows an example of Mongolian text in horizontal lines in which the Mongolian characters are rotated 90 degrees counterclockwise with respect to the code charts prior to Unicode 7.0. (...)
"
Unicode® Technical Report #50
Unicode Vertical Text Layout
3.3 Vertical Glyphs in the Code Charts
http://www.unicode.org/reports/tr50/#code_charts

Comment 11

[Gérard Talbot]

> I will adjust and slightly modify your test so that it complies perfectly
> with all of the Test writing guidelines and checkpoints. I will create a
> reference file for your test. You will still be author of your test. The
> test will still identify you as the author; I will be reviewer+approver of
> your test. It is best that way.

[test]
http://www.gtalbot.org/BrowserBugsSection/CSS3WritingModes/s51-text-orientation-Mongolian-upright-002.xht

[reference file]
http://www.gtalbot.org/BrowserBugsSection/CSS3WritingModes/s51-text-orientation-Mongolian-upright-002-ref.xht

Kari, 

To be recognized as test author, you need to be registered somewhere... I do not know if this registration process is still possible now. But, for now, you are credited in the source code in a comment as the original contributor about the related bug report.
 
> The only issue now is that I am not
> familiar, I do not understand how to use GitHub at all, which is what is now
> used to review, create, modify and submit tests to the CSS Writing Modes
> test suite. I will struggle, tackle GitHub in June 2017, next month.

That test and its reference file have not been submitted to the Web Platform Test repository

https://github.com/w3c/web-platform-tests/tree/master/css/css-writing-modes-3

I will be tackling this task (it's in my to-do-list now) in the next few days.

Comment 12

[KariP]

I realized that it may not be enough to say: "Test passes if the mongolian glyphs in all 3 columns have the same layout and the same orientation."

Originally, I made the test for Firefox and Chrome, which show at least one of the text orientations correctly. On Safari, all three columns are shown wrong, but they all look wrong the same way. Does it mean that the test passes for Safari?

I don’t mind if I’m not recognized as a test author. Credits in the source code sounds good.

Comment 13

[Gérard Talbot]

Thank you for your feedback: this is appreciated.

(In reply to KariP from comment #12)

> Originally, I made the test for Firefox and Chrome, which show at least one
> of the text orientations correctly. 

The test is using <div id="mixed"> as reference. In s51-text-orientation-Mongolian-upright-002-ref.xht , the default text-orientation value is mixed. So, in the test design, div#upright should display the same as div#mixed renders.

> On Safari, all three columns are shown
> wrong, but they all look wrong the same way.

This means I may have to add 

<link rel="mismatch" href="s51-text-orientation-Mongolian-upright-002-notref.xht" />

and create s51-text-orientation-Mongolian-upright-002-notref.xht 

KariP, please upload a screen shot of how Safari handles the current test. Also, indicate which version of Safari you are using.

> Does it mean that the test
> passes for Safari?

No, the test does not pass for Safari. It means that the test currently creates a false positive in Safari. If all 3 columns are wrong, then it possibly means that 'text-orientation' is not supported or that 'writing-mode' requires a vendor-prefix or something like that.

> I don’t mind if I’m not recognized as a test author. Credits in the source
> code sounds good.

I will check with others how to include new test authors regarding tests submitted. It is fairer and more honorable to officially and explicitly recognize all contributors of tests. (And also, I can become approver+reviewer of such test which normally speeds up integration into the test suite.) Test authors are also often acknowledged in #acknowledgments sections of respective CSS specifications. Example given:

https://www.w3.org/TR/css-writing-modes-3/#acknowledgements

Comment 14

[Gérard Talbot]

> On Safari, all three columns are shown
> wrong, but they all look wrong the same way.

Bug 112488: REGRESSION (r145854): Mongolian characters are not upright in vertical writing mode
https://bugs.webkit.org/show_bug.cgi?id=112488

Bug 86071: Incorrect behavior of upright text in vertical writing modes
https://bugs.webkit.org/show_bug.cgi?id=86071
UNCONFIRMED and now 4 years old ...

but the existence of those bug reports and a bunch of others [1] still does not explain how the 3 columns look the same way.

[1]:
https://bugs.webkit.org/buglist.cgi?quicksearch=text-orientation&list_id=2892638

Comment 15

[siqinbilige]

For a reference
1. All exist Mongolian fonts has counter-clockwise 90deg glyphs(<--). 
  Because, It can not be make upright(vertical) glyphs which its height is different and must connect each other no space.
　　Actually we should have created clockwise 90deg glyphs(-->), but　RTL layout was not perfect in DOS.
2. Mongolian is vertical only language.
  In vertical writing, we need to rotate glyphs clockwise 90deg.
  In horizontal writing, for easy to read one word, let glyphs as it is.
3. sideways not have much meaning for mongolian word.
4. both mnglwhiteotf.ttf(OpenType) and mnglwhiteaat.ttf(AAT) font has a GPL license. ( Actually I created these fonts. We recommend　use these fonts for test. )

Comment 16

[siqinbilige]

Attachment: layout.png

Comment 17

[siqinbilige]

Attachment: mnglwhiteaat.ttf

Comment 18

[siqinbilige]

Attachment: mnglwhiteotf.ttf

Comment 19

[Gérard Talbot]

> On Safari, all three columns are shown
> wrong, but they all look wrong the same way.

I've done research on Safari's support of CSS vertical text properties. According to 

https://developer.mozilla.org/en-US/docs/Web/CSS/writing-mode#Browser_compatibility

and to

http://caniuse.com/#feat=css-writing-mode

'writing-mode' property is supported in Safari since version 5.1 but with the -webkit- vendor prefix. On the other hand, according to

https://developer.mozilla.org/en-US/docs/Web/CSS/text-orientation#Browser_compatibility

'text-orientation' property is _not_ supported in Safari. But

http://caniuse.com/#search=text-orientation

claims that Safari 10.1 supports text-orientation with a -webkit- vendor prefix.

Also, according to Richard Ishida's testing website

https://www.w3.org/International/tests/repo/results/writing-mode-vertical#lr_glyphs

Safari Version/10.1.1 Safari/603.2.5 ( 23-5-2017 ) fails :

writing mode:vertical-lr, default Mongolian orientation
writing-mode-vlr-005-prop.html 	By default, writing-mode:vertical-lr will display Mongolian characters with the right orientation. 

writing mode:vertical-lr, Mongolian joining
writing-mode-vlr-006-prop.html 	By default, writing-mode:vertical-lr will display Mongolian characters with correct joining behaviour.

- - - - - - - - - - - -

So, KariP, I have decided to add the -webkit- vendor prefix to the test and reference file for now but you must keep in mind that all tests submitted to the Writing-mode test suite must not have any vendor prefix. So I will remove the -webkit- declarations before submitting. 

I may have to create s51-text-orientation-Mongolian-upright-002-notref.xht and an associated mismatch link anyway since Edge 16 has no support for text-orientation and, as designed, Edge 16's test result would also be a false positive.

Comment 20

[Gérard Talbot]

> On Safari, all three columns are shown
> wrong, but they all look wrong the same way.

"
Webkit [Safari 10.x] doesn't correctly display the Mongolian text. Glyphs are rotated anti-clockwise away from the correct orientation, and don't join. 
(...)
For Webkit-based browsers [Safari 10.x] you need to use the proprietary -webkit-text-orientation property.
(...)
For Webkit browsers you need to use the prefixed property name, ie. -webkit-text-orientation: sideways.
"
Styling vertical Chinese, Japanese, Korean and Mongolian text by Richard Ishida
https://www.w3.org/International/articles/vertical-text/

Comment 21

[siqinbilige]

Webkit using -webkit-text-orientation: sideways-right to rotate the mongolian glyphs 90deg clockwise. But it is a not good idea. This idea born from issue https://bugs.webkit.org/show_bug.cgi?id=112488 which not been resolved yet.
  Chrome resolved it in issue https://bugs.chromium.org/p/chromium/issues/detail?id=397856.
  Firefox and IE and Edge not using text-orientation at all, simply rotated mongolian glyphs 90deg clockwise in vertical writing mode.

Comment 22

[KariP]

Attachment: Safari Test Results

Sorry, I didn’t realize that Safari doesn’t support text-orientation without a prefix.

I attached an image (safari-mongolian.png Safari Test Results) which includes screenshots taken from Safari 10.1.1 (12603.2.4). The screenshots are:

1a) s51-text-orientation-Mongolian-upright-002.xht without -webkit prefix
1b) s51-text-orientation-Mongolian-upright-002-ref.xht without -webkit prefix

2a) s51-text-orientation-Mongolian-upright-002.xht with -webkit prefix
2b) s51-text-orientation-Mongolian-upright-002-ref.xht with -webkit prefix

Cases 3 and 4 are just to test how Latin and Japanese display on Safari.

3) Latin and Japanese characters without prefix.
4) Latin and Japanese characters with -webkit prefix. This seems to work for Latin and Japanese.

So, I wouldn’t worry too much about Safari, if it can be considered as a false positive. It means that the test doesn’t need to be changed.

Comment 23

[Gérard Talbot]

KariP,

Your Safari 10.1.1 test results screenshot is helpful. 

Mongolian is a vertical-only writing system and Mongolian glyphs must join (there should be no white space between glyphs of the same word). So, 2b ref with -webkit- prefix screenshot shows that there are several problems with Safari... albeit the problem seems to be related to the font itself: how glyphs are displayed in a Mongolian font, which vo value they are assumed to have: [U or R], which revision of UTR#50 they comply with, etc.

https://bugs.webkit.org/show_bug.cgi?id=112488#c10

states:

"
some font systems have pre-rotated [counter-clockwise 90 degrees, when defined in UTR#50] glyphs for these [vertical only] scripts. This is mentioned in CSS Writing Modes Level 3:
 # In some systems (e.g. when using OpenType fonts), to correctly orient
 # a character belonging to the Mongolian or Phags-pa script upright, the
 # UA must actually typeset it sideways.
"

So, at minimum, font creators for Mongolian language have to be aware of this, including

https://bugs.webkit.org/show_bug.cgi?id=112488#c65

"
implementers must interpret [vo value] U in the table to be [vo value] R for Mongolian and Phags-Pa, but because fonts being pre-rotated is not defined in Unicode
"

I am afraid I can not do much in here with regards to Safari. I believe, from reading bug reports like 112488, Safari developers are aware of the problem.

- - - - - - - -

> I wouldn’t worry too much about Safari, if it can be considered as a false positive. 
> It means that the test doesn’t need to be changed.

I can (should!) probably give a more specific, more precisely descriptive pass-fail-conditions sentence and then create a s51-text-orientation-Mongolian-upright-002-notref.xht and an associated mismatch link. This way, we surely should be able to avoid a false positive in Safari 10+.

Comment 24

[Gérard Talbot]

(In reply to siqinbilige from comment #21)

> Firefox and IE and Edge not using text-orientation at all, simply rotated
> mongolian glyphs 90deg clockwise in vertical writing mode.

This makes sense since Mongolian glyphs since revision 11 of UTR#50 have a vo value of R

Revision 11
    Characters moved from U to R: U+1800..U+18AF (Mongolian), U+A840..U+A87F (Phags-pa).

R 	Characters which are displayed sideways, rotated 90 degrees clockwise compared to the code charts.

Comment 25

[Jonathan Kew [:jfkthame]]

Attachment: patch 1 - Rearrange handling of textrun/glyph orientation so that we pass the resolved 'orientation' value of each glyph run from gfxFontGroup::InitScriptRun through to gfxFont::SplitAndInitTextRun, rather than just a boolean 'vertical' flag. (No function

This is a preparatory refactoring that doesn't change any behavior; it's just so we can fix the Mongolian/Phags-pa issue in a single place instead of having to special-case it in two parts of the code.

Comment 26

[Jonathan Kew [:jfkthame]]

Attachment: patch 2 - For Mongolian and Phags-pa script, always shape/render vertical text in a sideways-right mode (applying 90° rotation), even if text-orientation:upright is in effect, because fonts are designed to give the 'native' vertical rendering when used t

And then here's the patch that actually fixes the behavior for vertical Mongolian and Phags-pa, as per the testcase above.

Comment 27

[Xidorn Quan [:xidorn] UTC+10]

Should we add a testcase, or is there any existing testcase we can enable for this change?

Comment 28

[Jonathan Kew [:jfkthame]]

Attachment: Reftest for vertical Mongolian with text-orientation:upright

We should be able to do a simple reftest with the Noto font to verify that text-orientation:upright doesn't have any effect on Mongolian.

Comment 29

[Xidorn Quan [:xidorn] UTC+10]

Comment on attachment 8920092 [details] [diff] [review]
Reftest for vertical Mongolian with text-orientation:upright

Review of attachment 8920092 [details] [diff] [review]:
-----------------------------------------------------------------

r=me with the issues addressed.

::: layout/reftests/writing-mode/1361631-mongolian-upright-1-ref.html
@@ +1,1 @@
> +<meta charset=utf-8>

It's generally a good idea to add
> <!DOCTYPE html>
at the beginning of the file, so that it uses standard mode and have more predictable behavior (although it may not matter in this case).

@@ +1,5 @@
> +<meta charset=utf-8>
> +<style>
> +@font-face {
> +  font-family: test;
> +  src: url(../fonts/NotoSansMongolian-Regular.ttf);

It seems that we don't have this file in tree. Probably you should include this file.

Since it isn't that large (only ~130KB) it's probably fine?

::: layout/reftests/writing-mode/reftest.list
@@ +177,5 @@
>  fuzzy-if(gtkWidget,1,25) fuzzy-if(cocoaWidget,1,2) == 1302389-scrolled-rect-2b.html 1302389-scrolled-rect-2-ref.html
>  == 1302389-scrolled-rect-2c.html 1302389-scrolled-rect-2-ref.html
>  fuzzy-if(Android,54,852) == 1302389-scrolled-rect-2d.html 1302389-scrolled-rect-2-ref.html
>  
> +HTTP(..) == 1361631-mongolian-upright-1.html 1361631-mongolian-upright-1-ref.html

You don't need HTTP here. See bug 1318526.

Comment 30

[Jonathan Kew [:jfkthame]]

(In reply to Xidorn Quan [:xidorn] UTC+10 (less responsive Nov 5 ~ Dec 16) from comment #29)
> It's generally a good idea to add
> > <!DOCTYPE html>
> at the beginning of the file, so that it uses standard mode and have more
> predictable behavior (although it may not matter in this case).

True. Not important here, but I'll add it.

> > +  src: url(../fonts/NotoSansMongolian-Regular.ttf);
> 
> It seems that we don't have this file in tree. Probably you should include
> this file.

Included in this patch.

> > +HTTP(..) == 1361631-mongolian-upright-1.html 1361631-mongolian-upright-1-ref.html
> 
> You don't need HTTP here. See bug 1318526.

Ah yes, thanks! I tend to forget we made that change. (Old habits...)

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/06b56cb214c5
patch 1 - Rearrange handling of textrun/glyph orientation so that we pass the resolved 'orientation' value of each glyph run from gfxFontGroup::InitScriptRun through to gfxFont::SplitAndInitTextRun, rather than just a boolean 'vertical' flag. (No functional change.) r=xidorn
https://hg.mozilla.org/integration/mozilla-inbound/rev/3926e6d7c102
patch 2 - For Mongolian and Phags-pa script, always shape/render vertical text in a sideways-right mode (applying 90° rotation), even if text-orientation:upright is in effect, because fonts are designed to give the 'native' vertical rendering when used this way. r=xidorn
https://hg.mozilla.org/integration/mozilla-inbound/rev/245d06d41dce
Reftest for vertical Mongolian with text-orientation:upright. r=xidorn

Comment 32

[Xidorn Quan [:xidorn] UTC+10]

(In reply to Jonathan Kew (:jfkthame) from comment #30)
> > > +  src: url(../fonts/NotoSansMongolian-Regular.ttf);
> > 
> > It seems that we don't have this file in tree. Probably you should include
> > this file.
> 
> Included in this patch.

Oh, I see:
> New Binary File: layout/reftests/fonts/NotoSansMongolian-Regular.ttf

I somehow overlooked it... Sorry.

Comment 33

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/06b56cb214c5
https://hg.mozilla.org/mozilla-central/rev/3926e6d7c102
https://hg.mozilla.org/mozilla-central/rev/245d06d41dce