907793 - OpenSans font shows cedillas instead of commas (Şş instead of Șș) for Romanian language (bug in OpenSans font)

Status: RESOLVED FIXED

(www.mozilla.org :: L10N, defect, P1)

Description

[Francesco Lodolo [:flod]]

Our localizers for Romanian (ro) reported a problem with mozilla.org

Take for example the Partners page
https://www.mozilla.org/ro/firefox/partners/

Main title is: Croiește-ți propriul drum

The ș on Mac is rendered with a cedilla instead of a comma under the letter. Raul reports that also the ț has the same problem on his system (not sure if Windows or Linux).

Comment 1

[Raul Nicolae Malea]

Yes. Also this bug are present here:

- http://www.mozilla.org/ro/about/ - „Cunoaște mai multe despre Mozilla” , „.....sau pur și simplu unde să ne găsești, ați ajuns la locul potrivit.” etc. etc.

- http://www.mozilla.org/ro/products/ - „...promovarea și dezvoltarea Web-ului și menținerea lui deschisă....” , „Inspectați, depanați, analizați și vedeți mai multe în interiorul navigatorului dumneavoastră.” etc.

- http://www.mozilla.org/ro/contribute/  „...  Mozilla există pentru a promova deschiderea, inovația și oportunitatea pe internet. Facem asta de 15 ani și următoarele 15 date ...”


Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0 ID:20130821050136 CSet: b2486721572e

Comment 2

[Steven Garrity [:sgarrity]]

I can confirm this issue. I looks like the correct character is being used ("'LATIN SMALL LETTER S WITH COMMA BELOW' (U+0219)". The Open Sans Light font being used appears to have the character when I try it locally, or in Google Webfonts. Even the http://www.fontsquirrel.com/fonts/open-sans?q=open site where we got this particular version of Open Sans shows the character correctly in their preview too.

Anyone with more font/character-set expertise able to help us sort this out?

Comment 3

[Francesco Lodolo [:flod]]

Attachment: minimum_test.html

This is weird. I saved the page locally and started stripping down content until I reached this file.

If you remove lang="ro", or set it to something else (even lang="it", which doesn't have those characters), the problem disappears.

Comment 4

[Raul Nicolae Malea]

(In reply to Francesco Lodolo [:flod] from comment #3)
> Created attachment 794036 [details]
> minimum_test.html
> 
> This is weird. I saved the page locally and started stripping down content
> until I reached this file.
> 
> If you remove lang="ro", or set it to something else (even lang="it", which
> doesn't have those characters), the problem disappears.

Confirm. „ș” are see correctly in this test.

Comment 5

[Francesco Lodolo [:flod]]

Attachment: Updated minimum test

Updated minimum test with both lang=ro and lang=en-US displayed at the same time (save the file locally to try it).

I'm wondering if this should be moved over to Core -> Layout. Note that Chrome and Safari on my Mac are working correctly.

Comment 6

[Raul Nicolae Malea]

In my Chromium (Version 30.0.1553.0 (209444)) see correctly (first comma, second sedilla). Same to Opera 12.16 Linux. But in my Nightly not see first ș with comma.

Comment 7

[Raul Nicolae Malea]

Same problem in Google: http://code.google.com/p/android/issues/detail?id=9808 :) (I know, offtopic :D )

Same bug: https://bugzilla.mozilla.org/show_bug.cgi?id=404848 ?

Comment 8

[Cristian Silaghi]

Attachment: uite.png

I also have this issue on Ubuntu (virtualized) and Windows 8. Look at my attachment.

Best Regards,
Cristian Silaghi

Comment 9

[Raul Nicolae Malea]

Hello all, any news for this? :P Thanks!

Comment 10

[Jonathan Kew [:jfkthame]]

This is a bug in the Open Sans font. When the text is tagged as Romanian, the 'locl' (Localized Forms) feature replaces the glyphs "Scommaaccent" and "scommaaccent" (i.e. the default glyphs for U+0218 Ș and U+0219 ș) with "Scedilla" and "scedilla" (which are the default glyphs for U+015E Ş and U+015F ş).

In some versions of the font, or in some faces (I haven't checked them all, just a couple), there's a similar unwanted replacement for the [Tt]commaaccent glyphs in the Romanian 'locl' feature.

I suspect this was a result of a misunderstanding, and the intended Romanian localized forms feature should have done the opposite replacement: rendering the Unicode characters S and T with cedilla as comma-accent forms instead. This makes sense because historically, the Romanian comma-below forms were not always encoded separately from cedilla forms; in some systems and encodings, this was treated as a glyph variant for the same character code. As a result, there's a history of Romanian text being encoded with the characters with cedilla; the localized forms feature is intended to make such text display correctly.

(For more on the tortuous and confusing history of Romanian encoding, see [1]. It's not a pretty story. Some discussions about the 'locl' feature for Romanian - operating in the cedilla->commaaccent direction - can be found in font design forums, e.g. [2], [3].)

In short: this should be reported as a bug to the Open Sans font maintainers.


[1] http://blogs.msdn.com/b/michkap/archive/2011/08/24/10199324.aspx
[2] http://diacritics.typo.cz/index.php?id=9
[3] http://typophile.com/node/16705

Comment 12

[Raul Nicolae Malea]

More info here: https://groups.google.com/forum/#!topic/diacritice/5aVdqFzkapY
Change importance to major.

Comment 13

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I think the next step here is for somebody to figure out:
 (a) who maintains the OpenSans font, and
 (b) how to report a bug to them
and then do so.

Comment 14

[Cristian Adam]

Google Fonts entry for Open Sans: http://www.google.com/fonts/specimen/Open+Sans

"Open Sans is a humanist sans serif typeface designed by Steve Matteson, Type Director of Ascender Corp".
Unless somebody pays Ascender Corp. money, the chances of getting this font fixed are very slim.

Comment 15

[Jonathan Kew [:jfkthame]]

Well, the problem should certainly be reported to them.

It's Apache-licensed, so we or someone could make a corrected version, regardless of anything Ascender does.

Comment 16

[Cristian Adam]

I haven't researched for "Open Sans" but I know that "Droid" font family is also licensed from Ascender Corp. as Apache license. But it's not the source code of the font which is licensed, only the binary.

It took a couple of years to get the Droid fonts fixed for Romanian.

Redhat also licensed Liberation font families from Ascender Corp., but they payed more and got the source code, which they licensed first as GPL and then SIL Open Font license.

Comment 17

[Cristian Adam]

Reported this problem at Google Web Fonts early access discussion group:
https://groups.google.com/forum/?fromgroups=#!topic/early-access-fonts/S6C6e8AweSk

Comment 18

[Cristian Adam]

I've reported it to the (moderated) Google Web Fonts google groups directory:
https://groups.google.com/forum/#!topic/googlefontdirectory-discuss/ykg-IUvwZ7c

The fix seems very easy to do (if you have the proper tools):
"Yes, this substitution is reversed in Open Sans. Fixing can be done in  
a jiffy by exporting the GSUB table in DTL OTMaster, correcting the  
error in a text editor, and importing the improved features file in  
OTM (which will recompile the GSUB table)."

Comment 19

[Jonathan Kew [:jfkthame]]

Attachment: Open_Sans-fixed-ROM.zip

I don't have DTL FontMaster, but I've hacked the Open Sans fonts to fix the ROM (and MOL) language support using TTX.[1] I'm attaching an archive with both .ttf and .woff versions of the fonts.

In addition to reversing the 'locl' substitutions in the GSUB table, I added some text to the Unique ID field in the 'name' table to note that these are modified versions, as required by the Apache license; otherwise, these are identical to the versions currently available for download from Google Fonts; in particular, all the glyph data (including hinting, metrics, etc.) should be identical.

AFAICT, the resulting fonts work as expected for lang="ro" content. It'd be great if someone could verify this, and then get the updated versions installed on the Mozilla pages that use Open Sans as a web font.

[1] http://sourceforge.net/projects/fonttools/

Comment 20

[Denis Jacquerye]

FYI, some Romanian users prefer not to have any substitution at all: https://bugs.launchpad.net/ubuntu-font-family/+bug/635615
Given that more than 80% of Romanian data uses the cedilla characters, it’s understandable that the substitution should be applied, but at the same time this substitution make it impossible for the normal user to know which character is being used (which is good as it follows the style preference and bad as it breaks as soon as a different font is used or an application doesn’t treat both as equivalent like in search).

Cristian, when it comes to font binary code is often considered to be source code, as long as it follows the OpenType specifications.

> Redhat also licensed Liberation font families from Ascender Corp., but they
> payed more and got the source code, which they licensed first as GPL and
> then SIL Open Font license.

Liberation is still GPL, Liberation 2.0 fonts are really renamed Arimo, Cousine and Tinos which are OFL and have completely different hinting instructions. Where did you get that RedHat got source code for Liberation 1.0 fonts?
Fontforge sources were created from font binaries for both versions.

Comment 21

[Cristian Adam]

(In reply to Denis Jacquerye from comment #20)
> FYI, some Romanian users prefer not to have any substitution at all:
> https://bugs.launchpad.net/ubuntu-font-family/+bug/635615
> Given that more than 80% of Romanian data uses the cedilla characters, it’s
> understandable that the substitution should be applied, but at the same time
> this substitution make it impossible for the normal user to know which
> character is being used (which is good as it follows the style preference
> and bad as it breaks as soon as a different font is used or an application
> doesn’t treat both as equivalent like in search).

I am curious how did you get the 80% number? Ubuntu bug #635615 dates 2010,
things have changed since then, mostly Windows XP is being phased out.
http://gs.statcounter.com/#os-RO-monthly-201011-201311

> 
> Cristian, when it comes to font binary code is often considered to be source
> code, as long as it follows the OpenType specifications.

Good to know. Thank you.

> Liberation is still GPL, Liberation 2.0 fonts are really renamed Arimo,
> Cousine and Tinos which are OFL and have completely different hinting
> instructions. Where did you get that RedHat got source code for Liberation
> 1.0 fonts?
> Fontforge sources were created from font binaries for both versions.

I've got this impression by the fact that RedHat was able to fix some 
Romanian issues in the font:
https://fedorahosted.org/liberation-fonts/log/source?limit=100&rev=6a0937d12046ea7430315a89420ec4846dd37d42&mode=stop_on_copy&format=changelog

Comment 22

[Behdad Esfahbod]

(In reply to Jonathan Kew (:jfkthame) from comment #19)

> [1] http://sourceforge.net/projects/fonttools/

You should be switching to https://github.com/behdad/fonttools really.

Comment 23

[Raul Nicolae Malea]

Hello! Any news for this bug?

Comment 24

[Raul Nicolae Malea]

This bug are still present here: https://www-demo4.allizom.org/ro/firefox/desktop/

Perhaps for Romanian language need changed fonts used in Mozilla websites?

Comment 25

[Raul Nicolae Malea]

See here one good suggestion from Mișu Moldovan: https://groups.google.com/forum/#!topic/diacritice/5aVdqFzkapY

Comment 26

[Jonathan Kew [:jfkthame]]

(In reply to Raul Malea from comment #24)
> This bug are still present here:
> https://www-demo4.allizom.org/ro/firefox/desktop/
> 
> Perhaps for Romanian language need changed fonts used in Mozilla websites?

I already attached a zip file with fixed versions of the fonts; see comment 19. So what we need is for the fixed fonts to be deployed on the relevant sites.

Moving this to the www.mozilla.org component, in the hope that'll bring it to the attention of the right people to get the site(s) updated.

Comment 27

[Raul Nicolae Malea]

Hello guys. I see at this bug are still present. :(

Comment 28

[Pascal Chevrel:pascalc (PTO until April 26)]

(In reply to Jonathan Kew (:jfkthame) from comment #26)
> (In reply to Raul Malea from comment #24)
> > This bug are still present here:
> > https://www-demo4.allizom.org/ro/firefox/desktop/
> > 
> > Perhaps for Romanian language need changed fonts used in Mozilla websites?
> 
> I already attached a zip file with fixed versions of the fonts; see comment
> 19. So what we need is for the fixed fonts to be deployed on the relevant
> sites.
> 
> Moving this to the www.mozilla.org component, in the hope that'll bring it
> to the attention of the right people to get the site(s) updated.

Thanks Jonathan. I tried the patched fonts in a local install of mozilla.org and I can confirm that now it works. I am going to assign this bug to myself and see if I can get it fixed in prod soon.

Comment 29

[Pascal Chevrel:pascalc (PTO until April 26)]

Attachment: Capture d'écran 2015-06-17 à 16.11.32.png

Here is a screenshot of the Romanian home page with the patched fonts, we no longer have typos because of the wrong fonts.

Comment 30

[Jobava]

Thank you, Pascal!

This behavior or glyph substitution is apparently unique to the Romanian locale. It's problematic for text that contains both Romanian and Turkish since S-cedilla is a valid letter for Turkish, Azerbaijani and Turkmen. But other than those limited instances then substitution should work, from a typographical perspective at least.

Comment 31

[Pascal Chevrel:pascalc (PTO until April 26)]

Attachment: Link to Github pull-request: https://github.com/mozilla/bedrock/pull/3073

Comment 32

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/0711ccd734a301e1baf7a9fa99ce2ab15e98297b
Bug 907793: Fix Open Sans to be able to render Romanian
- The patched files are provided by Jonathan Kew, see https://bugzilla.mozilla.org/show_bug.cgi?id=907793#c19
- CSS files referencing the OpenSans font are updated with a comment to force a cache bust

https://github.com/mozilla/bedrock/commit/82307473ff5be27554fd4c818362ecfafd4aa9f4
Merge pull request #3073 from pascalchevrel/bug907793-fix-romanian-fonts

Bug 907793: Fix Open Sans to be able to render Romanian

Comment 33

[Raul Nicolae Malea]

So, this bug are fixed now?

Comment 34

[Pascal Chevrel:pascalc (PTO until April 26)]

This is fixed for www.mozilla.org, not for every site that uses Open Sans. So marking fixed as this was opened for this site.

Comment 35

[qdev labs]

The bug it's still there I see.

OS: Mac OS X Yosemite (10.10.4)
Browser: Firefox 41.0 (latest available)
Font: Open Sans (from Google web fonts)

țș are displayed (wrong) as ţş

Safari displays them correctly.

https://en.wikipedia.org/wiki/Romanian_alphabet#Comma-below_.28.C8.99_and_.C8.9B.29_versus_cedilla_.28.C5.9F_and_.C5.A3.29

Comment 36

[Francesco Lodolo [:flod]]

(In reply to qdev labs from comment #35)
> The bug it's still there I see.

This bug is fixed for www.mozilla.org. Where are you seeing the issue?

Comment 37

[qdev labs]

Here's a screenshot: http://labs.qdev.ro/screenshots/qdev_labs_2015-09-28_10-07-28_pm.png

I'm developing a new website, I (always) use Firefox for development side (cache disabled).

Comment 38

[Francesco Lodolo [:flod]]

(In reply to qdev labs from comment #37)
> Here's a screenshot:
> http://labs.qdev.ro/screenshots/qdev_labs_2015-09-28_10-07-28_pm.png
> 
> I'm developing a new website, I (always) use Firefox for development side
> (cache disabled).

This bug was filed specifically for mozilla.org, and it was caused by the font provided on the CDN. As said, it's fixed now.

I suggest you to file a different bug (Firefox::General should be a good start) if you see different behaviors between Firefox and other browsers, and also to provide an example online in order for other people to verify the mismatch (I would also check that the actual font used in all browsers is the same).

Comment 39

[Jonathan Kew [:jfkthame]]

(In reply to Francesco Lodolo [:flod] from comment #38)
> (In reply to qdev labs from comment #37)
> > Here's a screenshot:
> > http://labs.qdev.ro/screenshots/qdev_labs_2015-09-28_10-07-28_pm.png
> > 
> > I'm developing a new website, I (always) use Firefox for development side
> > (cache disabled).
> 
> This bug was filed specifically for mozilla.org, and it was caused by the
> font provided on the CDN. As said, it's fixed now.
> 
> I suggest you to file a different bug (Firefox::General should be a good
> start) if you see different behaviors between Firefox and other browsers,
> and also to provide an example online in order for other people to verify
> the mismatch (I would also check that the actual font used in all browsers
> is the same).

This isn't a Firefox bug, it's a bug in the Open Sans fonts. It was fixed on mozilla.org by deploying a corrected font, but if you're getting Open Sans from some other source, it's most likely still broken.

If you're stuck with using the bad Open Sans fonts, a workaround might be to try adding the CSS property

  font-language-override: "ENG";

so that Firefox does not use the (incorrect) Romanian alternate glyphs they provide.

Comment 40

[qdev labs]

(In reply to Jonathan Kew (:jfkthame) from comment #39)
> This isn't a Firefox bug, it's a bug in the Open Sans fonts. It was fixed on
> mozilla.org by deploying a corrected font, but if you're getting Open Sans
> from some other source, it's most likely still broken.
> 
> If you're stuck with using the bad Open Sans fonts, a workaround might be to
> try adding the CSS property
> 
>   font-language-override: "ENG";
> 
> so that Firefox does not use the (incorrect) Romanian alternate glyphs they
> provide.

Jonathan - thanks for the tip with font-language-override ! That did the trick ;)

Normally, the last part of my optimisation is to use font files served from local / client's server (as Google complains about their font cache lifetime) but that requires some (more) manual work (remove unnecessary glyphs, convert in different formats, aso) and depends of available time !

Comment 41

[qdev labs]

So basically it's using different character based on lang attribute of the page ;)

If I change <html lang="ro"> to <html lang="en"> the same (correct) result is rendered without the hack above, but this wouldn't be correct anymore as the real language is Romanian.

Strange that the other browsers do get the correct character without this hack !

Comment 42

[Jonathan Kew [:jfkthame]]

(In reply to qdev labs from comment #41)
> So basically it's using different character based on lang attribute of the
> page ;)

Different *glyph*, to be more precise.

> 
> If I change <html lang="ro"> to <html lang="en"> the same (correct) result
> is rendered without the hack above, but this wouldn't be correct anymore as
> the real language is Romanian.

Yes -- that's why I suggested the font-language-override property instead.

> 
> Strange that the other browsers do get the correct character without this
> hack !

That's because they don't support language-specific forms, they just use whatever the font's default happens to be.

Comment 43

[Jonathan Kew [:jfkthame]]

Oh, one last point: if you add the property

  -webkit-font-feature-settings: "locl";

you can see the incorrect glyphs in Chrome, too, as this will cause it to activate the (incorrect) "localized forms" feature in the Open Sans font.

The key difference between the browsers is that Firefox applies "localized forms" by default.

Comment 44

[qdev labs]

Thank you very much for ALL the clarifications !