1054780 - Gecko picks Segoe UI Emoji over Segoe UI Symbols on Windows 8.1, leading to colored (and not-CSS-color-alterable) symbols

Status: NEW

(Core :: Graphics: Text, defect)

Description

[iansan5653]

User Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0 (Beta/Release)
Build ID: 20140716183446

Steps to reproduce:

I'm trying to use the proper Unicode characters to develop a website, and these include such characters as ⚠ (warning sign U+26A0), ✉ (e-mail symbol U+1F4E7), ✔ (heavy check mark U+2714), and ⚡ (high voltage sign U+26A1), etc.


Actual results:

The characters appear prestyled with colors, but only in Aurora 33, not Firefox 31.


Expected results:

The characters should have been plain text as any other character.

Comment 1

[iansan5653]

Attachment: characters.png

Image of the effect.

Comment 2

[kenkon]

[bugday-20140818]

Hi. I viewed this page in Firefox 31.0 and Aurora 33.0a2 (build 20140817004001) on Windows 7 64-bit and the above characters appear identical in both versions (i.e. they appear normal).

Unfortunately I don't have Windows 8.x.

Did you use a clean profile?
Did you try with add-ons disabled?
Did you try the very latest Aurora/Nightly build?
Can you reproduce this issue on other OSes (e.g. Windows 7)?

Comment 3

[iansan5653]

I am using the most recent version of Aurora (33.0a2) on 64-bit Windows 8.1 and this occurs, on both computers I'm testing with. I know it's not the system because it does not happen with the latest release (31.0) or on Chrome. I am using a fresh profile in safe mode for all tests. I don't have another OS to test on.

Comment 4

[u468019]

User Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0

✉ is envelop U+2709, not 

Comment 5

[u468019]

User Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0

✉ is envelop U+2709, not 

Comment 6

[u468019]

Some people do seem to have altered their colors through CSS: http://graphemica.com/%E2%9C%89

Comment 7

[iansan5653]

I guess that the color can be styled after a font is selected that has that glyph. If you uncheck the font styling rule in the style inspector (which styles the envelope at http://graphemica.com/%E2%9C%89 as EmojiSymbols), it goes back to the yellow outline. It's still yellow in the tab title and URL bar no matter what.

Comment 8

[:Gijs (he/him)]

This is reproducible on Nightly on Windows 8.1. AFAICT Segoe UI Emoji is new in Windows 8, and that's what's causing issues here.

There might be a way to turn off Windows's coloring for the font (apparently this exists in XAML, but perhaps only for Windows Phone, and I don't know if they expose that any other way), which would be one option.

The simpler option seems to be using Segoe UI Symbol instead, which has at least all of these characters (I'm unfamiliar enough with unicode that I don't know if/when this is a complete substitution for the Emoji variant of the font, and/or for what ranges).

John, is the latter easy enough to do on Windows?

Comment 9

[:Gijs (he/him)]

(In reply to :Gijs Kruitbosch from comment #8)
> This is reproducible on Nightly on Windows 8.1. AFAICT Segoe UI Emoji is new
> in Windows 8, and that's what's causing issues here.
> 
> There might be a way to turn off Windows's coloring for the font (apparently
> this exists in XAML, but perhaps only for Windows Phone, and I don't know if
> they expose that any other way), which would be one option.
> 
> The simpler option seems to be using Segoe UI Symbol instead, which has at
> least all of these characters (I'm unfamiliar enough with unicode that I
> don't know if/when this is a complete substitution for the Emoji variant of
> the font, and/or for what ranges).
> 
> John, is the latter easy enough to do on Windows?

This time with needinfo...

Comment 10

[Jonathan Kew [:jfkthame]]

We could move Segoe UI Symbol before Segoe UI Emoji in the fallback lists at

  http://mxr.mozilla.org/mozilla-central/source/gfx/thebes/gfxWindowsPlatform.cpp#724
  http://mxr.mozilla.org/mozilla-central/source/gfx/thebes/gfxWindowsPlatform.cpp#803

and that would presumably help in this case. However, it would mean that people who enjoy seeing colored emoji in their text, and are getting them by default right now on Win8.1, would revert to the monochrome glyphs. Which is the better default behavior? I don't know.

Authors who explicitly want their sites to show the monochrome glyphs can add "font-family:Segoe UI Symbol;" to their CSS to ensure that we use that font (if present) rather than potentially-colorful fallback.

(Note that OS X also has a color-emoji font, as do modern Android systems; the character coverage may vary across systems, but I suspect there's an increasing possibility that various symbols may be displayed with colorful glyphs if authors do not explicitly control the font choices for them.)

Comment 11

[John Daggett (:jtd)]

For the most part, I concur with Jonathan here.

Given that the reporter appears to want to use these characters as icons, I think the simplest solution is to provide a font containing those symbols. Especially for characters in the emoji range, I doubt it will be possible to find fonts with monochrome glyphs on all platforms.

For the "emoji" Unicode range (U+1F300–1F5FF) I think it's reasonable to favor emoji fonts over symbol fonts for font fallback. For mixed symbol ranges, I think we should always favor symbol characters rather than emoji fonts. This means that authors who want to use monochrome glyphs for characters in this ranges will need to either provide a font or choose a fontlist containing exclusively symbol fonts (e.g. Segoe UI Symbol rather than Segoe UI Emoji).

So the only thing I think that makes sense to change at this point is to favor Segoe UI Symbol before Segoe UI Emoji for symbol ranges. But that won't solve the reporter's problem for all the codepoints they have listed here.

Comment 12

[John Daggett (:jtd)]

Attachment: patch, move up Segoe UI Symbol for symbol ranges

For symbol ranges, move Segoe UI Symbol ahead of Segoe UI Emoji but leave the priority for emoji ranges the same.

Comment 13

[Makoto Kato [:m_kato]]

Comment on attachment 8476478 [details] [diff] [review]
patch, move up Segoe UI Symbol for symbol ranges

Review of attachment 8476478 [details] [diff] [review]:
-----------------------------------------------------------------

for compatibility, we should move down color emoji font for bmp area.

Comment 14

[Jonathan Kew [:jfkthame]]

(In reply to John Daggett (:jtd) from comment #11)
> For the most part, I concur with Jonathan here.
> 
> Given that the reporter appears to want to use these characters as icons, I
> think the simplest solution is to provide a font containing those symbols.
> Especially for characters in the emoji range, I doubt it will be possible to
> find fonts with monochrome glyphs on all platforms.
> 
> For the "emoji" Unicode range (U+1F300–1F5FF) I think it's reasonable to
> favor emoji fonts over symbol fonts for font fallback. For mixed symbol
> ranges, I think we should always favor symbol characters rather than emoji
> fonts.

Maybe this is the best compromise, but the problem it has is that when emoji were added to Unicode, there was a deliberate decision to unify them with existing symbols where possible, and so we have a scattering of the emoji symbols -- as derived from the original Japanese carrier sets, etc -- that are encoded among the BMP symbol ranges. Things like the basic WHITE SMILING FACE at U+263A, and symbols like AIRPLANE (U+2708) and ANCHOR (U+2693), Zodiac signs, etc.

So if we fall back differently for the SMP emoji block compared to the BMP symbols, then users will find that *most* emoji and similar pictorial symbols appear colored by default, but certain ones "fail" and give the old b/w symbol rendering, and I suspect most authors and users will have no idea why.

The "emoji" view in the OS X character palette, for example, includes a number of such unified symbols, and displays them in full color. I think if authors insert such symbols into a document (without doing anything specific to control the font), and they end up rendering in a mixture of b/w and color, it'll seem like a bug to them.

Comment 15

[John Daggett (:jtd)]

Hmmm, so are you saying for all emoji codepoints, BMP and SMP, we should always prefer the emoji font?

I definitely agree that there's ambiguity here. But I think to *always* prefer an emoji font will lead to lots of problems in documents that use symbols. I think it's better to prefer symbol fonts for system font fallback and address these emoji problems in other ways.

I posted a couple proposals on www-style.

http://lists.w3.org/Archives/Public/www-style/2014Aug/0322.html

  font-variant-color: auto | color | monochrome

This allows an author to explicitly disable color glyphs with 'font-variant: monochrome".

http://lists.w3.org/Archives/Public/www-style/2014Aug/0323.html

A new generic font, 'emoji'. This allows authors to explictly favor an emoji rendering for symbol characters.

When discussing this with Kato-san this morning he mentioned that one of the carriers here in Japan wants to ship with an emoji font that supports color glyphs for an even wider range of characters, including things like punctuation characters, the copyright symbol, etc. We definitely want to give authors a CSS way of controlling this rather than trying to guess intentions within system font fallback code.

Comment 16

[Jonathan Kew [:jfkthame]]

(In reply to John Daggett (:jtd) from comment #15)
> Hmmm, so are you saying for all emoji codepoints, BMP and SMP, we should
> always prefer the emoji font?

I don't think I'm saying that; I'm saying that I think it's far from clear what the most appropriate fallback order should be. Your patch here changes the order for the BMP symbols, which *may* on balance be a better choice, but IMO it's not at all clear-cut. While it will help some cases, there are likely to be others that it hurts, and I don't know how to decide among the trade-offs...

It's not always clear whether a given Unicode codepoint is an "emoji codepoint" or some other kind of symbol.

> 
> I definitely agree that there's ambiguity here. But I think to *always*
> prefer an emoji font will lead to lots of problems in documents that use
> symbols. I think it's better to prefer symbol fonts for system font fallback
> and address these emoji problems in other ways.
> 
> I posted a couple proposals on www-style.
> 
> http://lists.w3.org/Archives/Public/www-style/2014Aug/0322.html
> 
>   font-variant-color: auto | color | monochrome
> 
> This allows an author to explicitly disable color glyphs with 'font-variant:
> monochrome".

Yes, that's probably a useful thing. Though authors can achieve pretty much the same thing, at least for well-known platforms with reasonably predictable font repertoires, by appropriate additions to the font-family list.

> 
> http://lists.w3.org/Archives/Public/www-style/2014Aug/0323.html
> 
> A new generic font, 'emoji'. This allows authors to explictly favor an emoji
> rendering for symbol characters.

Another option for authors would be to use the Unicode variation selectors U+FE0E/FE0F to choose between "text style" and "emoji style" for the symbols where this distinction is defined; see http://www.unicode.org/Public/UNIDATA/StandardizedVariants.html.

Comment 17

[Laurent Barbareau]

When developing an app, this is always surprising to see that a unicode character has changed of color whereas nobody is supposed to have modified it... (just because you've changed of FF version since last time you've displayed the page containing such characters).

Moreover, when you open the FF developer window, you discover that the unicode character is always colored with a color that is never specified anywhere, neither CSS rules nor computed CSS rules. And if you try to modify its color attribute, it doesn't change anything.

So, as it seems difficult to find a solution to this issue (basing that assessment on the fact it didn't moved for a long time), I propose to make at least something into the developer window, informing clearly the developer when a unicode character is intrinsically colored because of its glyph.

I have no particular idea to implement it and if it's a good option, I'm just trying to avoid to others loosing their time with that kind of issues.

Comment 18

[cincodenada]

I just ran into this issue myself developing a web application.  As a web developer and someone interested in emoji and Unidcode, I think the best solution is to properly support the Unicode Consortium's existing solutions to this issue: namely, the "text style" and "emoji style" variation sequences that Jonathan Kew mentioned, as documented here: http://www.unicode.org/emoji/charts/emoji-variants.html and the Emoji_Presentation attribute here http://www.unicode.org/emoji/charts/emoji-style.html - both are more formally addressed here: http://www.unicode.org/reports/tr51/index.html#Presentation_Style

With these properly supported, this character: ▶︎ (U+270C U+FE0E) should always render in black-and-white, styleable "text style" (in this case, using Segoe UI Symbols), whereas this character: ▶︎ (U+270C U+FE0F) should always render in full-color "emjoi style" (e.g. using Segoe UI Emoji). A character without variant (▶, U+25B6) should be rendered depending on its Emoji_Presentation attribute.

As an example, most of the mentioned symbols (▶, ⚠, ✉, ✔) are all default text presentation, and thus without a variation selector should be rendered text-style.  ⚡ is default emoji presentation.
 
I don't know the details of implementing such support, as I don't know how the existing support was added or anything about font-rendering in Firefox, so I can't speak to implementation.  But it seems to me that this kind of decisionmaking about what should be symbols, what should be emoji, etc has already been well-explored by the Unicode Consortium and they have offered a reasonable standard that Firefox should implement, rather than trying to wade into such decisionmaking on its own.

For reference: I am on Ubuntu/FF 47, and following the excellent guide here http://probablement.net/txt/emojilinux have gotten all my ducks in a row to have color emoji support where available, such as Firefox.  My Firefox, at least, seems to ignore Variation Selectors and the Presentation attribute, rendering all emoji characters in emoji style.  The "text-vs" row at www.unicode.org/emoji/charts/emoji-style.html should be all rendered text-style, in contrast with the "text+vs" box where the characters have the variation selector, but in my browser both rows they are rendered identically.

(Sidenote: for those who want to investigate unicode symbols and the variantion selectors, I recommend http://r12a.github.io/apps/conversion/ as an easy, useful, open-source tool to do so)

Comment 19

[cincodenada]

An additional note on my thoughts as a web developer: when supporting the Unicode standards, you get sensible default behavior for most cases. Symbol-like emoji such as ▶, ⚠, ✉, ✔ used as UI elements will by default be style-able, more purely emoji symbols such as 

Comment 20

[cincodenada]

Eek, the system seems to have eaten part of my comment.  Continued:

...such as 

Comment 21

[cincodenada]

Um, okay, Bugzilla seems to be breaking on emoji-range emoji, so I will have to omit them.  Continued:

...such as [pizza, laughing with tears, cactus] will by default be rendered full-color emoji-style.  It's the best of both worlds: sensible defaults, with developers still having the ability to control the end-result with Variant Selectors if they need to.  Awareness of the variant selectors is already out there (see this SO question: http://stackoverflow.com/questions/32915485/how-to-prevent-unicode-characters-from-rendering-as-emoji-in-html-from-javascrip#32916381), but lack of support will hamper its adoption.

Other solutions proposed here, such as simply defaulting all to text-style, or choosing arbitrary Firefox-selected ranges to show as default emoji vs text, are awkward and unnecessary.  The Unicode Consortium has already done the hard work of choosing sensible defaults for us and offered up an excellent standard for us to follow.  Firefox should use it.

Comment 22

[Jonathan Kew [:jfkthame]]

Yes, it seems obvious that we should try to use the defaults specified by Unicode, along with supporting the variation selectors to explicitly choose one rendering or the other.

However, it becomes less obvious what to do when the character is explicitly styled (via CSS font-family) with a particular font; that font doesn't support both the text- and emoji-style presentations; and the presentation that it offers conflicts with the style requested by a variation selector, or the default specified by Unicode for the given character.

To take an example, suppose you have

  <span style="font-family:Segoe UI Emoji">&#x270C;&#xFE0E;</span>

where the variation sequence calls for text-style, but the specified font only has a color emoji-style glyph. What should happen?

Conversely, what about

  <span style="font-family:Segoe UI Symbols">&#x270C;&#xFE0F;</span>

which has the opposite issue?

In these examples, should the variation selector override the font choice, or vice versa?

And the same question arises regarding the Unicode-specified default presentation style, too. Consider

  <span style="font-family:Segoe UI Emoji">&#x270C;</span>

and recall that U+270C is defined as having default-text presentation. Should we enforce that, or respect the specified (emoji) font? And in the opposite case, if U+270B (which is specified as emoji-style by default) is styled with a text-only font family?

I do think there are ways in which the existing behavior can be improved (in particular, I think bug 1032671 should help), but it's not clear to me that there's any definitively "right" answer to all the potential cases here.

Comment 23

[cincodenada]

(In reply to Jonathan Kew (:jfkthame) from comment #22)

Indeed, there are lots of edge cases here - but 1032671 does seem to be heading in the right direction in integrating the default presentation attributes, thanks for that pointer.

On the emoji-style example page, the second table is an example of expected behavior in your last example (Emoji font specified for default-text character), and specifies that they should be shown emoji-style if specified with an emoji font.  UTR#51 also notes: "For example, in some CSS implementations, if any font in the lookup list is an emoji font, then emoji presentation is used whenever possible.", supporting that behavior.

The Font + variation selectors are more ambiguous, because there are two specification methods in direct conflict.  I would lean towards honoring variation selectors, just because it seems more useful.  It's more likely that I would want to have most things one style and specify a few characters as different, than having existing text with selectors that I want to override.  It also goes along with the cascading nature of CSS - the most specific "rule" wins.

But that's just my reaction, those are indeed much less clear-cut.  But as you noted, the existing behavior can be improved, I'm happy to leave those edge cases up to whatever makes sense when implementing, as they seem pretty undefined.  In the end, the Unicode Consortium does have the disclaimer "presentation style is never guaranteed" - but where there is reasonable consensus, it'd be nice to go along with it.  And it seems like bug 1032671 is moving in that direction indeed.

My apologies if I came off as strident - I'm aware that being simply an interested member and not (yet) a contributor, I am unaware of many complexities of implementation, existing efforts, and the like.  My intent was to summarize my interpretation of the Unicode Consortium's recommendations in this context, and register my vote as a web developer and web user to hew closely to it when possible.

Thanks for the feedback and info!