1366134 - [UX] Decision about locale selection in environment where Firefox language doesn't match OS language

Status: RESOLVED FIXED

(Core :: Internationalization, defect, P3)

Description

[Zibi Braniecki [:zbraniecki][:gandalf]]

Over the course of three releases - between 53 and 55 - we reworked how we handle language selection and locale-aware date/time formatting.

We ended up with a pretty flexible API that can handle picking up regional settings from the OS or using locale defaults from Firefox.
The setup works well, except for a scenario where the OS language is completely different from Firefox language.

At the moment, there's no "standard" in the industry for solving such scenarios. Mac, Windows, Android, and iOS all handle that differently, and there are differences between apps as well. The model Gecko is using now is a compromise and it does not satisfy all use cases.
I'd like to get an opinion from several UX/Intl/L10n stakeholders at Mozilla on the current setup and potential changes we can make.

I'd like to note that it seems to me that there's no "perfect" solution, and each choice makes one setup work while it breaks another.

======= Setup

To picture the matrix of choices, let's imagine that we want to display a date that formatted to "en-US" would look like this: "Thursday, 5/18/2017".
For the sake of this example, I'll use two more languages - "pl", as an example of a language that two of the users reporting their setups not to work use, and "ar", as an example of a language that exemplifies major issues in an attempt to mix languages.

en-US: "Thursday, 5/18/2017"
pl: "czwartek, 18.05.2017"
ar: "الخميس، ١٨/‏٥/‏٢٠١٧"

====== Problem

When the user has OS and Firefox language selection matching, the consistency is preserved. All translations in the Firefox UI will match the localized date and time strings.
The issues arises when the user is using an operating system in one language, and Firefox in another.
So far, it seems that the more common case for the problem is to use Firefox in en-US and OS in another language.
For the sake of this example, we'll assume that Firefox is in "en-US" and OS is either in "pl" or "ar" locale.

Several users who use such setups expressed [0] that they'd like Firefox en-US to pick up "pl" or "ar" regional settings from Windows.

This creates a consistency and UX problem.

The consistency problem comes from the mix of Firefox translation into one language, and date/time formatting into another where formatting of date also involves translation of month or weekday names.

An example message which exemplifies this problem is "Today - Thursday, 5/18/2017".
The localization message in en-US is: "Today - { $date }".

If Firefox would pick up the Windows regional settings language, the message would be displayed as:

pl: "Today - czwartek, 18.05.2017"
ar: "Today - الخميس، ١٨/‏٥/‏٢٠١٧"

The first example, of "pl", is just an inconsistency that may be expected or unexpected for the user.
The latter is more problematic due to the mixing of RightToLeft and LeftToRight we not only introduce inconsistent product translation, but also mess with layout directionality, which may have consequences for any rich layout that are hard to plan for.

The UX problem comes from the fact that in certain scenarios, different languages may get the same date string meaning different things.
For example "5/7/2017" in "en-US" means May 7th 2017, while in "fr" it would mean July 5th 2017.

There are two ways to think about it:
 - since the user set their regional preferences to "fr", he expects the date/time to be in "fr", so he'll read "5/7/2017" in a french manner irrelevant of Firefox localization he uses.
 - since Firefox is in en-US, the user has the right to expect the date string to be matching en-US and he'll read "5/7/2017" in an American English manner since all the UI around is in American English.

This problem will escalate further as we will start adding more internationalization APIs, like for unit formatting (35 meters, 15 kilobytes, 5 grams) and we'll have to decide in which language to display them - Firefox, or OS?

=== Manual mingling

On top of just selecting a language for regional settings, the user can also manually toy with the date/time formatting patterns.
That allows the user to build a custom pattern for English date, like "2017/Tuesday/03" by providing "Y/D/m" in Windows settings.

That adds another dimension to consideration because we can assume that if the user put so much thought to manually select their date format, they really want us to use it.

On the other hand, that of course brings another layer of potential confusion. Letter "M" may be narrow variant of a weekday in English (Monday - M), or a narrow version of the polish month name (Marzec - M).

=== Other products

I tested several product in the Windows 10 environment:
 - Microsoft Mail
 - Microsoft Edge
 - Chrome
 - Opera

Both Microsoft products, use the regional settings from the Operating System. They do not allow the user to select a locale for the product, so they also match OS locale all together.
Windows itself, allows the user to select a language, and different locale for regional preferences, so things like "Today is لخميس،" are perfectly possible if the user will select English language and Arabic regional settings.

On the other hand, Chrome and Opera allow user to select the language of the product, and they format date/time to match the selected language, so irrelevant of Windows regional preferences, they'll always display "Today is Thursday" or "اليوم هو الخميس" depending on the product locale.

=== Firefox

At the moment in Firefox, our behavior is closer to what Chrome and Opera do, so we keep the consistency between date/time formatting and the product locale, but we look into regional locale selected by the user and if it matches the language portion, we use the regional preferences to format the date/time.

That means that if the user has Windows in "en-AU" and Firefox in "en-US" we recognize that both are in English, and use the Windows date pattern.
This allows the user to manually mingle with the patterns but preserves the consistency and directionality of the Firefox User Interface.
Same works with "fr-CA" and "fr-FR" etc.

On the other hand if the user selected different *language* for date/time settings, than Firefox UI is in, then we assume that we cannot use the OS regional settings, and we use the patterns for the Firefox UI locale.

So if the Windows are in "pl" or "ar", and Firefox is in "en-US", we keep the dates in "en-US" and we ignore the manual customizations of date/time patterns from Windows.

I'd like to seek a consensus between Intl/L10n and UX module owners and peers at Mozilla on whether we should stick to the model we use right now, follow Microsoft products and open it up to take any regional settings from the OS, or follow Chrome and Opera and close it down to get regional settings follow only the region of the product's locale.



[0] bug 1344594, bug 1355307, bug 1354339 are examples

Comment 1

[Zibi Braniecki [:zbraniecki][:gandalf]]

Attachment: Firefox History UI with (en-US Firefox, pl Windows, uses en-US for datestime)

Comment 2

[Zibi Braniecki [:zbraniecki][:gandalf]]

Attachment: Windows Mail UI (en-US Language, pl Regional Settings, uses pl for datetime)

Comment 3

[Zibi Braniecki [:zbraniecki][:gandalf]]

Attachment: Edge History UI (en-US Windows, pl regional settings, uses pl for datetime)

Comment 4

[Zibi Braniecki [:zbraniecki][:gandalf]]

Attachment: Chrome History UI (en-US Chrome, pl Windows Regional settings, uses en-US for datetime)

Comment 5

[Zibi Braniecki [:zbraniecki][:gandalf]]

Attachment: Opera History UI (en-US Opera, pl Windows regional settings, uses en-US for datetime)

Comment 6

[Ben Bucksch (:BenB)]

> At the moment, there's no "standard" in the industry for solving such scenarios. Mac, Windows, Android, and iOS all handle that differently, and there are differences between apps as well.

While I'm sure there are apps that handle things differently, there is a clear standard in the industry: To have separate settings for language and date/time formats (often called region), and for apps to use the OS setting.

So far, Firefox has observed the OS setting for the date/time format.
Zibi changed it in FF53 so that the date format should match the Firefox build language.

However, it goes against almost every application on almost every operating
system, all the way back to Windows 2.0, GeoWorks and Mac, up to most current
Windows 10 and macOS Sierra. All these have *separate* settings for region and
date/time formats that can be changed independently from the UI language.

And almost all native applications on all OS follow that OS region settings and not
the app language.

There's an important reason for that: Date and time formats are highly dependent on the country, and not on the language. The most obvious example is US vs. UK: 04/01/17 means Apr 1 in the US and Jan 4 in the UK. There's no way to tell which one is used, so if the app choses the one that the user does not expect, that's going to trip him off extremely. There's no way that a normal user will learn that 04/01/17 means Apr 1 in one application and Jan 4 in another application. It is *crucial* that all applications behave the same. Consequently, applications *MUST* follow the OS setting.

This matters not only for UK vs. US, but for everybody around the world who prefers English software, but is home in another culture. But I am German and used to 24h clocks, and need to see 19:00, not 7 PM. There are probably a million non-English people around the world who use English software, and they will not be able to cope with US date/time formats.

This is a serious regression. Personally, I am no longer able to use recent self-compiled Thunderbird builds, because of this bug. Many people have complained about this, in various bugs: bug 1354339, bug 1355307, bug 1344594 and others.

Comment 7

[Ben Bucksch (:BenB)]

Everybody, please take a look at:
http://www.guidebookgallery.org/screenshots/international
It shows that essentially all OSes treat date/time formats separate from UI language.

Comment 8

[Zibi Braniecki [:zbraniecki][:gandalf]]

Please, do not change the title of the bug.

Comment 9

[Jorg K (CEST = GMT+2)]

Thank you, Zibi, for your well written description of the problem (comment #0). You haven't asked me, but I will give you my opinion anyway ;-)

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #0)
> I'd like to seek a consensus between Intl/L10n and UX module owners and
> peers at Mozilla on whether
> 1) we should stick to the model we use right now,
> 2) follow Microsoft products and open it up to take any regional settings from
> the OS,
> 3) or follow Chrome and Opera and close it down to get regional
> settings follow only the region of the product's locale.

IMHO 3) is not possible, since Mozilla products don't provide localisations for en-AU, en-India and many other flavours of English, it's even worse for French. I consider 1) an improved version of 3).

I don't have strong feelings about whether to do 1) or 2). Personally I find it quite ugly to see
  "Today - Freitag, 19. Mai 2017"
in an otherwise English product. I think there is only a small group of people who require this. On the other hand, there is no harm done implementing 2) since whoever wants it ugly, can get it ugly, the ones who want "consistent" will still get consistent. There could also be a preference to switch between 1) and 2).

Comment 10

[Zibi Braniecki [:zbraniecki][:gandalf]]

> You haven't asked me, but I will give you my opinion anyway ;-)

Apologies Jorg! I was sure I did set an NI on you as well! Thank you for providing your feedback :)

Comment 11

[Zibi Braniecki [:zbraniecki][:gandalf]]

To help people on NI asses the significance:

> But I am German and used to 24h clocks, and need to see 19:00, not 7 PM (...)
> This is a serious regression. Personally, I am no longer able to use recent self-compiled Thunderbird builds, because of this bug. 

As a workaround for the missing "en-DE", BenB can use en-GB version of Thundebird to get 24h clock and English UI.

Comment 12

[Jorg K (CEST = GMT+2)]

Not really, since he self-compiles en-US.

Comment 13

[Ben Bucksch (:BenB)]

Yes. But even if I use normal builds, UK format is still wrong.
My date format is "17.05.2017", not "17/5/17".

I would like to point out that most times when we write dates or times, there are no words, only numbers and separators. I don't care whether I get "Monday" or "Montag" (given that I'm on a mixed system, I clearly understand both), feel free to use the build language for words, as long as the format follows my date/time region settings.

ZiBi: Your title is flat-out wrong. You write "locale" when you mean "date/time format", and "OS language" when you mean "OS region setting". I find it offensive that you reset my change. To avoid an edit war, I refrain from correcting it again, but I fail to understand your attitude.

Comment 14

[Zibi Braniecki [:zbraniecki][:gandalf]]

> ZiBi: Your title is flat-out wrong. You write "locale" when you mean "date/time format", and "OS language" when you mean "OS region setting". I find it offensive that you reset my change. To avoid an edit war, I refrain from correcting it again, but I fail to understand your attitude.

Ben, I mean locale selection. I mean OS language. I'm the author of the bug.

Comment 15

[Ben Bucksch (:BenB)]

Attachment: TB Calendar in 52: Correct German date, time, and English "May"

This screenshot shows that it is indeed possible to use English words (from the app language) and at the same time the German OS region date/time formats. This is what 52 did.

That said, almost all places that show a date and time use the short format. Particularly the most prominent ones. Very few places use words at all, so the problem does arises only in very few places. And even these can be fixed without using English date formats.

Comment 16

[Ben Bucksch (:BenB)]

Zibi, *nobody* is proposing to follow OS language, but OS region and date/time settings. And "locale" != date/time format - that's the whole point here. You're deliberately confusing the issue. OK, fine, have your way.

Comment 17

[Francesco Lodolo [:flod]]

Sorry for pushing the discussion one step back, but I need some help understanding: this bug seems to be only about which locale to pick when Application and OS diverge. In your example, Firefox is in English but the OS is in Polish, which one do I pick? I think that's one problem (#A).

Then there's problem #B: I have customized regional settings in my OS. So, Firefox in English, OS in Polish, and a random combination of regional settings. Is this out of scope for this bug and covered by another, or completely out of scope?

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #0)
> 1) we should stick to the model we use right now,
> 2) follow Microsoft products and open it up to take any regional settings from
> the OS
> 3) follow Chrome and Opera and close it down to get regional
> settings follow only the region of the product's locale.

I'm not sure if it's covered by any of these, but would it be an option to let the user decide? Following your example, Firefox in English and custom regional settings in the OS, have a setting that lets the user decide what to use.

Regional settings:
a) Follow browser language (en-US) - Default
b) Follow OS language (pl)
c) Use custom regional settings from OS

Example: Friday, 5/15/2017 - 5.00 $

The default being the current status (follow browser's language).

Comment 18

[Zibi Braniecki [:zbraniecki][:gandalf]]

Attachment: Microsoft Mail (Language English, Regional Settings, manually modified italian, Mail is showing dates in Italian, ignoring modifications)

> Zibi, *nobody* is proposing to follow OS language, but OS region and date/time settings.

OS region and date/time settings exist in a language and, unless modified by hand, they derive from a locale.
The question I asked here, on purpose, is, what locale should we format date/time into, when OS language is different from Firefox language selection.

In short, we need to know how to translate weekday, month name, should we show eastern or western numerals, should we display the result string in LTR or RTL. Those bits depend on language we're in.

> You're deliberately confusing the issue.

Please, stop assuming bad intentions just because you disagree.

> This screenshot shows that it is indeed possible to use English words (from the app language) and at the same time the German OS region date/time formats. This is what 52 did.

Can you produce the same screenshot for Arabic OS region date/time formats and en-US Thunderbird please?

> Then there's problem #B: I have customized regional settings in my OS. So, Firefox in English, OS in Polish, and a random combination of regional settings. Is this out of scope for this bug and covered by another, or completely out of scope?

This is totally within the scope of this discussion, but I'm afraid it doesn't work the way you expect it.
That's not "a random combination", but a manually modified set in a given language.

So, for example, if you select your Windows language to "English (American)", and your Regional Settings to Italian, and then modify date pattern to "Y-m-d" you're still going to see italian date string, just based on different pattern.

My point here is, the behavior of Microsoft Windows, Microsoft Mail and Microsoft Edge is not to show you "2017-March-27", but "2017-marzo-27". And if instead of Italian, you select Arabic, it will also translate numbers and delimiting literals.

While looking into the manual modifications I also noticed that both Microsoft Edge and Microsoft Mail *ignore* manual modifications.

The screenshot attached with this comment is from English Windows, with Italian Regional Settings and both short and long date modified.
Microsoft Mail displays UI in English, Dates in Italian, but takes the default patterns for Italian and doesn't follow custom modifications.

Comment 19

[Zibi Braniecki [:zbraniecki][:gandalf]]

Attachment: Microsoft Edge and Mail (Language English, Regional Settings, manually modified arabic, Mail is showing dates in arabic, ignoring modifications, Edge respecting, both show a mix of English and Arabic)

Here's an example in English Windows with manually modified Arabic settings.

Things to notice:

 * Mail ignores manual modifications showing default values for arabic
 * Edge respects manual modifications showing modified values for arabic
 * Both show a mix of en-US dates and arabic dates

Comment 20

[Zibi Braniecki [:zbraniecki][:gandalf]]

> I'm not sure if it's covered by any of these, but would it be an option to let the user decide? Following your example, Firefox in English and custom regional settings in the OS, have a setting that lets the user decide what to use.

> Regional settings:
> a) Follow browser language (en-US) - Default
> b) Follow OS language (pl)
> c) Use custom regional settings from OS

> Example: Friday, 5/15/2017 - 5.00 $

> The default being the current status (follow browser's language).

The only difference here is that for the third one, we still need to know which language we're in.

That extends the matrix of possibilities:
> Regional settings language:
> a) Follow browser UI language (en-US) - Default
> b) Follow OS regional settings language (pl)

> Regional settings
> a) Use custom regional settings from OS - Default
> b) Use default for my regional settings language

In bug 1355307 comment 17 I gave an example of a problematic API where we'd need to justify the default locale being one thing in some scenarios, and a different thing when we follow OS. One solution would be to separate DefaultLanguage and DefaultRegionalSettingsLanguage in JS Context.

Comment 21

[Ben Bucksch (:BenB)]

> In short, we need to know how to translate weekday, month name, should we show eastern or western numerals,
> should we display the result string in LTR or RTL. Those bits depend on language we're in.

Now I'm confused. The issue we have is the format (order of elements, and their separators), not the content of the variables (month names). You are talking about the latter now.

Do you accept that we should use the OS region settings for date/time *format*?

If you accept that, and that will be the case, including the user customizations of date/time formats, then it's nice if we can use the Firefox language for the month names. I think that's what you care about, and nobody objects to that, as far as I saw. As long as the OS date/time *format* is used.

Comment 22

[Zibi Braniecki [:zbraniecki][:gandalf]]

(In reply to Ben Bucksch (:BenB) from comment #21)
> > In short, we need to know how to translate weekday, month name, should we show eastern or western numerals,
> > should we display the result string in LTR or RTL. Those bits depend on language we're in.
> 
> Now I'm confused. The issue we have is the format (order of elements, and
> their separators), not the content of the variables (month names). You are
> talking about the latter now.

Separators are also in language (and script). Numbers are also in language (and script).
Basically, the order and presence of pattern tokens is the only thing that is irrelevant of the language.

> Do you accept that we should use the OS region settings for date/time
> *format*?

Not sure.
That's why I started this discussion.

I see three paths for when OS regional settings language differs from product language:

1) Follow product language and patterns for that language
2) Follow OS language and patterns for that language
3) Follow product language and patterns for the OS

The first is what Chrome and Opera do. The second is what Microsoft Windows, Mail and Edge do.
I could not find any product that would do the third and I can imagine how problematic it would be if the pattern was in Arabic, and month name in English.

So, in short, I'm concerned that following pattern without language will not work. And hence I want to discuss what language we follow, before I discuss what pattern we follow :)

Comment 23

[Ben Bucksch (:BenB)]

> Separators are also in language (and script).

"/" and "." and " " are ASCII characters.

> > Do you accept that we should use the OS region settings for date/time *format*?

> Not sure. That's why I started this discussion.

So, if that is really the question you're trying to answer, then you *are* deliberately misstating the question.

Of course everybody prefers to have the Firefox language used for words (!) in the date/time format. That is not the question we discuss. So, if you ask the question, of course everybody will agree. Even I will agree - it would be nice.

However, that does *not* mean that we can ignore OS date/time formats. That is the conclusion you draw yourself. That's the point where we all object, and where you broke our applications to the point where I can't use it anymore: with the format.

Most date/time formats are numbers and separators only, and language does not even come into play. And consistency of date formats (with OS) is more important than consistency of language (within the app), partially because most date and time formats are abbreviations without words. Most actual problems are with the short date formats, not the long ones.

By phrasing the question this way, you are trying to hide the actual problem that we all have.

Comment 24

[Zibi Braniecki [:zbraniecki][:gandalf]]

(In reply to Ben Bucksch (:BenB) from comment #23)
> > Separators are also in language (and script).
> 
> "/" and "." and " " are ASCII characters.

hy uses "թ" as a delimiter, zh-Hans uses "年", ky uses "ж", asturian uses "de" as a delimiter, korean uses '년', japanese '年', fo uses 'í'.
There are more.

> Of course everybody prefers to have the Firefox language used for words (!)
> in the date/time format. That is not the question we discuss. So, if you ask
> the question, of course everybody will agree. Even I will agree - it would
> be nice.

If we agree that date/time formats should be localized into product language, then we diverge from what Microsoft HiG wants us to do, and what Windows, Mail and Edge do.

We also open up a new flock of problems that I tried to hint here.

I would be surprised if Mozilla Intl/L10n module owners and peers wanted to violate HiG and do what you believe "everyone will agree" on.

> Most date/time formats are numbers and separators only, and language does
> not even come into play.

In Gecko we have at least 7 different numbering systems that depend on language: http://www.unicode.org/cldr/charts/31/by_type/core_data.numbering_systems.html

Numbers and separators are in language and script.

Comment 25

[Jorg K (CEST = GMT+2)]

Just a random thought: I just hope the final solution won't allow language/format mixes as declared acceptable in bug 1354339 comment #50, for example: "Thursday, 1.4.2017, 19:00" (or, worst case, "1. March 2017") (where "." is the German separator).

Comment 26

[Francesco Lodolo [:flod]]

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #20)
> > I'm not sure if it's covered by any of these, but would it be an option to let the user decide? Following your example, Firefox in English and custom regional settings in the OS, have a setting that lets the user decide what to use.
> 
> > Regional settings:
> > a) Follow browser language (en-US) - Default
> > b) Follow OS language (pl)
> > c) Use custom regional settings from OS
> 
> > Example: Friday, 5/15/2017 - 5.00 $
> 
> > The default being the current status (follow browser's language).
> 
> The only difference here is that for the third one, we still need to know
> which language we're in.

You're absolutely correct, we still need the language.

Let's say we have 3 languages:
* app.locale = language used for Firefox UI
* os.locale = language used for the OS
* regional.locale = language used for regional settings

As far as I can tell, macOS doesn't have the latter: you pick the language for the whole OS, then you can customize the default regional settings, but there's no explicit language defined there. You could work around it by assuming regional.locale = os.locale if regional.locale is not defined.

I'm not completely surprised by Mail on Windows ignoring custom regional settings besides the language, in the end it seems common to define a pattern in the app, not asking the OS "give me the format for short dates".

If we think that's acceptable (and I personally think it is), it could be simplified to.

Regional settings
(*) Use browser language (default)
( ) Use setting for: [language picker]

And let the user select the language they prefer, ignoring custom regional settings from the OS. It could work like the current implementation of Performances in prefs, where the the controls appear only after deselecting "Use suggested defaults".

If we really think it's important to allow custom regional settings:

Regional settings
(*) Use browser language (default)
( ) Use setting for: [language picker]
    [ ] Use custom regional settings defined in the OS

But still pick the language from user's selection.

It might be a naive approach, but looking at bug 1355307 comment 17: if the user is using Firefox (en-US) on Windows (en-US), and selects en-GB for "Regional settings", I would expect all API call to return en-GB. I also expect that to be an edge case, most users will want to stick to the default (same as app.locale).

Comment 27

[Jorg K (CEST = GMT+2)]

Hmm, I think the options are:
1. Use browser language - gives straight ICU formats
2. Use browser language augmented by OS customisation if language matches (current state)
3. Use setting for: [language picker] - gives straight ICU format for that language
4. Use language configured in regional settings in OS (format locale)
   (will use straight ICU format for the selected language, formats ignored
   (don't laugh, we had that state for a while for C++ dates)
5. Use custom regional settings defined in the OS
   (will use language *and* formats defined in OS)

You could fold 3., 4. and 5. into
3'. Use setting for: [language picker] - gives straight ICU format for that language
    [ ] Use custom regional *formats* defined in the OS if language matches

So maybe (solution 1):
1'. Use browser language (straight ICU)
    [ ] Use custom regional *formats* defined in the OS if language matches
3'. Use setting for: [language picker] - gives straight ICU format for that language
    [ ] Use custom regional *formats* defined in the OS if language matches

Or (solution 2):
1'. Use browser language (straight ICU)
3'. Use setting for: [language picker] - gives straight ICU format for that language
[ ] Use custom regional *formats* defined in the OS if language matches (applicable to both).

In 3' the language could be pre-populated with the OS (format) locale, so the locale used for formatting, not the OS system locale.

Or (solution 3):
1'. Use browser language (straight ICU)
    [ ] Use custom regional *formats* defined in the OS if language matches
5. Use custom regional settings defined in the OS
   (will use language *and* formats defined in OS)

In all options it would be nice to actually see the language, so - say - solution 3 will look line:
(*) Use browser language (English United States)
    [ ] Use custom regional formats (see note below).
( ) Use custom regional settings defined in the OS (German Germany)

Note: Can only be ticked iff some en-* is selected in OS, so not in Ben's case ;-)

Comment 28

[Liz Henry (:lizzard) (relman/hg->git project)]

Clearing the tracking request for 53.  While I'm not sure release management needs to track this for 55, I'll go ahead so that the owner for 55 is aware of the discussion here.

Comment 29

[Ben Bucksch (:BenB)]

Sorry As the one who marked it tracking ff53 ?, I retract my request for tracking. You may remove it, if you wish.

Comment 30

[Liz Henry (:lizzard) (relman/hg->git project)]

gandalf - have you all settled on an approach? Do we plan to change this for 56 (or 55)?
We have a lot of open needinfos here. We could try to meet and discuss at the all hands, or afterwards if you think we need wider input from ux or product.

Comment 31

[Link @ QPG]

I don't know how this translates into the selection options you are evaluating, but as a Linux user, I expect date formatting to respect the $LC_TIME environment variable.

HTH

Comment 32

[Makoto Kato [:m_kato]]

(In reply to Link @ QPG from comment #31)
> I don't know how this translates into the selection options you are
> evaluating, but as a Linux user, I expect date formatting to respect the
> $LC_TIME environment variable.
> 
> HTH

LC_TIME support is gone away by bug 1301640 and Chrome doesn't support LC_TIME as long as I know.  Also, we don't support LC_NUMERIC, LC_COLLATE now.

Comment 33

[Link @ QPG]

In the end all I care is that there is a very clear means for me to get ISO-8601 date formats and 24 hour time to appear everywhere in Thunderbird and Lightning, while still using English.

I find the inability to use the date formats that work best for me to be disruptive, particularly since it had been possible for years until recently being broken.

Comment 34

[takatomatsuki]

Why would Firefox not use the system datetime format where available? The user should be able to choose
* OS-defined datetime format [default where available]
* Firefox's installed format [default where the OS isn't defining one]
* Custom locale selection (user selects from a list of locales Firefox recognises]

In Windows for Firefox 54 and older it always used the OS-defined datetime format (in my case d/MM/yyyy and h:mm:ss tt) however my only option as of Firefox 55 is to use en-US, en-GB, or en-SA, none of which are suitable.

I agree with Link comment #31. The inability to use the date formats that work best for me is disruptive, particularly since I had been using it for years. Removing this ability removes yet another one of Firefox's selling points over Chrome.

Comment 36

[Zibi Braniecki [:zbraniecki][:gandalf]]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #30)
> gandalf - have you all settled on an approach? Do we plan to change this for
> 56 (or 55)?

I don't see any consensus building up yet.

> We have a lot of open needinfos here. We could try to meet and discuss at
> the all hands, or afterwards if you think we need wider input from ux or
> product.

I'd be happy to be part of it.


> I don't know how this translates into the selection options you are evaluating, but as a Linux user, I expect date formatting to respect the $LC_TIME environment variable.

I believe that bug 1337069 may lead us to a solution with that.
If you want to help writing a patch for that, I'd love to review it! :)

> Why would Firefox not use the system datetime format where available?

There's a wall of comments explaining the complexity with this above. Please, read it if you're curious.

> In Windows for Firefox 54 and older it always used the OS-defined datetime format (in my case d/MM/yyyy and h:mm:ss tt) however my only option as of Firefox 55 is to use en-US, en-GB, or en-SA, none of which are suitable.

That's not true. We do pick a date time format from OS if the browser and OS language match.

I'm all for discussing solutions here, but I expect every person suggesting a solution to consider the impact of their proposed solution on language/locale interactions beyond the particular setup they're using. It means understanding the impact on LTR/RTL combo, gregory/non-gregory calendars, arabic/non-arabic numerals etc.

Comment 37

[Eugene Savitsky]

Was it a problem for users before the change to FF/Tb? No.

So, why do we spend so much resources on the problem that did not existed before? 
Just revert the change to how it used to be and case closed.

Comment 38

[Zibi Braniecki [:zbraniecki][:gandalf]]

> Was it a problem for users before the change to FF/Tb? No.

It's called self-selection bias.

> So, why do we spend so much resources on the problem that did not existed before?

Please, read the comment 0. 

The problem/limitation existed before and affected our ability to use intl from l10n.

Once again, please stop commenting in this bug if you don't bother to dive into complexity of the problem we're trying to solve. It's not contributing to the conversation.

Comment 39

[Ben Bucksch (:BenB)]

> I don't see any consensus building up yet.

I see a lot of people very unhappy with the code change, and you unwilling to move your position that the change was good and correct. So, unless you move, that "consensus" is not going to happen.

> We do pick a date time format from OS if the browser and OS language match.

...which excludes all non-English countries. Talk about "bias"...

> I'm all for discussing solutions here, but I expect every person suggesting a
> solution to consider the impact of their proposed solution on language/locale
> interactions beyond the particular setup they're using. It means understanding
> the impact on LTR/RTL combo, gregory/non-gregory calendars, arabic/non-arabic
> numerals etc.

So, you're happily doing "selection bias" here, by discounting all users and developers who complained about the change and are affected by it, and declaring them all incompetent (your last comment above) and thereby irrelevant.

You simply won't acknowledge that you "fixed" one "problem" that existed since 20 years, and broke something else that many many people rely on. Yet, you haven't fixed what you broke, and demand that others propose a solution for the problem you created.

Here's a clear problem statement:
* I'm a German, with German OS, and my OS settings say date format is 12.04.2017 and time format is 21:02. That's the default for all Germans and what every German will expect.
* A German user seeing 04/12/2017 and 9:02 PM will be confused and  consider it wrong, ***even in an English application***.
* Almost all native applications honor and use that OS format, even if the app is English (with a few exceptions)
* Firefox and Thunderbird nightly builds since 2 decades honored that OS format
* Since your change, Firefox and Thunderbird nightly builds in English use the English format.
* For me, that's clearly broken, totally unusuable, and a regression that must be fixed immediately.
  I cannot use Thunderbird and Lightning builds until that is fixed.

You locked me out of latest Thunderbird builds, even though I am a Mozilla contributor since almost 20 years.

So, instead of insulting people who report bugs (last comment), or trying to justify your change, how about fixing the bugs you introduced?

Comment 40

[Ben Bucksch (:BenB)]

Reproduction:
* German OS, OS date/time settings as default, with date format 03.04.2017 and time format 21:02.
* Install Thunderbird nightly in English

Actual result:
Date/time shows as 03/04/2017 9:02 PM
or even (depending on situation):
Date/time shows as 04/03/2017 9:02 PM

Expected result:
Date/time shows as 03.04.2017 21:02

Problems:
* Unclear which position day and month are in, therefore unclear whether April 3 or March 4.
* Wrong separators
* Wrong 12h format

Importance:
* Unusuable

Comment 41

[Zibi Braniecki [:zbraniecki][:gandalf]]

:BenB - I don't see a value in repeating all the arguments of both sides yet again.

I respect your position and I disagree with it for the reasons laid in comment 0. I recommend you switch to a "de" version of the product to get German date/time formatting (or "en-GB" for non-american English).

I know that's not what you want, but I believe that that's what we will support. "en-US" will have US regional customizations, while "de" will have DE regional customizations.
We may be able to stretch it here and there, but I believe that ultimately, you'll be disappointed to find increasing number of "americanisms" leaking into the UI if you keep using the en-US build.

I wasn't "insulting" anyone in my previous comment. I was just responding to the question on "why we can't revert to the old behavior" which is explained in comment 0 to the best of my ability.

also, I'd like to point out that through this thread we got multiple interesting UX proposals that would improve the situation for users like yourself and solve problems listed in comment 0 (see comment 26 and comment 27).
My concern is that you're pushing the thread in a different direction which may discourage people from participating in the task of looking for a solution.

Comment 42

[Adrian Kalla [:adriank]]

@BenB: Having a German OS and installing an app using the English language is not a common use case for a normal user, but:

@gandalf: having a German OS and installing an app using the Lower Sorbian language is a valid use case in Lower Lusatia (or in general: we have Firefox in ab-CD but an OS that is not available in that locale at all). I don't know if in this particular case Lower Sorbian does use another date format than for de-DE, but if it would, then it would be important for Firefox to respect that - or as a minimum to have an option, to respect the format set by the user in the OS settings.

Comment 43

[Jorg K (CEST = GMT+2)]

Personally I think it would be beneficial to have an option as detailed in comment #27:

Either:
Solution 2 (from comment #27):
(*) Use browser language (English United States)
( ) Use setting for: [German]
[ ] Use custom regional *formats* defined in the OS if language matches.

Solution 3 (from comment #27):
(*) Use browser language (English United States)
    [ ] Use custom regional formats.
( ) Use custom regional settings defined in the OS (German Germany).

In solution 2 the first option would show straight en-US ICU, combined with the check-box it would show en-US but modified by any en-* OS settings (current state).
The second option would show straight de ICU, combined with the check-box it would show "de" but modified by any de-* OS settings, like for Swiss or Austrian.

In solution 3 the first option would show straight en-US ICU, combined with the check-box it would show en-US but modified by any en-* OS settings (current state).
The second option would show "de" ICU, but modified by the de-* OS settings.

Obviously solution 2 is more flexible since you have four options: app locale and OS language, both straight ICU and with OP modifications. In solution 3 you only have three options: app locale with/without OS modifications, and OS language *with* modifications, which was the state until FF 53 (or was it FF 54?).

The "Lower Sorbian" use case is covered by these options as is Ben's use case.

My suggestions is just a little more flexible than Francesco's from comment #26, so I think it would be very easy to get a consensus here. The default would position FF as a consistently localised product in all languages, including LTR/RTL, (non-)Arabic numerals, etc. The additional option would open the door for our niche users with special requirements. It's a win-win solution ;-)

Comment 44

[Frank-Rainer Grahl (:frg)]

My vote would go naturally to the solution which would be the most flexible. But if it turns out to be a lot of work I would be happy with any which at least restored the old state. Broken or not it caused a lot less complains than now.

This currently bites every mail and calendar user of both Thunderbird and SeaMonkey not running the simple l10n build which matches OS language setup. And these are a lot. I am not aware of any other program I use which handles it this way. It also affects Firefox users to a lesser degree. I do not know one user who is affected and finds the current solution acceptable.

There are many reasons not to run a language matched build. For most languages they do not even exist.

Would be great if this could be fixed before the next ESR59 cycle.

Comment 45

[Ben Bucksch (:BenB)]

> I recommend you switch to a "de" version of the product to get German date/time formatting
> I know that's not what you want, but I believe that that's what we will support.

There is no nightly for DE, and I can't compile from source with DE. That's why this is "dogfood" (see Bugzilla definition). By definition of bugzilla, this also classifies as Severity: Blocker, because you're impairing your fellow developers to use the product. Mind you, it's not just developers and testers, though.

There are many people around the world in non-English speaking countries that prefer English builds. You just killed them all.

> This currently bites every mail and calendar user of both Thunderbird and SeaMonkey
> not running the simple l10n build which matches OS language setup.
> And these are a lot.

+1

That's what I keep saying, but Zibi just ignores them.

Comment 46

[Francesco Lodolo [:flod]]

(In reply to Ben Bucksch (:BenB) from comment #45)
> There is no nightly for DE
https://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-central-l10n/

If they're not up to date, that's a bug in Thunderbird::Build.

Having said that, I think it would really help if we could try to focus the discussion on the problem at hand (figuring out the best way to deal with this scenario) instead of attacking each other or repeating things that have been said multiple times, here on in other bugs.

We're at comment 46, most of them are comments that help very little in finding and implementing a solution (including this one, sadly).

Comment 47

[Jorg K (CEST = GMT+2)]

<unrelated>
(In reply to Francesco Lodolo [:flod] from comment #46)
> https://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-central-l10n/
> If they're not up to date, that's a bug in Thunderbird::Build.
I try to keep the l10n Dailies going, looks like Linux and Windows failed today (30/6), but all platforms worked yesterday (29/6).
</unrelated>

Comment 48

[Ben Bucksch (:BenB)]

Thanks. I didn't know there are German nightlies. But I compile from source, and I need use the English UI to communicate with my other team members.

> We're at comment 46, most of them are comments that help very little in finding and implementing a solution

Zibi is attacking everybody who suggests a solution. So, I leave it to him to how to solve the problem.

I just want the OS date/time preferences honored (and that to be the default setting). If comment 40 is fixed, I should be happy. I don't care how he does it. I think Zibi said it can be done. How that works in the new code, is up to him, as he designed it.

Comment 49

[Francesco Lodolo [:flod]]

Fair warning to make the bug actionable and readable: I'm going to start marking comments as "advocacy" or "off-topic" unless they're adding useful information to the problem, they are still in the bug but hidden by default when loading the bug. 

The bug is clearly defined, as it's also clear the impact on a group of users. Repeating it doesn't help anyone.

Please do not take this personal, and let's try to keep it polite and professional.

Comment 50

[Zibi Braniecki [:zbraniecki][:gandalf]]

:Jorg - I really like your proposal! Here are couple notes about it:

 - we cannot reliably distinguish between "manual customizations" and "OS defaults". We can only differentiate between taking data from OS Regional Settings or not.
 - I'd like to make sure that the language reflects the "best-effort" nature of the operation. We won't usually have all regional settings reflected in the UI. We can do some date/time patterns (not all still!), we will be able to pick up some more, like, maybe measuring system, but for example, I don't see us anytime soon picking up number formatting patterns despite the fact that MacOS and Windows expose them.
 - we don't have an easy access to localized names of locales ("American English"), so we'll likely have to show locale code to the user ("en-US", "en-CA") - at least initially.

So, maybe this would be 

```
Regional settings
(*) Use browser locale [en-US]
( ) Use OS locale      [de]

[ ] Use custom regional settings defined in the OS
```

and the last checkbox would only be enabled if the selected locale's language matches the OS locale's language.

This would preserve the consistency between language/locale while also allowing the user to select OS locale for the Intl operations.

Technically, all proposed solutions are easy to implement. The only challenge I see is that generally all APIs expect a single locale to be used. There's "Firefox UI locale", "Addon locale", "JS environment locale".
With this we would basically introduce two locales - one for localization, and another for internationalization.
For example, JS env. has a single "defaultLocale". This locale is used for DateTimeFormat (regional setting), and PluralRules (localization). In this world, would it be "en-US" or "de"? 
Similarly, if a WebExtension asks for a locale it should localize itself to, should we pass "en-US" or "de"?

I'm ok going with - "en-US" is the app locale and wherever possible we slowly introduce a complementary variable called "regionalSettingsLocale" - we'll first format date/time to it, and then maybe get it used in more places - but overall, it seems like a change that may be hairy compared to consistently collapsing everything onto a single "appLocale".

Comment 51

[Jorg K (CEST = GMT+2)]

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #50)
> We won't usually have all regional settings reflected in
> the UI. We can do some date/time patterns (not all still!), we will be able
> to pick up some more, 
Indeed. For example the date picker shows "06/16" even if the other full dates follow "en-GB" in an "en-US" app. We sort of discussed that pattern issue in bug 1356403 comment #7 (and further down).

> ```
> Regional settings
> (*) Use browser locale [en-US]
> ( ) Use OS locale      [de]
> 
> [ ] Use custom regional settings defined in the OS
> ```
> and the last checkbox would only be enabled if the selected locale's
> language matches the OS locale's language.
Nice!!

> With this we would basically introduce two locales - one for localization,
> and another for internationalization.
> For example, JS env. has a single "defaultLocale". This locale is used for
> DateTimeFormat (regional setting), and PluralRules (localization). In this
> world, would it be "en-US" or "de"? 
> Similarly, if a WebExtension asks for a locale it should localize itself to,
> should we pass "en-US" or "de"?
Yes, you pointed out this issue various times and I don't have a good answer. If you return "en-US", the extension might display dates/times in "en-US" not matching the rest of FF. If you return "de" the extension might display German strings in an otherwise English FF. This appears to be a lose-lose situation.

Comment 52

[Zibi Braniecki [:zbraniecki][:gandalf]]

> Indeed. For example the date picker shows "06/16" even if the other full dates follow "en-GB" in an "en-US" app. We sort of discussed that pattern issue in bug 1356403 comment #7 (and further down).

Yeah, we don't have a good way to solve it today, but with the proposal from comment 50, if the user selected "OS locale" we would eventually start using the "en-GB" dates in the date picker so even if we couldn't pick Regional Preferences for a given date pattern (06/16).

>  If you return "en-US", the extension might display dates/times in "en-US" not matching the rest of FF. If you return "de" the extension might display German strings in an otherwise English FF. This appears to be a lose-lose situation.

My current thinking is that if we do this in a forward-compatible way, we can just slowly introduce the new variable to more and more places. If WebExtension (or sth else) goes for the appLocale ('en-US') instead of regionalPrefsLocale ('en-GB'), then the worst case is what we have today already! And over time, we can introduce this duality there as well.
Same for other places in the code.

Comment 53

[Dirk Munk]

The fun of all this is that it has hardly anything to do with language.

I’m running an English version of Windows 10 with the English version of SeaMonkey. Up to recently everything was fine, but then with a new version I got American date and time stamps (AM / PM etc.).

I live in the Netherlands, so I could use the Dutch region settings for Windows. However, that would give me Dutch words for the months and days (where applicable), and that looks a bit silly in an otherwise English set up.

I can also use the British settings, but then things like the currency symbol and the decimal separator are ‘wrong’. But guess what, you can change that too! 

So now I use the British settings, but I changed the decimal symbol from dot to comma, and the group separator from comma to dot, and the currency symbol from £ to €  (I’m sure this will give Brexit supporters a heart attack). 

What this proofs is that there are no ‘real’ regions settings, except for the language used for words for the days and months. All other settings can be changed to other formats than the default for that language. The regions settings are nothing more than a set of pre-defined settings that can be changed to your liking.

So the only thing that has to happen is that SeaMonkey etc. can read the region settings that are actually in use by the operating system. I’m sure there must be a way to do that, perhaps by writing C++ modules for each operating system (Windows, OsX, Linux etc.).

Comment 54

[Zibi Braniecki [:zbraniecki][:gandalf]]

It seems that UTS35 offers a possible way out as well!

There's an "rg" unicode extension key [0] which allows for regional settings override.
If I read it correctly, that means that we could define Gecko appLocale to be, for example, "en-US-u-rg-dezzzz" which means "American English with regional preferences for de".

This would save us from defining two variables and juggling between them, and would only require us to start recognizing this extension in places where it matters - particularly in OSPreferences.

What I like about this solution also is that this is easy to add to ECMA402 as an extension to DateTimeFormat, UnitFormat, NumberFormat etc. while it wouldn't affect PluralRules since `rg` extension key doesn't make sense in context of that API.

The challenge here is that since it's a regional subtag, not whole locale tag, we'd need to extract it out of the OS to offer in the UI for users.

I'll play with extracting the regional subtag out of WinAPI/Cocoa.
That, unfortunately, wouldn't work well with POSIX locale tag (like what Linux/Unix do with LANG etc.), but I believe that it's ok, since we're already drifting away from POSIX locales and toward UTS35 with our focus on CLDR/ICU/ECMA402.

[0] http://www.unicode.org/reports/tr35/#Key_And_Type_Definitions_

Comment 55

[Zibi Braniecki [:zbraniecki][:gandalf]]

I've evaluated a number of possibilities and unfortunately it doesn't seem like BCP46/UTS35 or ICU will be helpful here as they don't seem to follow the `rg` extension tag.

I want to get this into 56 so I filed bug 1379420 to introduce LocaleService::GetRegionalPrefsLocales. Once it lands, I'll switch mozIntl and intl/locale/DateTimeFormat.cpp to use it.

I'm removing NI's from people since it doesn't seem like we're getting a consensus here on anything more sophisticated and I believe that an API like proposed in bug 1379420 will at least solve the problem reported by users in this bug.
We can work on incremental improvements to the system later.

Comment 56

[onesolo+github]

I'm really mad at you guys from Thunderbird...

I'm Portuguese but I like to have my OS (Win10), Firefox and Thunderbird in English. FF and TB are in en-US. Win 10 is setup with my country's local time settings, 24h and dd/MM/YYYY. Guess what, on TB I'm having to deal with the en-US, 12h and MM/dd/YYYY and I have no possible way to change this!!! uuggghhh!!!

How hard is to let the user to have a language in TB but time and date can be modified to whatever the time and date that user desires?! Like in Windows for example!!!
Is that so hard to do it?!? Why do you guys want to impose this?!

Comment 57

[Jorg K (CEST = GMT+2)]

(In reply to onesolo+github from comment #56)
> I'm really mad at you guys from Thunderbird...
Note that this is a Mozilla core decision, not a Thunderbird decision. Besides, the situation is currently being improved, watch bug 1379905 and bug 1379910.

Comment 58

[Zibi Braniecki [:zbraniecki][:gandalf]]

Hi all,

As of yesterday, nightly builds have a new preference that can be switched in about:config "intl.regional_prefs.use_os_locales". Switching it to "true" should make your product follow Regional Preferences Locale from your OS which I hope solves the "Windows in de, Thunderbird in en-US, but user wants dates in de" scenario.

I'd like to ask anyone affected to test it and verify that it works as expected.

Thank you for your patience and apologies that it took so long

Comment 59

[Jorg K (CEST = GMT+2)]

Thunderbird users please watch bug 1384007: The preference will be visible in the UI.

Works as expected, I tested it ... and it's really ugly, since for example in Thunderbird/Calendar all the month/day names are still in English even if you switch to German, but then, there's no accounting for taste ;-)

Comment 60

[Zibi Braniecki [:zbraniecki][:gandalf]]

So, Tb landed the UI for the pref, Firefox pref bug is bug 1379910 and I marked it as good-first-bug if someone wants to grab it.

I do not expect to work on this issue much more. My understanding is that we now have the behavior we intend for Firefox as default in Firefox and flexibility for advanced users to select the old behavior which I verified on Mac and Windows to pick date/time from OS and localize them into the OS Regional Settings languages.

I'm going to keep this bug open for now, to help people find it, but I expect that we may want to close it and once we develop a need for more finetuning around this, open a new bug.

Comment 61

[Winson Wen]

Hi, not sure if this is the right place to report this, but I don't think the mechanism of using the OS locale when the language is the same as Firefox is actually working.
The description in this bug says that using en-US Firefox on en-AU Windows would make Firefox use en-AU date layout, but on my computer it's using en-US date layout instead. I also don't appear to be the only one, with someone at bug 1393673 hitting the same issue.

Comment 62

[Zibi Braniecki [:zbraniecki][:gandalf]]

hi winson.

Can you file a new bug please and share which version of Firefox are you using? The bug you linked has been marked as a duplicate of a bug that has been already fixed. It was also reported against Firefox 54, while the correct behavior cross-locale should be in 57 and 58.

Please, test if the behavior is still there in the current Nightly (58) and if it is, please, file a new bug and CC me. I am not aware of any bugs in the implementation that would prevent en-US Firefox and en-AU Regional Prefs on Windows from working correctly.

Comment 63

[Winson Wen]

Filed bug 1411851. Was already running Firefox Nightly 58.

Comment 64

[Zibi Braniecki [:zbraniecki][:gandalf]]

As per comment 60, I'm going to close this.