914558 - [B2G][Contacts]: Vertical sorting order in Contacts App should not display Latin letters for non-Latin languages

Status: RESOLVED INCOMPLETE

(Firefox OS Graveyard :: Gaia::Contacts, defect)

Description

[Peiying Mo [:CocoMo]]

The Contact list doesn't support Cyrillic alphabet, which comes in front of the Latin letter based contact list.  

Right now all Cyrillic contacts are listed under #.

Comment 1

[Axel Hecht [:Pike]]

This is nothing we can fix in the Serbian localization, over to b2g.

Comment 2

[Delphine Lebédel]

Peiying: Are you using v1.1? Do you have the version of Gecko and Gaia as well please? 

When I switch my Leo phone (on today's v1.1 Moz RIL) to Serbian Cyrillic and go to contacts, my keyboard is directly in Serbian Cyrillic, and I can save contacts in Serbian Cyrillic.

However, I notice when I enter Contacts that the alphabet letters on the vertical line on the right side (that is used to switch from one letter to another quickly within the contacts) still shows up in Latin alphabet. Is this what you are describing?

Comment 3

[Delphine Lebédel]

I realize my last explanation might lead to confusion, so reformulating: 
However, I notice when I enter the "Contacts" App, that the alphabet letters on the vertical line on the right side (that is used to switch from one letter to another quickly, in order to find contacts) show up in Latin alphabet. Is this what you are describing?

Comment 4

[Axel Hecht [:Pike]]

Yes, and even more so, that if you have a 100 contacts in Cyrl, the all show up in the '#' category.

Comment 5

[Delphine Lebédel]

Good catch Peiying

BTW even with a single contact it seems to show up under the "#" category apparently...

What's more, this seems to reproduce for all locales, not only Serbian Cyrillic. Seems to me these are 2 separate bugs though:

1. Vertical sorting order under Contacts should not display Latin letters for non-Latin languages
and
2. Entries in contacts all appear under the "#" category (but shouldn't)

I will change the title of this bug to reflect number 1 issue and file a new bug for number 2.

Issues confirmed on latest Leo v1.1 Moz RIL build
Gaia   946623575b3fbf6138fff5f77ddead3cadffacce
BuildID 20130910041240
Version 18.1

Comment 6

[Delphine Lebédel]

Nominating for leo as this reproduces in all non-Latin alphabet shipping locales

Comment 7

[Peiying Mo [:CocoMo]]

Delphine, on comment 5, item 2, we should reword what correct behavior should be.  For instance a Greek person may choose Greek UI but chances are s/he may have contacts from FIGS speaking countries, so the Latin based characters should be made available as well.  We just need to get input from the community which goes on top of one another.

Comment 8

[Jason Smith [:jsmith]]

Triage - too late to take this at this point, even though this is an important to fix. Not blocking.

Comment 9

[Delphine Lebédel]

Ok nominating this for koi as it is an important fix

Comment 10

[Fabien Cazenave [:kaze]]

FTR, this bug blocks a leo+ one (bug 914942).

Comment 11

[Joe Cheng [:jcheng] (please needinfo)]

triage: leo+ as it blocks a leo+

ni? Jose

Comment 12

[Jose Manuel Cantera]

Yes, 

this is happening. I believe we need to deal with it but the issue is the vertical space in cyrillic languages as the number of characters is longer than in roman languages. I believe the right person to deal with this bug is :kaze ... but I guess there are also UX implications

Comment 13

[Peiying Mo [:CocoMo]]

What if the list contains both Cyrillic and Latin-letter based contacts, the list will definitely extends beyond the capacity of the vertical height.  I don't think it is that uncommon these days that one could have contacts in different scripts.  

Do we list all the letters or only the letters with entries?  Can we skip letters at certain intervals so both sets of alphabet can fit?  

For instance I have contacts in both Chinese and English (or Latin based). When I use Traditional Chinese setting, the Chinese characters would sort by number of strokes.  Some can get very complex, and I don't think we want to list all the strokes up to 50 or whatever the most complex one contains.

Comment 14

[Peiying Mo [:CocoMo]]

I think we should change the bug summary - it is very likely we have both Latin-based and non-latin based scripts needed on the vertical list.

Comment 15

[Fabien Cazenave [:kaze]]

Attachment: screenshot — contact list on Android 4.0

I’m afraid there’s no way we can go this far for a leo+ bug.

On my Android 4.0 phone, Greek and Cyrillic names are sorted after Roman names, and the vertical scrollbar is always “A-Z” — even when my system is configured in Russian, see attached screenshot.

I’d say that given the time frame, being on par with Android 4.0 should be considered as an acceptable solution for FxOS 1.2, and it should be covered by bug 914942. In other words, I recommend not to consider this bug a dependency of bug 914942 and remove the leo+ flag.

Comment 16

[Fabien Cazenave [:kaze]]

Now if we want to think long-term and really address the reported issue…

The way all names are sorted in Gaia is flawed, even for Roman scripts. It relies on a “Normalizer” script [1] that gets rid of diacritics in Roman letters, thus allowing to sort “Ädela” under “A” and “Étienne” under “E”. This is fine in most cases, but in Swedish “Ädela” should be sorted under “Ä”, which is collated after “Z”. And some digraphs can be considered as one letter in some languages (e.g. “ll” is one letter in Spanish, two in other languages).

Again: I think this is acceptable for 1.2. But for later versions I think we should consider making the “A-Z” index locale-specific by:
 • dropping the “Normalizer” script;
 • defining these “index letters” in a *.properties file;
 • drawing the “A-Z” scrollbar accordingly (= possibly with more letters);
 • relying on localeCompare [2] to sort names under the proper index letter.

Handling non-Roman alphabets (e.g. Greek, Cyrillic, Arabic, Hebrew, Thai…) would be a similar story, though there are a few additional UX concerns:
 • (as Jose mentioned) how do we fit all symbols of the Cyrillic or the Thai alphabets in 480px?
 • what alphabet should be displayed in the scrollbar when several scripts are used? Should that be the preferred script of the current locale, or the script that matches the biggest number of contacts? Or should there be a way to select the scrollbar in the Contacts app?

Handling non-alphabetical scripts is way more difficult, and we’ll probably need locale-specific code for these languages if we want to display something else than an “A-Z” scrollbar (see bug 914589 for instance, which is related to Chinese… which is one of the targeted locales for 1.2, btw *cough*). This could be a separate bug.

[1] https://github.com/mozilla-b2g/gaia/blob/master/shared/js/text_normalizer.js
[2] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/localeCompare

Comment 17

[Fabien Cazenave [:kaze]]

Pike, Gandalf, feel free to jump in and please correct me if I missed something.

Comment 18

[Fabien Cazenave [:kaze]]

Josh, can you spread some UX light regarding the index scrollbar (see comment 16)?
 • how should we fit long alphabets on 480px? Reduce font-size / skip some letters / other?
 • when several scripts are used (e.g. Roman + Cyrillic, which might be quite common for Serbian), which alphabet should we use?
    - the default script of the current locale?
    - the script that matches the biggest number of contacts?
    - or should there be a way to select the scrollbar alphabet?
    - or should we concatenate all used alphabets, as suggested in comment 13?

I think the latter would be difficult to use along with specific index scrollbars for non-alphabetic languages (e.g. Chinese, see bug 914589), but I’m no UX engineer and I don’t speak Chinese.

Comment 19

[Fabien Cazenave [:kaze]]

(In reply to Fabien Cazenave [:kaze] from comment #16)
>  • relying on localeCompare [2] to sort names under the proper index letter.

Forgot to mention: this doesn’t work yet in Firefox, because the `locales' argument is not supported yet [1]:
 • 'ä'.localeCompare('z', 'en') should be negative
 • 'ä'.localeCompare('z', 'sv') should be positive, but is negative with Firefox 27.

This has not been fixed by bug 769871. Adding a dependency to bug 866301.

Gregor, I think you told me this was a work in progress. Do we have an ETA for this?

[1] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/localeCompare#Browser_Compatibility

Comment 20

[Ben Kelly [:bkelly, not reviewing]]

If possible, it would be nice to have any scrollbar customization occur as part of the l10n fixup prior to the first paint.  Previously we were dynamically creating the A-Z bar, but it was causing a visual flash and slowing time-to-first-results due to the DOM being modified late in the launch process.  See bug 919692.

Just mentioning as additional information to factor in to the design.

Thanks!

Comment 21

[Axel Hecht [:Pike]]

I think it'd be nice to find the buckets according to a collation, and then show only the first glyph of the bucket. That's much of an intl problem, though, and I haven't come across any helping libraries.

Comment 22

[Kevin Grandon :kgrandon]

I am fairly certain that we should define the jump points for each locale in the *.properties file. Each jump point is essentially a "range", and currently we have one english letter per jump point. The displayed ranges should update when you change the locale of your phone. Even for chinese, it should be possible to show all relevant ranges within one screen. We can also follow specific language ranges with a selection of latin characters, for example Japanese has thousands of Kanji, but it's locale file could contain the following jump links:

あ か さ た な は ま や ら わ A D G J M P T Z #

Comment 23

[Fabien Cazenave [:kaze]]

We could also use ranges with two letters: A-B, C-D, E-F, etc. However I think it should be adjusted dynamically (think of HD devices) rather than being hard-coded in *.properties files. In all cases, yes, the index depends on the locale (obviously): ABCDEFGHIJKLMNOPQRSTUVWXYZ for English, ABCDEFGHIJKLMNOPQRSTUVWXYZÆØÅ for Norwegian, etc.

My question was, what do we do when two or three or more scripts are used in the Contacts list?
 • in your example (Japanese) you suggest to concatenate several indexes, but when should we display Latin jumps? When there are more than xx% of Latin names? Or are you suggesting that all locales need the Latin A-Z index?
 • if we don’t want to concatenate different indexes, how do we choose which index should be shown?

And the main question is: do we really want to support different scrollbars for 1.1? (= is that really a leo+ bug?)
As José mentioned, this is a significant work, which implies a significant risk.

Comment 24

[Kevin Grandon :kgrandon]

My suggestion is to always show the same ranges initially. I think it's just way too much work to show customized ranges based on dataset and HD device - but this is definitely something we can do in the future. 

A good first step would be to come up with one set of ranges per locale which would match a working localeCompare() sort - assuming localeCompare() would the the order that we want to display entries. It seems like we have ranges that would work for English, Japanese, and Norwegian :)

Note: You mentioned indexes here, I'm assuming you were referring to indexedDB? Ideally we'll have locale based sorting in the near future, so it *should* just be one index. Bugs you may be interested in: 909935, 871846

> And the main question is: do we really want to support different scrollbars
> for 1.1? (= is that really a leo+ bug?)

I think this would require a huge amount of effort and at this point could only be considered for 1.3 or later.

Comment 25

[Jose Manuel Cantera]

Dev_wise my suggestion for v1.1 would be to hide the vertical alpha scroll for non latin languages. 

For 1.2 we can pursue a more sophisticated solution but honestly I would prefer to leave that for 1.3 and code something robust and for the longer term.

best

Comment 26

[Håvar I. Henriksen [:havarh]]

What Apple does in iOS (and general sorting in OS X) if the user has set English as language, Æ and Å gets sorted under A, and Ø gets sorted under O, while あ gets sorted under #.
If the user has set Norwegian as language ÆØÅ gets sorted under their own letters after the Z and before the #, Aa also gets sorted under Å.
So I guess it would be easy to fix the first case, sorting ÆØÅÄÖ, etc, under their similar characters.

Comment 27

[Fabien Cazenave [:kaze]]

Removing the dependency with bug 914942, as it appears that supporting Cyrillic names without displaying a Cyrillic scrollbar is common solution on other mobile OSes.

Comment 28

[David Scravaglieri [:scravag]]

To many change are involved for leo timeframe, postponed to 1.3

Comment 29

[Fabien Cazenave [:kaze]]

Thanks for your feedback Håvar, I think that’s clear for me now.

(In reply to Jose M. Cantera from comment #25)
> Dev_wise my suggestion for v1.1 would be to hide the vertical alpha scroll
> for non latin languages. 
> For 1.2 we can pursue a more sophisticated solution but honestly I would
> prefer to leave that for 1.3 and code something robust and for the longer
> term.

That would make sense to me. I’m told that Android 4.2 doesn’t display any A-Z scrollbar any more (at least in Russian).
As this bug just lost its leo+ flag, I’m adding some code to hide the scrollbar for non-Latin languages in my patch for bug  914942 so that we’re clean for 1.1 and 1.2.

(In reply to Kevin Grandon :kgrandon from comment #24)
> A good first step would be to come up with one set of ranges per locale
> which would match a working localeCompare() sort - assuming localeCompare()
> would the the order that we want to display entries. It seems like we have
> ranges that would work for English, Japanese, and Norwegian :)

At the moment, our .localeCompare() does not take the current locale into account, which means there’s no way to sort Ädele after Z for Swedish, for instance. We need some platform support for that.

I have already written a PoC library to do locale-sensitive alphabetic sorting. It works on Chrome (except for sr-Latn because Chrome only supports Cyrillic on Serbian) but not on Firefox.

> Note: You mentioned indexes here, I'm assuming you were referring to
> indexedDB? Ideally we'll have locale based sorting in the near future, so it
> *should* just be one index. Bugs you may be interested in: 909935, 871846

Errr, no. By “index” I mean a subset of the locale’s alphabet that is used for sorting names under letters, e.g. 
“ABCDEFGHIJKLMNOPQRSTUVWXYZ” for English or French, “ABCDEFGHIJKLMNOPQRSTUVWXYZÅÄÖ” for Swedish, “АБВГДЕЁЖЗИЙКЛМНОПРСТУФХЦЧШЩЭЮЯ” for Russian, “АБВГДЂЕЖЗИЈКЛЉМНЊОПРСТЋУФХЦЧЏШ” for Serbian, etc. Note that for Russian, three letters are not used in the “index” (Ъ, Ы, Ь) because they never start any word. That would be the same for the German “ß”.

Sorry if “index” is not the proper word in English. :-/

We need locale-based sorting (= `locales' argument in localeCompare), and this should lead to locale-specific “indexes” in Gaia.

Comment 30

[ayman maat :maat]

the path Håvar and Kaze are proposing a perfectly actionable solution from a UX perspective. ni? me again if you require further detailed input

Comment 31

[Joe Cheng [:jcheng] (please needinfo)]

triage: add to backlog bug 891754

Comment 32

[Teiichiro Fukuda]

Attachment: screenshot - japanese contact list on Android 4.0.png

Hi,

Because the index letter to be displayed by the locale is different, I agree to the proposal to adjust in *.properties files.

Thanks.

Comment 34

[Fabien Cazenave [:kaze]]

I’ve just seen that our current Gecko nightlies support the `locales' argument in String.localeCompare (see bug 769871 comment 29), and it works fine with my lib. If it’s still enabled in Gecko 28 we should be able to address this issue properly for the 1.3 branch.

Comment 35

[Axel Hecht [:Pike]]

Note that that's only on for desktop gecko, not firefox os, see bug 866301 (which this one already depends on, so we don't need to add that)

Comment 36

[Johan Lorenzo [:jlorenzo]]

Shall we move this feature request to the backlog, Wilfred?

Comment 37

[Wilfred Mathanaraj [:WDM]]

yes ( ID: CONTACT-20)

Comment 38

[Peiying Mo [:CocoMo]]

This is no longer relevant.