Closed Bug 800880 Opened 12 years ago Closed 9 years ago

Allow a second "fallback locale" different from English

Categories

(support.mozilla.org :: Localization, task)

Product:
support.mozilla.org
Component:
Localization
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: scoobidiver, Assigned: safwan)

References

Details

(Whiteboard: u=user c=wiki p=0 s=2015.12)

Attachments

(1 file)

Proposed Patch
44 bytes, text/x-github-pull-request
Details | Review 
Currently, a new locale shows English articles because en-US is the single fallback locale.
There are locales where people speak two languages (regional and country's one) like Occitan, Britain, Catalan, and Aragonese and similar locales like Brazilian and Portuguese. For those, a second fallback locale different from English would be really helpful.

The following rules would apply:
* If there is not a localized article, show the article translated in the fallback locale if it exists or the English one otherwise.
* If there is a localized article:
** if it's up-to-date, show it
** if it's not up-to-date:
*** if the article translated in the fallback locale exists, show the most recent one, i.e. the one that is based on the most recent English revision.
*** otherwise, show it

Note: It's different from bug 608443 that is about selecting the bridge language (the one in the source editor) for localizers.
This is a pretty big change. More like a project. It will need to go through the process with SUMO team to see if and when we will do it.
Whiteboard: u=user c=wiki p= s=
I have seen this come up a number of times, so here goes for future reference:

Benefit: Support for people in their own language, which leads to better article helpfulness ratings.

Downside: major development effort needed. Maintanance of localized articles is unclear (see questions below)

Open issues, and questions:
Assuming French is the fallback language and Occitan is the actual language:

* How many people would benefit from this feature?
* What if an article is not localized in Occitan AND French? Do we fall back to English?
* How do we make localizers of Occitan aware of those missing locales?
* How is the L10n dashboard going to be affected by this?
* What happens when an Occitan article goes out of date? Do we display the French one?
* Usually we show a box on top of not localized articles "help us translate this article", to recruit new contributors. How would we do that when the fallback language covers the translation? Would we still show the box?
(In reply to Kadir Topal [:atopal] from comment #2)
> Benefit: Support for people in their own language, which leads to better article
> helpfulness ratings.
Other indirect benefit:
* Better involvement of the community because Mozilla cares about their locale.
Benefits of the system itself:
* the entry ticket is null, i.e. new localizers can translate articles at their own pace without having to translate top 20 or 50 articles at once. Then seeing some articles localized can encourage new localizers.
* it's robust to missing localizers because, once an article is too old, it's automatically replaced by the article in the fallback locale if it's maintained.


> * How many people would benefit from this feature?
How many Firefox users of locales with a maybe fallback locale (Romansh, Fulah, Occitan, Breton, Bosnian, Aragonese, Catalan, Galician, Asturian, Friulan, Ligurian) or similar locales (pt-BR & pt-PT, zh_CN & zh_TW) are there? Enough for me.
Note that pt-BR can be the fallback of pt-PT and vice versa, the same for Chinese.

> * What if an article is not localized in Occitan AND French? Do we fall back
> to English?
Yes. See rule #1 in comment 0: Locale -> Primary fallback locale -> Secondary fallback locale (English)

> * How do we make localizers of Occitan aware of those missing locales?
I don't understand your question. Which missing locales?

> * How is the L10n dashboard going to be affected by this?
The L10n dashboard behaves as usual without taking into account the fallback locale, which is only for users.

> * What happens when an Occitan article goes out of date? Do we display the
> French one?
We display the most up-to-date of both, probably the French one. See my rules in comment 0.

> * Usually we show a box on top of not localized articles "help us translate
> this article", to recruit new contributors. How would we do that when the
> fallback language covers the translation? Would we still show the box?
You show this message for articles in fallback locales (French and English in that case).
No longer blocks: 800420
(In reply to Scoobidiver from comment #3)
> (In reply to Kadir Topal [:atopal] from comment #2)
> > * How many people would benefit from this feature?
> How many Firefox users of locales with a maybe fallback locale (Romansh,
> Fulah, Occitan, Breton, Bosnian, Aragonese, Catalan, Galician, Asturian,
> Friulan, Ligurian) or similar locales (pt-BR & pt-PT, zh_CN & zh_TW) are
> there? Enough for me.
> Note that pt-BR can be the fallback of pt-PT and vice versa, the same for
> Chinese.

If anyone is interested in moving this forward, providing a full list would be the best way. That way we can calculate how many more people we might be able to reach and justify spending resources on this feature.

 
> > * What if an article is not localized in Occitan AND French? Do we fall back
> > to English?
> Yes. See rule #1 in comment 0: Locale -> Primary fallback locale ->
> Secondary fallback locale (English)
> 
> > * How do we make localizers of Occitan aware of those missing locales?
> I don't understand your question. Which missing locales?

Sorry, I meant missing articles in their locale.


> > * How is the L10n dashboard going to be affected by this?
> The L10n dashboard behaves as usual without taking into account the fallback
> locale, which is only for users.

Okay, that is one option, but how do those localizers decide on what to work on next? Example. I'm an Occitan localizers and see on my dashboard that the 5 articles are missing. However I can't see which of those already have a French fallback and which one are not even localized into French. I assume that people would rather create translations for pages that don't have any translation at all.
I was just about to file a new bug on this, but this bug equals my request. For fy-NL I would like to first fall back to Dutch (nl) and after that to English. Maybe the simplest way to implement would be respecting the usersettings; my first language in Options > Language is Frisian and the second is Dutch, but for untranslated articles I get the English version.
I see your point of the box to encourage people to translate the article, but since the language-code is still in the address-bar you can still show the box based on that address.
For occitan language, locales are:

oc-ocpn Oc-Cisalpin	 	Cisaupenc
oc-gsc 	Gascon 			Gascon
oc-brne Biarnés 		Biarnés
oc-bigd Bigordan 		Bigordan
oc-cmge Comengés 		Comengés
oc-csrn Couseranés 		Coseranés
oc-aran Aranés 			Aranés
oc-prol Proensal-L 		Occitan ancian literari
oc-lnc 	Languedocian 		Lengadocian
oc-prv 	Provençal 		Provençal
oc-nart Nissart 		Niçard
oc-gvot Gavouot 		Gavòt
oc-vdph Vivaro-Dauphinois 	Vivarodaufinenc
oc-avgn Auvergnat-N 		Auvernhat
oc-lms 	Limousin 		Lemosin
oc-mrho Marchois 		Marchés
oc-auv 	Auvergnat		Auvernhat

http://fr.ekopedia.org/Projet:Babel/Codes_ISO_639-6
Best regards

Cédric Valmary
(In reply to Kadir Topal [:atopal] from comment #4)
> (In reply to Scoobidiver from comment #3)
> > (In reply to Kadir Topal [:atopal] from comment #2)
> > > * How many people would benefit from this feature?
> > How many Firefox users of locales with a maybe fallback locale (Romansh,
> > Fulah, Occitan, Breton, Bosnian, Aragonese, Catalan, Galician, Asturian,
> > Friulan, Ligurian) or similar locales (pt-BR & pt-PT, zh_CN & zh_TW) are
> > there? Enough for me.
> > Note that pt-BR can be the fallback of pt-PT and vice versa, the same for
> > Chinese.
> If anyone is interested in moving this forward, providing a full list would
> be the best way. That way we can calculate how many more people we might be
> able to reach and justify spending resources on this feature.
I don't have access to the ratio of each locale but I think dialects are negligible.
Nevertheless, the pt-PT locale (around 0.5% of installed Firefox - 1% of articles translated) and the zh-TW locale (traditional Chinese - unknown percentage of installed Firefox - 9% of articles translated) would use the pt-BR locale (around 6% of installed Firefox - 35% of articles translated) and the zh-CN locale (simplified Chinese - around 3% of installed Firefox - 20% of articles translated).
Assuming the zh-TW ratio is 0.3%, this bug would benefit at least 0.8% of users and maybe more if articles localized are not the same rising artificially the percentage of pseudo-localized articles in pt-BR and zh-CN.

> > > * How do we make localizers of Occitan aware of those missing locales?
> > I don't understand your question. Which missing locales?
> Sorry, I meant missing articles in their locale.
First with the localization dashboard that works as if there was not a second fallback locale and also with the usual warning at the top of the article localized in the fallback locale.

> > > * How is the L10n dashboard going to be affected by this?
> > The L10n dashboard behaves as usual without taking into account the fallback
> > locale, which is only for users.
> Okay, that is one option, but how do those localizers decide on what to work
> on next? Example. I'm an Occitan localizers and see on my dashboard that the
> 5 articles are missing. However I can't see which of those already have a
> French fallback and which one are not even localized into French. I assume
> that people would rather create translations for pages that don't have any
> translation at all.
Clicking an untranslated article in Occitan from the Occitan dashboard would show either a French article or an English article. If it's the English article, it means it's not translated into French (see rule #1).
Marking as "fixed", since we have custom fallback languages enabled on SUMO.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Vesper, Thoughts about this bug?
Actually this should work like as below:
Situation:
# An English KB Article translated to pt-BR
# But the Article is not localized in pt-PT
# A user go to the KB Article page of pt-PT

Current Result:
# It shows the User English Version of the KB

Expected Result:
# We can make something like that it will show the user pt-BR version of that Article. and say the user that its not available in pt-PT but available in pt-BR.

## We can make choose which will be fallback local for a local
Flags: needinfo?(mdziewonski)
(In reply to Safwan Rahman from comment #9)
> Vesper, Thoughts about this bug?
> Actually this should work like as below:
> Situation:
> # An English KB Article translated to pt-BR
> # But the Article is not localized in pt-PT
> # A user go to the KB Article page of pt-PT
> 
> Current Result:
> # It shows the User English Version of the KB
> 
> Expected Result:
> # We can make something like that it will show the user pt-BR version of
> that Article. and say the user that its not available in pt-PT but available
> in pt-BR.
> 
> ## We can make choose which will be fallback local for a local

We can also just set up PT-PT to have PT-BR as a fallback locale, using the existing settings in Kitsune (unless I'm terribly mistaken in understanding how the current fallback setup works). The feature is already there.

We are not showing any information to the user about the article not being available in their language, but we can possibly add that.
Flags: needinfo?(mdziewonski)
(In reply to vesper from comment #10)
> We can also just set up PT-PT to have PT-BR as a fallback locale, using the
> existing settings in Kitsune (unless I'm terribly mistaken in understanding
> how the current fallback setup works). The feature is already there.
> 
> We are not showing any information to the user about the article not being
> available in their language, but we can possibly add that.

The Current Fallback System does not work for each Article. It works for whole system. It works like below:
# A user tries to go to the support site with pt-PT local
# It should redirect them to pt-BR Locale  (If we set with current fallback system)

This should be happen and expected:
Situation: There are two KB article named "A" and "B"
# "A" is translated to pt-PT and pt-BR both Local
# "B" is not translated to pt-PT but is translated to pt-BR

Actual Result:
** Now a user goes to "A" article with pt-PT local
# It will show them the pt-PT version of the article
** A user goes to the "B" Article with pt-PT local
# It shows them the English version of the article

Expected Result:
# It should show the user pt-BR version of the article

Note:
We should implement a fallback that will work like below
# While the article is translated to pt-PT, the user would see pt-PT version of article
# While the article is missing translation of pt-PT, the user would see pt-BR version of '''That Article'''

Can I make you clear vesper?
Flags: needinfo?(mdziewonski)
No need to make it any clearer, Safwan. So, that would require extending the existing "whole site" fallback to individual articles. I don't think we should be creating a separate, additional list of fallback locales.

Does that sounds good, Safwan?
We discussed this with Safwan right now and he outlined how the feature should work.

Based on his feedback, he will set up a separate fallback for article-level pages, with conditions described above and we'll test it to see if it works as expected.

Thank you, Safwan!
Assignee: nobody → safwan.rahman15
Status: RESOLVED → REOPENED
Flags: needinfo?(mdziewonski)
Resolution: FIXED → ---
Hi vesper,
Thanks for that. Need the Local mapping for fallback locals. Can you provide that? Be informed that a local can have many fallback locals.
Also, Can you please provide the message that will be show in the Article while this fallback happen?
Flags: needinfo?(mdziewonski)
Fallback mapping (initial selection of locales)

bn-IN => bn-BD
ca => es
eu => es
gl => es
gu-IN => hi-IN
wo => fr
fy-NL => nl
pt-PT => pt-BR
ta-LK => ta
zh-CN => zh-TW
zh-TW => zh-CN


As for the message - what locale will the message appear in? The redirected locale? The currenty page locale? Maybe it should appear in both?
Flags: needinfo?(safwan.rahman15)
(In reply to vesper from comment #15)
> zh-CN => zh-TW
> zh-TW => zh-CN

Wut?
(In reply to Francesco Lodolo [:flod] from comment #16)
> (In reply to vesper from comment #15)
> > zh-CN => zh-TW
> > zh-TW => zh-CN
> 
> Wut?

Never mind, I completely missed the subject of this bug. But at this point I'd have a different question, and that's why we're not using accept_languages header for this kind of fallback.
(In reply to vesper from comment #15)
> Fallback mapping (initial selection of locales) 
> As for the message - what locale will the message appear in? The redirected
> locale? The currenty page locale? Maybe it should appear in both?

I believe the user should see a message when
# A user go to a Article with pt-PT
# The article does not have any version of pt-PT, so redirect to pt-BR
# The user see pt-BR version of that Article.

When the user see another version of the Article, he should see a message appear in. like "The Article you are trying to read does not have a version in pt-PT. So we redirect you to see pt-BR of the version. If you would like to contribute localizing this document in pt-PT, click here(link). Otherwise click here to see the en-US version"
Flags: needinfo?(safwan.rahman15)
(In reply to Francesco Lodolo [:flod] from comment #17)
> (In reply to Francesco Lodolo [:flod] from comment #16)
> > (In reply to vesper from comment #15)
> > > zh-CN => zh-TW
> > > zh-TW => zh-CN
> > 
> > Wut?
> 
> Never mind, I completely missed the subject of this bug. But at this point
> I'd have a different question, and that's why we're not using
> accept_languages header for this kind of fallback.

Lodolo, Can you please check the comment 11? If still its unclear to you, please let me know.
Moreover Michal can describe more.
Flags: needinfo?(francesco.lodolo)
(In reply to Safwan Rahman from comment #19)
> Lodolo, Can you please check the comment 11? If still its unclear to you,
> please let me know. Moreover Michal can describe more.

I have the feeling we're reinventing the wheel.

When the user visits SUMO with Firefox in Portuguese (pt-PT), it sends an accept_language header with "pt-pt, pt, en, en-us".
https://transvision.mozfr.org/string/?entity=toolkit/chrome/global/intl.properties:intl.accept_languages&repo=aurora

The only thing you'd need to do is to map pt to pt-BR, if pt-BR is the locale with more articles translated. 

Also note that users can customize this from preferences, and ask for 'es' as a second locale if pt-PT is not available. That preference would be completely ignored if you manage redirections like you plan to do.

If you really want to create your own set of mappings, double check with localizers: are you sure that Gujarati prefers Hindi to English? From the current accept_languages I wouldn't be completely sure.
Flags: needinfo?(francesco.lodolo)
(In reply to Francesco Lodolo [:flod] from comment #20)
> (In reply to Safwan Rahman from comment #19)
> > Lodolo, Can you please check the comment 11? If still its unclear to you,
> > please let me know. Moreover Michal can describe more.
> 
> I have the feeling we're reinventing the wheel.
> 
> When the user visits SUMO with Firefox in Portuguese (pt-PT), it sends an
> accept_language header with "pt-pt, pt, en, en-us".
> https://transvision.mozfr.org/string/?entity=toolkit/chrome/global/intl.
> properties:intl.accept_languages&repo=aurora
> 
> The only thing you'd need to do is to map pt to pt-BR, if pt-BR is the
> locale with more articles translated. 

No. We dont running the Wheel twice. Actually, The accept_language header works for site. Like a site is available or not. So with pt-PT version of Firefox, it will check if there is any version of the SUMO website in pt-PT version. But, it will not check the content of the site. As we support both pt-PT and pt-BR version, SUMO site have both version. But the fact is about content. Does all the content available in pt-PT or pt-BR? So what happen if any content is not available in pt-PT? does not it make sense to show pt-BR Content rather than English Content to the user?

Yes, we could use the accept_language header if pt-PT version of SUMO website is not available. Like we did it in past for 'bn' Local with Bug 1071060. 

>Also note that users can customize this from preferences, and ask for 'es' as a second locale if >pt-PT is not available. That preference would be completely ignored if you manage redirections like >you plan to do.

As We have the both local pt-PT and pt-BR available in our website, I dont think we can use the accept_language header to determine the fallback local of the user and "Show content" according to it. So actually I could not get any ways to determine the users preference and make it fallback local to "show content" according to it. If we did not have pt-PT or pt-BR local in our SUMO, we could simply do it by adding fallback local in our settings. Like we did in the Bug I mentioned.

>If you really want to create your own set of mappings, double check with localizers: are you sure >that Gujarati prefers Hindi to English? From the current accept_languages I wouldn't be completely >sure.

I think The best answer about it can be given by Vesper. But As I can say, hi-IN is the first official Language of India. As its used in all the official Work in India, the best fallback local of gu-IN would must be hi-IN. As they belong from Same country and hi-IN is the official Language which must be used all over the country in India for Official work. So it must be better to show hi-IN version rather than en-US version if we dont have any content available in gu-IN
Thanks for all the comments and discussion.

I admit I put the list of backup locales together without knowing about accept_languages.

As much as I understand, Safwan's code allows the SUMO site to display fallback content on a single article scale, rather than just on the whole site scale (which is what we already have in place). If accept_languages is the Mozilla-wide standard, we can just adapt that list to work with Safwan's code.

(In reply to Safwan Rahman from comment #18)
> I believe the user should see a message when
> # A user go to a Article with pt-PT
> # The article does not have any version of pt-PT, so redirect to pt-BR
> # The user see pt-BR version of that Article.
> 
> When the user see another version of the Article, he should see a message
> appear in. like "The Article you are trying to read does not have a version
> in pt-PT. So we redirect you to see pt-BR of the version. If you would like
> to contribute localizing this document in pt-PT, click here(link). Otherwise
> click here to see the en-US version"

OK, so I understand we should show the pt-BR message on the pt-BR locale, in order to keep it simple and obvious.

How about: "This page does not exist in ((pt-PT)). You have been redirected to the ((pt-BR)) version instead. If you would like to localize it into ((pt-PT)), <<click here>>. You can also see the <<en-US>> version of this page."

The ((pt-PT)) and ((pt-BR)) parts should be automatically set based on the locale the user wanted to see and the one they get redirected to.

<<click here>> should be a link pointing to https://support.mozilla.org/kb/localize-mozilla-support

<<en-US>> should be a link pointing to the en-US version of the article.

Does that sound clear, Safwan?
Flags: needinfo?(safwan.rahman15)
Flags: needinfo?(safwan.rahman15)
Flags: needinfo?(mdziewonski)
Here's the final summary of the process and the message:

1. A user visits a KB Article which is yet not localized in the locale s/he requested.

2. We first find if any fallback locale mentioned in accept_languages exists, and whether it is supported by SUMO:

    2A) If supported, we find out if the article is localized in that locale.

    2B) If not supported, we check if we have any other fallback locale available in our KB for the missing locale that is supported by SUMO.

3) If neither 2, nor 2A, nor 2B is true, we check the fallback locale in our custom mapping for each article.

4) If 3 is not true, we show the en-US version of the page.

The message has been described in comment #22 and will be displayed in the case of any redirect happening.
Francesco, Can you please check comment 23? Do you like this process that being made? Or you would like to make some other thing.
Flags: needinfo?(francesco.lodolo)
(In reply to Safwan Rahman from comment #24)
> Francesco, Can you please check comment 23? Do you like this process that
> being made? Or you would like to make some other thing.

Nothing to add, sounds good.
Flags: needinfo?(francesco.lodolo)
(In reply to Francesco Lodolo [:flod] from comment #25)
> (In reply to Safwan Rahman from comment #24)
> > Francesco, Can you please check comment 23? Do you like this process that
> > being made? Or you would like to make some other thing.
> 
> Nothing to add, sounds good.

Thanks a lot Francesco for your concern about the Fallback. Therefore we could brainstorm and redesign the plan and go ahead with a much better thing. So I will start working on the plan described in comment 23
Again Thanks a lot.
(In reply to vesper from comment #15)
> zh-CN => zh-TW
> zh-TW => zh-CN

zh-HK => zh-TW would be somehow acceptable, but please DO NOT have zh-TW fallen back to zh-CN and vice versa. the terms and usages would be more different than to other similar locales and it breaks user experience and localizer's efforts.
Whiteboard: u=user c=wiki p= s= → u=user c=wiki p=0 s=2015.12
@Safwan: we are ready to roll for the following

bn-IN => bn-BD
bn-BD => bn-IN
ca => es
eu => es
gl => es
wo => fr
fy-NL => nl

Let me know if there's anything else you need done from my side to make sure we roll this out successfully. As a reminder: the messaging for the redirect is in https://bugzilla.mozilla.org/show_bug.cgi?id=800880#c22
Flags: needinfo?(safwan.rahman15)
Thanks for the confirmation of locale and also helping me to correct the docstring. I have made a PR which is under review. Hope to get it merged soon.
https://github.com/mozilla/kitsune/pull/2573
Flags: needinfo?(safwan.rahman15)
Attached file Proposed Patch — 

    
    

    
  
This has been deployed to prod.
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED

    
    

    
  
Thanks to everyone who participated in the discussion and completion of this feature!

    
    

    
  
Catch a bug. I visit same link in mobile and desktop. I check the note in the top. In mobile no locale name is showing, but in desktop locale name is showing. Please have a look in the given screenshot. 

Desktop view : http://i.imgur.com/JiktHNa.png?1
Mobile view : http://i.imgur.com/uFMkP1R.jpg

    
    

    
  
(In reply to Ashickur Rahman from comment #34)
> Catch a bug. I visit same link in mobile and desktop. I check the note in
> the top. In mobile no locale name is showing, but in desktop locale name is
> showing. Please have a look in the given screenshot. 
> 
> Desktop view : http://i.imgur.com/JiktHNa.png?1
> Mobile view : http://i.imgur.com/uFMkP1R.jpg

Hmm, it was a bug in the code and fixed in the following PR. It will be fixed once the Patch got landed! :)
https://github.com/mozilla/kitsune/pull/2679

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: