867973 - [FHR] Add a fallback rewrite for missing locales to serve up en-US

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Pascal Chevrel:pascalc]

FHR should work with all locales we have on Aurora/Beta/Release channels. If a locale has not translated the page, we should generate the page anyway in English so as to avoid a 404 because this is a static site and pages are created with a script, there is no automatic fallback to en-US.

settings.py contains the data, the LANGS list should contain all our locale codes. The LANG_FALLBACK should be used for locales we know that we won't have completed on time and want to fallback to en-US instead.

Assigning to myself.

Comment 1

[Pascal Chevrel:pascalc]

Attachment: list if locale codes for FHR

Pike, this is the list I am using for FHR webparts, AFAIK, it has all the locales we ship on Aurora and release channels plus a couple of potential ones. Which locales should I add to play safe in the months to come? We can alias them to en-US but we need the locale codes in a list on the server as this is static content.

Comment 2

[Axel Hecht [:Pike]]

Comment on attachment 745166 [details]
list if locale codes for FHR

This is gonna pick the locale through the urlformatter, that is, selectedlocale of the global package. That includes all language packs out there etc.

This is open to all kinds of combinations of legal, non-legal, wrong casings etc settings, in particular for people with language packs.

I'd be very concerned if people got a 404 page on the healthreport as feedback on that user error.

We may have to code the fallback to en-US into the healthreport client code, if we can't encode it server side.

Comment 3

[Pascal Chevrel:pascalc]

After talking to Pike and Laura on IRC, let's morph that bug into adding a locale fallback mechanism to the site à la mozilla.org

We can probably do that vu

Comment 4

[Pascal Chevrel:pascalc]

We can probably do that with apache, what we need is that if we get a hit for a locale code we don't know and therefore don't have a folder for, we look at the accept-lang headers and redirect to the best locale available, en-US being the fallback locale. This way we make sure we never 404 for future locales or langpacks for locales we don't ship

Comment 5

[Pascal Chevrel:pascalc]

jakem confirmed on IRC that it shouldn't be a problem to deal with that.

Comment 6

[Schalk Neethling [:espressive]]

(In reply to Pascal Chevrel:pascalc from comment #5)
> jakem confirmed on IRC that it shouldn't be a problem to deal with that.

So I guess we can reassign to jakem?

Comment 7

[Pascal Chevrel:pascalc]

Yes, that's probably a rewrite rule to add to the apache config

Comment 8

[Jake Maul [:jakem]]

I missed this getting assigned to me. I'm going to move, retitle, and delegate.

To clarify:

If any files are 404 for a given locale, you want to silently rewrite (not redirect) them to serve the en-US ones instead?

Something like this (syntax not checked):

RewriteCond <file does not exist>
RewriteRule /<locale>/(.*) /en-US/$1

Does that sound about right?

Comment 9

[Chris Turra [:cturra]]

we could do something like this:

  RewriteCond     %{REQUEST_FILENAME} !-f
  RewriteRule     ^/.*/(.*)  /en-US/$1 [R=302,L]


it should do exactly what :jakem proposed above and i have validated the syntax ;)

Comment 10

[Chris Turra [:cturra]]

alright, scratch my last comment, :jakem pointed out some major flaws with it. first, i was assuming the first .* would find all matches between just the first and second / - i was wrong. second, we don't want an external redirect, but an internal one.

the following rewritecond/rule should accomplish just that:

  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteRule ^/([a-zA-Z]{2,3}(?:-[a-zA-Z]{2})?)/(.*)$ /en-US/$2 [PT]

Comment 11

[Chris Turra [:cturra]]

i have now applied a slightly modified redirect to fhr-dev. the rewriterule in my comment above was didn't work with one locale (ja-JP-mac). that has now been sorted. please give it some testing and let me know how things looks.

  RewriteRule ^/([a-zA-Z]{2,3}(?:-[a-zA-Z]{2})?(?:-[a-zA-Z]{3})?)/(.*)$ /en-US/$2 [PT]

Comment 12

[Pascal Chevrel:pascalc]

Thanks Chris, this is an improvement over the current situation as we now have a fallback to en-US for locales we don't support yet, but there is no language negociation (for example, fr-BE goes to en-US while it should go to fr).

If we don't have a short term solution to have language negociation before the fallback to en-US, I think a good first step and a big improvement over the current situation would be to use your rewrite rule now.

Comment 13

[Chris Turra [:cturra]]

good catch, while my rules worked in my test environment, they were not working correctly in fhr-dev. i have now applied an additional set of rewrite conditions to check for the locale's more explicitly.

 RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f
 RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-d
 RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-l


the example you gave of fr-BE doesn't appear to actually be in the web-output directory for fhr-dev. 

 [[e-mail]1 ~]# ls -l /data/www/fhr-dev.allizom.org/web-output/fr-BE/
 ls: cannot access /data/www/fhr-dev.allizom.org/web-output/fr-BE/: No such file or directory


i did use 'wo' to test a similar use case however, since it is a symlink to 'fr'.

 [[e-mail]1 ~]# ls -l /data/www/fhr-dev.allizom.org/web-output/wo
 lrwxrwxrwx 1 root root 2 May 13 09:16 /data/www/fhr-dev.allizom.org/web-output/wo -> fr

Comment 14

[Pascal Chevrel:pascalc]

The idea is  to make sure that if next year we have for example es-VE (Spanish from Venezuela) but we don't have a translation for it (which means no es-VE folder), the fallback locale would be another Spanish, not en-US (hence the locale negociation piece I mentionned as missing).

Comment 15

[Chris Turra [:cturra]]

understood and that's exactly what the rewrite condition check i add does. what it does is check that the requested files, directory (including symlinks) are not found. if that's a positive match, then it does the en-US internal redirect.

as a result, any additional locales that we add with not match these conditions and result in NO redirect.

Comment 16

[Chris Turra [:cturra]]

Pascal - please let me know when you have a chance to complete testing on this so we can push to production and get this bug closed off.

Comment 17

[Chris Turra [:cturra]]

Pascal - i just wanted to ping for an update on this bug. please let me know when your testing is complete so we can push these changes into production.

Comment 18

[Francesco Lodolo [:flod]]

(In reply to Chris Turra [:cturra] from comment #15)
> understood and that's exactly what the rewrite condition check i add does.
> what it does is check that the requested files, directory (including
> symlinks) are not found. if that's a positive match, then it does the en-US
> internal redirect.

(Pascal is on PTO these days)
Not sure you're talking about the same thing.

Your proposal, let's say a visitor with browser set to es-VE open FHR: folder is missing, redirected to en-US. One month from now we add the es-VE folder, redirect goes away.

What Pascal is asking is that we use language negotiation with the browser (accepted languages): a visitor with es-VE will probably have es-ES or es as an alternative to es-VE, so we should fallback to that, not to en-US.

Comment 19

[Chris Turra [:cturra]]

thanks for the clarification :flod. there is probably some mod_rewrite magic i can do with %{HTTP:Accept-Language}, but will need some time to think that through. will report back when i have an update.

Comment 20

[Pascal Chevrel:pascalc]

clearing the needinfo flag for me as flod answered while I was on PTO, thanks flod.

Comment 21

[Chris Turra [:cturra]]

(In reply to Francesco Lodolo [:flod] from comment #18)
> What Pascal is asking is that we use language negotiation with the browser
> (accepted languages): a visitor with es-VE will probably have es-ES or es as
> an alternative to es-VE, so we should fallback to that, not to en-US.

sorry for the delay getting back here -- i had some time this morning to look into this more and from what i can tell, if we're receiving the Accept-Language header from our users, we could setup a fall back other than en-US. however, we'll need to maintain a list of those "fall backs" that would need to be managed within the apache config. if you want to go this route, we'll need a list of those mappings.

below would be an example mapping for es-VE -> es-ES.
  
  RewriteCond  %{HTTP:Accept-Language} ^es-VE  [NC]
  RewriteRule  ^/([a-zA-Z]{2,3}(?:-[a-zA-Z]{2})?(?:-[a-zA-Z]{3})?)/(.*)$ /es-ES/$2 [R=302,PT]

Comment 22

[Francesco Lodolo [:flod]]

This, besides having us to maintain this list, basically always redirects a user on es-VE to es-ES, correct? 

The idea instead would be, assuming it's possible to do it (which I'm not sure)
* a user with Accept-Language "es-cl, es-es, en-us, en" visits FHR.
* we don't have es-cl, we try es-es, we have that, fall back to es-es (which is called es-ES, maybe another complication). In case we don't have any acceptable fallback in Accept-Language, just use en-US.

This would not require us maintaining an hard-coded list of redirects.

Comment 23

[Chris Turra [:cturra]]

:flod - what you're asking for totally makes sense, however, this however is not something we can do in apache. i would suggest if you need some sort of locale mapping/fall back, it would be better done within the application than on the web server.

Comment 24

[Chris Turra [:cturra]]

(In reply to Pascal Chevrel:pascalc from comment #0)
> FHR should work with all locales we have on Aurora/Beta/Release channels. If
> a locale has not translated the page, we should generate the page anyway in
> English so as to avoid a 404 because this is a static site and pages are
> created with a script, there is no automatic fallback to en-US.

circling back to the original request, which asked if locale's that are not found be send to english instead of a 404. i believe this is requirement is currently satisfied in dev. do you want to move forward with that or come up with a solution in app to solve the case of locale fallbacks as discussed in comment 22? we'll be able to do the former, but not the latter for you.

Comment 25

[Francesco Lodolo [:flod]]

I guess this locales should be fixed with the rewrite rule, and then we can evaluate the cost of doing locale fallbacks in app. But I'd like to see also what Pascal thinks.

Comment 26

[Chris Turra [:cturra]]

meant to mark this bug as assigned, not unconfirmed.

Comment 27

[Pascal Chevrel:pascalc]

Here is what I would propose:
If the locale code doesn't exist, Apache redirects to a python script that does language negociation and then the script shows the right fallback for the locale. what do you think?

Comment 28

[Chris Turra [:cturra]]

(In reply to Pascal Chevrel:pascalc from comment #27)
> Here is what I would propose:
> If the locale code doesn't exist, Apache redirects to a python script that
> does language negociation and then the script shows the right fallback for
> the locale. what do you think?

works for me! just let me know where you'd like the rewrite for locale's which are not found to be sent.

Comment 29

[Chris Turra [:cturra]]

:pascalc - i just wanted to ping on this again. please let me know where you'd like this rewrite to be sent so i can get that complete in dev for you.

Comment 30

[Pascal Chevrel:pascalc]

Chris, that's actually a question for the webdevs that work on FHR, I don't work on this codebase.

Comment 31

[Chris Turra [:cturra]]

understood. since this bug it not actionable, i am going to mark as r/wontfix. once the code is in place and there are details how to fulfill this request (where to point the redirect to), feel free to reopen.