656789 - Fix locale detection

Status: RESOLVED FIXED

(Websites :: webifyme.org, defect, P1)

Description

[Austin King [:ozten]]

Analogous to Markup's Bug#652912, we need proper locale detection and L10n support.

This can be fixed after Bug#650033, since gettext is used currently and that would not change.

Comment 1

[Austin King [:ozten]]

(In reply to comment #0)

> This can be fixed after Bug#650033

This is a comment on priority. Please don't wait until that bug is closed to start.

Comment 2

[Austin King [:ozten]]

Following the changes applied to Markup, we should use tower, commons middleware, etc.

Tower bug split out into Bug#662846.

Comment 3

[[:brez]]

Both tower and commons middleware are currently enabled in webifyme.. i.e. what exactly is the request here?  Please clarify - thanks

Comment 4

[[:brez]]

Did find some errors while processing though..

./manage.py makemessages --locale=es
processing language es
Error: errors happened while running xgettext on tests.py
./vendor/src/django/tests/regressiontests/i18n/tests.py:92: warning: internationalized messages should not contain the `\r' escape sequence

Will chk into that

Comment 5

[[:brez]]

And extract throws:


ImportError: No module named translate.storage

So I'm guessing its there (as previously stated) but not working - anyhow looking into it.

Comment 6

[Austin King [:ozten]]

What is the full stacktrace?

Perhaps I missed adding a dependency? Is your vendor and/or virtualenv up to date?

What version of Python are you running against? I think translate.storage is from the standard library. I'm using 2.6.

Comment 7

[[:brez]]

Full stacktrace / version info:


$ ./manage.py extract
/Users/jbresnik/Projects/mozilla/webifyme/ffenv/lib/python2.6/site-packages/jinja2/__init__.py:31: UserWarning: Module django was already imported from /Users/jbresnik/Projects/mozilla/webifyme/ff4/vendor/src/django/django/__init__.pyc, but /Users/jbresnik/Projects/mozilla/webifyme/ffenv/src/django is being added to sys.path
  __version__ = __import__('pkg_resources') \
/Users/jbresnik/Projects/mozilla/webifyme/ffenv/lib/python2.6/site-packages/jinja2/__init__.py:31: UserWarning: Module tower was already imported from /Users/jbresnik/Projects/mozilla/webifyme/ff4/vendor/src/tower/tower/__init__.pyc, but /Users/jbresnik/Projects/mozilla/webifyme/ffenv/src/tower is being added to sys.path
  __version__ = __import__('pkg_resources') \
Traceback (most recent call last):
  File "./manage.py", line 46, in <module>
    execute_manager(settings)
  File "/Users/jbresnik/Projects/mozilla/webifyme/ff4/vendor/src/django/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/Users/jbresnik/Projects/mozilla/webifyme/ff4/vendor/src/django/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/jbresnik/Projects/mozilla/webifyme/ff4/vendor/src/django/django/core/management/__init__.py", line 261, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "/Users/jbresnik/Projects/mozilla/webifyme/ff4/vendor/src/django/django/core/management/__init__.py", line 67, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "/Users/jbresnik/Projects/mozilla/webifyme/ff4/vendor/src/django/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/Users/jbresnik/Projects/mozilla/webifyme/ff4/vendor/src/tower/tower/management/commands/extract.py", line 15, in <module>
    from translate.storage import po
ImportError: No module named translate.storage
$ python -V
Python 2.6.1

Comment 8

[[:brez]]

Yea youre missing the translate-toolkit - adding now..

Comment 9

[Austin King [:ozten]]

(In reply to comment #8)
I mispoke, agreed not standard library.

translate-toolkit==1.6.0 is already in ff4/requirements/dev.txt

This is not needed at runtime, only by stas at build time to extract the strings. Please don't add to vendor.

Comment 10

[Austin King [:ozten]]

(In reply to comment #4)
(In reply to comment #5)
(In reply to comment #7)
Based on these comments, you are testing the work done in Bug#662846. That is cool, but unnecessary for fixing this bug.


Just to clarify a part of a test plan...

Reproduction Steps:

Locale detection Use Case
1) Switch primary language of Firefox 4 to French
2) https://webifyme-dev.allizom.org/

Actual
Not redirected from https://webifyme-dev.allizom.org/.

Expected
Redirected to https://webifyme-dev.allizom.org/fr/

I'm pretty sure we *don't need* ff4/locale/fr/LC_MESSAGES/django.po.

Fixing this is in the middleware, which doesn't use po files.

Work to be done. Compare this app to Markup
* Need middleware from https://github.com/mozilla/markup/tree/master/ffdemo/commons
* Need settings from https://github.com/mozilla/markup/blob/master/ffdemo/settings.py
** Settings such as KNOWN_LANGUAGES_DEV, KNOWN_LANGUAGES_PROD, etc

Comment 11

[[:brez]]

All set - redirection is working now - essentially there were a lot of updates to the config - implemented the same way markup is.. 

https://github.com/mozilla/webifyme/commit/4c09547cb8baac17dde6e2986ee00b61dbf6d35e

Comment 12

[[:brez]]

Updating vendor as well..

Comment 13

[[:brez]]

All set

https://github.com/mozilla/webifyme-lib/commit/a2942f13696a2cf54f5f8abd14c489f3c4bbe50f

Comment 14

[Austin King [:ozten]]

(In reply to comment #13)
https://webifyme-dev.allizom.org/en-US/ is 500.

I've filed Bug#664603 and will take a stab at fixing it.

Comment 15

[[:brez]]

forgot to add vendor update:

https://github.com/mozilla/webifyme/commit/cedfc0a3f00b6b4309f99b684b562021b2bbf1aa

Comment 16

[Staś Małolepszy :stas]

I think that the locale detection is too greedy right now. 

With (de, en-US, en) in my browser settings, I'm redirected to

https://webifyme-dev.allizom.org/de/

and the text is in German.  When I manually go to 

https://webifyme-dev.allizom.org/en-US

the text is still in German.


Interestingly, with (en-US, en, de) I'm redirected to 

https://webifyme-dev.allizom.org/en-US/

but the text is in German too. /me puzzled.

Comment 17

[[:brez]]

Well how it works is the browser sets the 'Accept-Language' header to whatever you set it to [1] (or in your case a range of preferred languages) - the reason it isn't changing is probably related to your browser's cache or the server's cache (etc )- I guess the real question is, is switching languages during a single session a realistic user scenario?   If so I can dig into the django details and figure out a way to force it with every request but let's make sure this is really needed first. 


[1] http://www.apps.ietf.org/rfc/rfc3282.html

Comment 18

[Staś Małolepszy :stas]

I agree that (In reply to comment #17)
> I guess the real question is, is switching
> languages during a single session a realistic user scenario?

I agree that it's not a realistic scenario, so if caching can be used, that's good.  However, since this is a dev server, should caching get in the way of testing if features work?

Also, we can try to detect user's preference, but whatever we detect should have lower priority than the locales explicitly specified in the URL.  So even if my prefs say (de, en-US), I should see English when I go to /en-US.

QA folks: I'd appreciate any help here to verify that the locale detection is working fine.  See comment 16 for testing scenarios.

Comment 19

[[:brez]]

Yea I'll keep digging to figure out what's going on here..

Comment 20

[[:brez]]

Been digging into this and discovered a few things: 

The LocaleURLMiddleware that we use ('commons' not the Django version) deliberately strips out the Locale as part of processing the request - so here's a scenario that explains what is going on by design:

1. I have my locale in the browser preferences set to es-mx,ru (mexican spanish, then russian) - this is confirmed by looking at the header in the incoming request for /fr/quiz

(Pdb) request.META['HTTP_ACCEPT_LANGUAGE']
'es-mx,ru;q=0.5'

Also the URL path:

(Pdb) prefixer.locale
'fr'

Which activates 'fr' for this request - problem is, since the locale is stripped out of the path, further requests default to en-mx (which I've specified in the browser preferences). 

So not sure if that's what we want or not..

Still looking into why the proper translation isn't being activated based on the URL locale setting.. more to come.

Comment 21

[[:brez]]

Ok so the problem was really just be misconfigured - you can now type in a locale into the URL and see that language (tested this many times) *but once you navigate to a different link it will return to what you have defaulted in your browser - this is *not a bug but in fact how the commons library is implemented. Also any unsupported locale (defined as *not having an associated po / mo file) will default to en-US, this is also *not a bug but how the commons library is implemented. 

https://github.com/mozilla/webifyme/commit/00070e41bd72860b66a3e6c43ef2d7ab30a49a49

Comment 22

[[:brez]]

Ok so the problem was really just *locale be misconfigured