Closed Bug 1238696 Opened 8 years ago Closed 8 years ago

move update_product_details cron job to celery

Categories

(developer.mozilla.org Graveyard :: Code Cleanup, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: willkg, Assigned: willkg)

Details

MDN uses django-mozilla-product-details 0.7.1 <https://pypi.python.org/pypi/django-mozilla-product-details> for maintaining details about Mozilla products (locales, versions, etc). It has a Django management command which is kicked off via cron.

We want to upgrade to 0.9 which allows us to save data to the db rather than the file system. Further, we want to switch this from a cron job to a celery task.

This work blocks the transition to AWS. It shouldn't otherwise affect anything.

This bug covers:

1. upgrading to version 0.9
2. changing MDN so that the product details get saved to the db rather than the filesystem
3. switching from a cron job to a celery task
Grabbing this to work on soon.
Assignee: nobody → willkg
Status: NEW → ASSIGNED
One caveat, the last time we tried updating to a newer version, it froze in a weird way since we populate the language selector using product details data. The work around of making the iteration lazy didn't work. Just a heads-up.
Exciting! I'll keep this in mind.
I've been waiting for the settings refactor and now the pip 8 update to land before I do more work on this. Soon!
The pip8 stuff landed, so we're good with that.

I'm watching the Mikes adding django-mozilla-product-details to normandy. I'll wait for the flurry of activity there to stop because it's possible we'd hit all the same issues. Right now they're dealing with some LocaleField problem or something like that.
Updated to django-mozilla-product-details 0.10. In a pr: https://github.com/mozilla/kuma/pull/3767

I'll switch to the db backend after that lands and we push it out and everything is good.

I looked into whether we'll hit the same issues Normandy did. I'm pretty sure we won't. MDN doesn't have a LocaleField like Normandy where they were populating choices with the value from product_details, but if that's coming from the db then everything goes awry if there's no db.

Anyhow, I think it should be ok to switch. I'll do that in a future PR.
:willkg FTR We do generate a mapping of lower case language names and their native names in kuma.core.apps.CoreConfig.language_mapping which is executed at the start of the process. According to the Django Docs (https://docs.djangoproject.com/en/1.9/ref/applications/#django.apps.AppConfig.ready) it's not recommended to interact with the database at that stage of the process though. We started to use this method to be able to provide that mapping to other parts of the project without fearing import time side effects or even worse settings init time side effects (that we had before).
:jezdez We have languages.json in the repo--it's a copy of what product-details gives us. That's used by settings to generate LOCALES. Then in kuma.core.apps.CoreConfig, it looks like we look at product-details to build language_mapping, but ... we've already got that data in settings.LANGUAGES. The difference between the two is that LANGUAGES is a sorted list and has en-US like things and get_language_mapping() returns a dict and has en-us like things.

I was thinking I'd switch get_language_mapping() to build the mapping based on settings.LANGUAGES.

If we don't want to do that and instead want to go the other way (fix LANGUAGES so it's not building from languages.json and instead builds from the db), then I think we've got bootstrapping issues and I'm not sure how to solve that other than changing product-details so it's updated during deploys or committed to the repo rather than updated via cron.

Thoughts on that?
Ah yes, I was worried that there are some differences in the various lists of locales and languages we maintain. The settings are a bit confusing in that regard. I'm in favor of making sure the settings will never reach out to a database to build a list.
I looked at the Kuma code and some other things. We've got three things that "use" product-details:

1. settings.py LANGUAGES is generated using data from get_locales() which pulls from a file kuma/languages.json which is a copy of the one that product-details gives us and is updated every time we add a new locale

2. kuma/core/apps.py:CoreConfig has a language_mapping property that pulls from product-details languages and is cached

3. kuma/users/forms.py has a NewsletterForm that has a country ChoiceField which has choices that pull from product-details regions for the specified locale

That's it.

So we're using regions and langauges from product-details. I'm sure these change, but probably not with the frequency of product version information. Further, even if they change, I'm not sure MDN needs the most-up-to-date version ever.

Moving on, the webheads have product detail information in /data/www/developer.mozilla.org/product_details_json/ . This directory is on the server--it's not an nfs mount. The admin node has a cronjob that runs and updates /data/developer/src/developer.mozilla.org/product_details_json/ . The cronjob just updates the files--it doesn't copy or push them out. Seems like it's only during a deploy that the files that update every night get copied out to all the servers.


Thus to summarize:

1. Kuma uses region and language information from product-details, but this information doesn't update very often.

2. Even though we've got a cronjob running on the admin node to update the files, the information in those files isn't available to any thing that uses them until we do a deploy.


Assuming that's correct (I'm pretty sure it is), we should nix the cron job and change the deploy process to also update product-details and we will have exactly the same situation we have now, but without cron.


Given that, I want to WONTFIX this bug and start a new bug that covers fixing our use of product-details and simplifying it a bit.
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
Product: developer.mozilla.org → developer.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.