538634 - Store raw data for product-details and locale-details as JSON for cross-language compatibility

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Webdev, task)

Description

[Fred Wenzel [:wenzel]]

A lot of projects and groups seem to need access to product-details and/or locale-details now or very soon: http://viewvc.svn.mozilla.org/vc/libs/product-details/

Current users:
- mozilla.com
- AMO
- ...?

Future users:
- crash-reporter
- AMO/Zamboni
- Bouncer
- Metrics? (bug 398366 comment 11)
- Browser Ballot

Problems:
- everytime the lib changes, all "consumers" need to be SVN upped in production
- it's PHP-only but we have a number of Python projects in the queue
- library changes are committed by webdev, not by L10n (for locales) or build (for products). They can't easily be automated right now.

Comment 1

[Fred Wenzel [:wenzel]]

Wil was thinking about making a web service for this data that would spit out JSON and/or XML. All consumers could then poll the service (and cache that data for a while) without the need to SVN up all the time, especially not mozilla.com at release time ;).

An initial service for this could be very, very simple: a PHP file that imports the existing library and outputs JSON. But that won't help with the last problem.

If we want to make this changeable by the people who actually produce the product versions and approve locales, we need a management app, in which case I'd prefer keeping the data in a database and probably using Django's admin pages to edit it. The front-end of that app would then take care of the JSON/XML stuff.

What do you guys think? clouserw: Can you fill in what I might have missed here?

Comment 2

[Austin King [:ozten]]

crash-stats db has a admin panel to manage a "product visibility" table.

This captures product, version, branch, release, start date, and end date.
product - Firefox
version - 3.5.6
branch - 1.9.1 (gecko)
release - major, milestone, development
start - roughly the release date
end - roughly when crash-stats should stop featuring it (stop doing trend analysis, etc)

It lacks "latest version" flags and probably some other features that are needed.

The quality of the data isn't stellar.

Comment 3

[Fred Wenzel [:wenzel]]

I discussed this with clouserw, and we decided to drop the locale and product data into one or more separate JSON files for now and parse it in what ever language we need for a project. PHP projects will keep being able to use the current library, Python projects might want to read JSON directly, or we'll write a new wrapper if we need helper functions.

We'll defer web service access and admin pages for now, until we decide to move forward and can allocate resources for it.

Comment 4

[Fred Wenzel [:wenzel]]

Attachment: Export script

This is an export script that takes the main data arrays in the PHP library and makes JSON files. It'll need to be run every time the PHP data changes.

Note: I only included Firefox and Thunderbird data for now, but whoever feels inclined may add Mobile as well.

Comment 5

[Wil Clouser [:clouserw]]

Comment on attachment 421073 [details] [diff] [review]
Export script

should work

Comment 6

[Fred Wenzel [:wenzel]]

r59505 for the export script, and r59506 for the initial set of JSON files. Calling this fixed, and everybody can start using these files as they see fit.

Comment 7

[Axel Hecht [:Pike]]

What are the right public URLs to those files?

Comment 8

[Wil Clouser [:clouserw]]

There isn't any official ones yet, that'll come with the admin interface.  If you want to use these you should check out the SVN external or, if you want to cheat, you can grab them from http://svn.mozilla.org/libs/product-details/json/