1083718 - Update ship-it to export the same json files as product-details

Status: RESOLVED FIXED

(Release Engineering :: Applications: Shipit, defect)

Description

[Sylvestre Ledru [:Sylvestre]]

To export the current list of releases and history, we used a series of PHP which will export the data into json files.
I think we should use the ship-it database to replace the old PHP(4) library.

For those who are not familiar with the product details, we have a few PHP files which we update by hand (should be automatic soon, see bug 1053814).
Basically, we provide:
* current aurora, beta & release version
* history of these versions
This applies for Firefox, mobile & Thunderbird (but not Firefox OS).
We used to include PHP files when our websites were developed in PHP.
I started to hack on this to remove the deprecated code and improve the json schema (bug 1080532). However, the www team is complaining that I am breaking many things (even if the values are not being used... like the filesize).

So, instead of hacking on top of product-details to add a new schema (and adding more legacy management on top of the product detail), I would like to know if it could make sense to use ship-it to manage that.

I see a few issues to make that happens but I don't see any blocker.
* it is only on the VPN while the json needs to be exposed publicly. We could just use the ship-it database (without its front end)
* we don't have the locale info. Example:
        var $languages = array(
            'af'         => array( 'English' => 'Afrikaans',                 'native' => 'Afrikaans'),
and the properties files

* we don't have the whole history but that should be trivial to import.

* we don't know yet if build has been shipped or not (but this is on the releng roadmap)

Comment 1

[Sylvestre Ledru [:Sylvestre]]

Attachment: 0001-Export-to-json-the-product-details.patch

Here is a proof of concept.
It is still need:
* thunderbird json files
* mobile_details.json to be implemented [1]
* import of the history
* cleaning


[1] The schema seems a bit odd to me:

{
    "version": "33.0",
    "beta_version": "34.0b4",
    "alpha_version": "35.0a2",
    "builds": [
        {
            "locale": {
                "code": "ar",
                "english": "Arabic",
                "native": "\u0639\u0631\u0628\u064a"
            }
        },
        {
            "locale": {
[...]
    "beta_builds": [
        {
            "locale": {
                "code": "cs",
                "english": "Czech",
                "native": "\u010ce\u0161tina"
            },
            "download": {
                "android": "market:\/\/details?id=org.mozilla.firefox_beta"
            }
        },
        {
            "locale": {
                "code": "de",
                "english": "German",
[...]
    ],
    "alpha_builds": [
        {
            "locale": {
                "code": "en-US",
                "english": "English (US)",
                "native": "English (US)"
            },
            "download": {
                "android": "market:\/\/details?id=org.mozilla.firefox"
            }
        }
    ]
}

Comment 2

[Sylvestre Ledru [:Sylvestre]]

Attachment: 0001-Export-to-json-the-product-details.patch

Not ready but getting closer.

Comment 3

[Sylvestre Ledru [:Sylvestre]]

FYI, I am now working on this branch:
https://github.com/sylvestre/ship-it/tree/json
and travis:
https://travis-ci.org/sylvestre/ship-it/builds

Comment 4

[[e-mail] (:bhearsum)]

Mass component change for ship it bugs.

Comment 5

[Kohei Yoshino]

Any ETA here? I'd like to have Thunderbird data for Bug 1123399, but if this takes time, I may probably go with the legacy product-details instead.

Comment 6

[Sylvestre Ledru [:Sylvestre]]

The goal is to provide the exact same json files as in p-d. So, you will/should have to update only the URL.
Anyway, I just got a VM to finalize the work. Hopefully, it is going to be live in April.

Comment 7

[Kohei Yoshino]

Oh, okay. Looks like I was confused. Thunderbird stable version data is already available in p-d. I wanted to see the Thunderbird Beta & Earlybird data too, but it can be added later. Thanks.

Comment 8

[Sylvestre Ledru [:Sylvestre]]

So, with my tb hat (I have been helping lately), it is something that we are not updating but we can start doing it if needed.

Comment 9

[Kohei Yoshino]

I know the Thunderbird Beta info is available in 

http://viewvc.svn.mozilla.org/vc/projects/mozilla.org/branches/staging/thunderbird/includes/thunderbirdBetaBuildDetails.php

It would be appreciated if this info could be added to the main p-d lib.

Comment 11

[Sylvestre Ledru [:Sylvestre]]

Attachment: implement the feature

Comment 12

[[e-mail] (:bhearsum)]

The patch from Github has been merged. We're planning to get it landed in production today, but git.m.o is having some issues at the moment (bug 1176192). If those are sorted out early enough my plan is to get this + bug 1149978 pushed to ship-it-dev to do some verification, and then get that pushed to prod.

Comment 13

[[e-mail] (:bhearsum)]

This is in dev, but broken because of this ISE 500:
2015-06-19 14:52:00,833 - kickoff.log_exception#1306: Exception on /releases.html [GET]
Traceback (most recent call last):
  File "/data/www/ship-it-dev.allizom.org/release-kickoff/vendor/lib/python/flask/app.py", line 1687, in wsgi_app
    response = self.full_dispatch_request()
  File "/data/www/ship-it-dev.allizom.org/release-kickoff/vendor/lib/python/flask/app.py", line 1360, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib64/python2.6/site-packages/newrelic-2.32.0.28/newrelic/hooks/framework_flask.py", line 98, in _nr_wrapper_Flask_handle_exception_
    return wrapped(*args, **kwargs)
  File "/data/www/ship-it-dev.allizom.org/release-kickoff/vendor/lib/python/flask/app.py", line 1358, in full_dispatch_request
    rv = self.dispatch_request()
  File "/data/www/ship-it-dev.allizom.org/release-kickoff/vendor/lib/python/flask/app.py", line 1344, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/lib64/python2.6/site-packages/newrelic-2.32.0.28/newrelic/hooks/framework_flask.py", line 40, in _nr_wrapper_handler_
    return wrapped(*args, **kwargs)
  File "/data/www/ship-it-dev.allizom.org/release-kickoff/vendor/lib/python/flask/views.py", line 83, in view
    return self.dispatch_request(*args, **kwargs)
  File "/data/www/ship-it-dev.allizom.org/release-kickoff/vendor/lib/python/flask/views.py", line 150, in dispatch_request
    return meth(*args, **kwargs)
  File "/data/www/ship-it-dev.allizom.org/release-kickoff/kickoff/views/releases.py", line 113, in get
    return render_template('releases.html', releases=sortedReleases(), form=form)
  File "/data/www/ship-it-dev.allizom.org/release-kickoff/kickoff/views/releases.py", line 32, in sortedReleases
    return sorted(getReleases(), cmp=cmpReleases)
  File "/data/www/ship-it-dev.allizom.org/release-kickoff/kickoff/views/releases.py", line 31, in cmpReleases
    return cmp(y._submittedAt, x._submittedAt)
TypeError: can't compare datetime.datetime to NoneType

Comment 14

[[e-mail] (:bhearsum)]

The previous error was fixed by making sure submittedAt was set for all rows (more fallout from bug 1149978). Sylvestre has identified two more issues:
 1) https://ship-it-dev.allizom.org/json/firefox_primary_builds.json => error 500

2015-06-19 17:02:15,834 - kickoff.log_exception#1306: Exception on /json/firefox_primary_builds.json [GET]
Traceback (most recent call last):
  File "/data/www/ship-it-dev.allizom.org/release-kickoff/vendor/lib/python/flask/app.py", line 1687, in wsgi_app
    response = self.full_dispatch_request()
  File "/data/www/ship-it-dev.allizom.org/release-kickoff/vendor/lib/python/flask/app.py", line 1360, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib64/python2.6/site-packages/newrelic-2.32.0.28/newrelic/hooks/framework_flask.py", line 98, in _nr_wrapper_Flask_handle_exception_
    return wrapped(*args, **kwargs)
  File "/data/www/ship-it-dev.allizom.org/release-kickoff/vendor/lib/python/flask/app.py", line 1358, in full_dispatch_request
    rv = self.dispatch_request()
  File "/data/www/ship-it-dev.allizom.org/release-kickoff/vendor/lib/python/flask/app.py", line 1344, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/lib64/python2.6/site-packages/newrelic-2.32.0.28/newrelic/hooks/framework_flask.py", line 40, in _nr_wrapper_handler_
    return wrapped(*args, **kwargs)
  File "/data/www/ship-it-dev.allizom.org/release-kickoff/kickoff/jsonexport.py", line 155, in firefox_primary_builds_json
    buildsVersionLocales = updateLocaleWithVersionsTable("firefox")
  File "/data/www/ship-it-dev.allizom.org/release-kickoff/kickoff/jsonexport.py", line 136, in updateLocaleWithVersionsTable
    lastStable[0][0])
  File "/data/www/ship-it-dev.allizom.org/release-kickoff/kickoff/jsonexport.py", line 109, in generateLocalizedBuilds
    locales = parsePlainL10nChangesets(l10nchangesets)
  File "/data/www/ship-it-dev.allizom.org/release-kickoff/vendor/lib/python/mozilla/release/l10n.py", line 4, in parsePlainL10nChangesets
    locale, revision = line.rstrip().split()
ValueError: need more than 1 value to unpack


2) https://ship-it-dev.allizom.org/json/languages.json => not found





However, he thinks this doesn't block going to production. If today's Beta ships  in the next couple of hours I'll go ahead and push to prod. If not, this waits until next week.

Comment 15

[[e-mail] (:bhearsum)]

This has been pushed to prod and I _think_ it's looking fine. I also did the necessary imports from bug 1149978. Sylvestre, can you verify?

Comment 17

[Michal Stanke (Mozilla.cz) [:mstanke][:MikkCZ]]

When is ETA for ship-it to replace https://svn.mozilla.org/libs/product-details/json/ JSONs? Is there any wikipage to track the current development?

Comment 18

[Sylvestre Ledru [:Sylvestre]]

I think this bug can be finally closed.
For more information: see bug 1288169 and bug 1288171