1200282 - Product details Firefox JSON stuck on 40.0.2

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Michal Stanke (Mozilla.cz) [:mstanke][:MikkCZ]]

Despite https://svn.mozilla.org/libs/product-details/json/firefox_versions.json is OK, https://www.mozilla.org/includes/product-details/json/firefox_versions.json is outdated and contains versions older versions 40.0.2, 38.2.0esr and betas.

Comment 1

[Sylvestre Ledru [:Sylvestre]]

Hmm, we stopped updating the svn external as it is no longer necessary for the download pages.
I fixed the external in r145814. Bug 1146897 will fix that.
Paul, do you know why this wasn't updated?

Comment 2

[Paul [:pmac] McLanahan]

I do. That 2nd URL in comment #0 is a direct link to that SVN external in prod on the PHP side. I had no idea that was available in production, and will be going away very soon (hopefully this quarter). Do we know if people use that? If so we should notify them that it will no longer be available at all very shortly. To my knowledge it is a mistake that anything under "/includes" is available on the web at all.

Comment 3

[Michal Stanke (Mozilla.cz) [:mstanke][:MikkCZ]]

Yes, it is being used, even by me on Mozilla.cz website to automatically obtain latest products versions. You can find more projects on GitHub here https://github.com/search?q=mozilla.org%2Fincludes%2Fproduct-details%2Fjson%2Ffirefox_versions.json&ref=searchresults&type=Code&utf8=%E2%9C%93 using these URLs. Bug 405088 and bug 519124 are also related.

Does it mean I should search for a replacement for these JSONs? Is there even any?

Comment 4

[Paul [:pmac] McLanahan]

You should search for a new source for them. Mozilla.org (and many other django-based sites) gets them from svn via the django-mozilla-product-details package[0]. But if you know the file it's pretty easy to just pull it down via cron or whenever you need.

The other issue is that my understanding is that SVN itself will be going away sometime next year, so beginning to prepare for that would be good as well. The lib above discovers and gets the file via HTTP[1].

If serving these files is a service that we'd like www.mozilla.org to continue to provide, then I can discuss that with the product owner and our team. We do need these data at a reliable location, and the one the site itself uses[1] seems like a good place for now. 

[0] https://pypi.python.org/pypi/django-mozilla-product-details
[1] http://svn.mozilla.org/libs/product-details/json/

Comment 5

[Michal Stanke (Mozilla.cz) [:mstanke][:MikkCZ]]

Well, the django-mozilla-product-details package is out of place. First, it uses svn.mozilla.org, second, we cannot use SVN nor Python. As a community, we have ordinary PHP hosting, nothing more.

So if the JSONs disappear from mozilla.org, the only solution is to use svn.mozilla.org and download the files via HTTP eventhough the server is not designed for that at all.

When shutting down svn.mozilla.org, I hope there will be a replacement (GitHub?) for all the code and files there. Trust me, I would very like to use any other source if more stable, but there is no except svn.mozilla.org. Should I file a separate bug for that?

Comment 6

[Paul [:pmac] McLanahan]

(In reply to Michal Stanke (Mozilla.cz) [:MikkCZ] from comment #5)
> Well, the django-mozilla-product-details package is out of place. First, it
> uses svn.mozilla.org, second, we cannot use SVN nor Python. As a community,
> we have ordinary PHP hosting, nothing more.

That's fine. I just mentioned that in case it'd be helpful to see what other sites are doing. It's pretty easy to sync those files locally via either svn or http requests.
 
> So if the JSONs disappear from mozilla.org, the only solution is to use
> svn.mozilla.org and download the files via HTTP eventhough the server is not
> designed for that at all.

Are you sure? Many of our sites (including www.mozilla.org) do just that. We pull those files via HTTP from svn.m.o every half hour or so.

> When shutting down svn.mozilla.org, I hope there will be a replacement
> (GitHub?) for all the code and files there. Trust me, I would very like to
> use any other source if more stable, but there is no except svn.mozilla.org.
> Should I file a separate bug for that?

There will for sure. We have git.mozilla.org as well as github, but it will be up to the maintainers of the code there to decide where it should live and move it there. There will soon be a much more stable place for the product-details data in the "ship it" project. :sylvestre can tell you more about that. It will (hopefully soon) be the primary source for up-to-date product details information, and will continue to support the same JSON schema currently in the svn repo.

Comment 7

[Michal Stanke (Mozilla.cz) [:mstanke][:MikkCZ]]

Thank you Paul, in that case I will migrate to svn.mozilla.org for now (according to the bugs mentioned above, it was not recommended to use it). I will get in touch with Sylvestre to get more information about ship-it and use the new JSON schema home when it is ready.

Comment 8

[Paul [:pmac] McLanahan]

The bugs say not to request files from there on every page view, but that syncing the data to your servers on a schedule (hourly or so) is fine. I'd agree. As long as you're not requesting those files very often it should be no problem.

Comment 9

[Sylvestre Ledru [:Sylvestre]]

Hmm, please don't switch to svn.mozilla.org, this is not either a proper solution as the svn repo will die. I will make sure that 
Instead, we will be publishing the same json files but generated from ship it database (the tool used to start new builds).

You should cc in bug 1146897 to know more about the updates.

Comment 10

[Michal Stanke (Mozilla.cz) [:mstanke][:MikkCZ]]

OK, I will stay with mozilla.org/includes and use svn.mozilla.org only in case mozilla.org/includes are removed before ship-it JSONs get available.