Closed Bug 405088 Opened 17 years ago Closed 14 years ago

Create latest mozilla products feed

Categories

(Websites :: other.mozilla.org, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: julenx, Assigned: wenzel)

References

Details

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; eu; rv:1.9b2pre) Gecko/2007112204 Minefield/3.0b2pre
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; eu; rv:1.9b2pre) Gecko/2007112204 Minefield/3.0b2pre

Mozilla should maintain a general feed with latest version information of its products. For example:

<versions>
  <product>
    <name>Firefox</name>
    <stable>2.0.0.9</stable>
    <development>3.0b1</development>
  </product>
  ...
</versions>

Thus, localizers could provide users the latest product information in their websites at the moment.

Reproducible: Always
There is some discussion about how such a data source should look like in bug 519124. There are various ideas about content and format, but the general agreement seems to be that there should be one source per product. So this bug should probably be moved to the mozilla.com component for Firefox with other bugs for other products.
From the discussion in bug 519124, the idea is to have a per-branch (or per-product) JSON file with latest data in the format like this:

{
  ab-CD":{
    "en-name":"Name",
    "native-name":"Native-name",
    "version":"3.5.3",
    "download":{
      "win":"http://download.mozilla.org/?product=firefox-3.5.3&os=win&lang=ab-CD"
      "osx":"http://download.mozilla.org/?product=firefox-3.5.3&os=osx&lang=ab-CD"
      "linux":"http://download.mozilla.org/?product=firefox-3.5.3&os=linux&lang=ab-CD"
    }
  }
}

We could then allow narrowing down data transfer by querying for specific set of locales like:

firefox-beta.json?locales=pl,da

This structure will allow local communities to link to latest product versions.

Opinions?
Assignee: nobody → gandalf
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Any chance to get the development builds into there? This would be useful for pointing users to the latest betas and nightly builds, so our chances to get more testers grow.
Julen: yes. My intention is to have a feed per branch firefox-stable.json, firefox-beta.json, firefox-nightly.json

Something like that.
Wil: who's the right person from the webdev team to CC here?
(In reply to comment #5)
> Wil: who's the right person from the webdev team to CC here?

morgamic
Zbigniew, Wil, morgamic: any news regarding this? Community websites are eagerly waiting for productinfo feeds to become available...
We'll set up a meeting to talk about what we can do here.  Hardest part is finding the data source.  I'd be okay with either bootstrapping the build system or just having an external process that watches the rsync image.
can we reuse what we use for mozilla.com?
Possibly, but even that should be using the same feed that we're asking for in this bug so we don't have to manually update available versions.  I'd like for data manipulation to occur on by one system (build) and for all other consumers of that data have an approve-then-release cycle.  Right now we all enter the data in different places.

There was some discussion in bug 372746 about this as well, with a simple diagram of what could benefit from this service in attachment 400062 [details].
Is there an ETA on that bug? It is quite old. For this feed, I don't really care if I get the data from mozilla.com or from the same source as mozilla.com gets it from. And it is irrelevant for me if mozilla.com gets its data by manual entry or an automatic feed, as long as I have the same data as mozilla.com. If it takes a long time to do the changes as bug 372746 describes on mozilla.com, could we then instead just have a machine readable version of the page, which already exists on mozilla.com?
(In reply to comment #12)
I have considered using the product-details lib before, but someone (from webdev I guess) said in a newsgroup (mozilla.dev.l10n or mozilla.dev.l10n.web) that the SVN server where the files are hosted might not be up to the traffic if community sites started requesting the SVN server on each request a user made to a community site, and therefore I dropped it, as I didn't want to risk DOS'ing SVN.
Are you sure that this bug should be marked as WORKSFORME? We need stable source. Is this source stable? I need guarantee that this source will be available in the future.
Do not use the feed from svn.mozilla.org, as this machine is dedicated for version control and not for automated web requests. The mozilla.com address should be better as it implement some caching and distributing mechanisms, although I will prefer to get us officially permitted to use this source.
What about SeaMonkey and Calendar products?
There is no support for l10n versions which should be the key feature of such feed.
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Yep, that too... Though it doesn't bother me too much as long as my locale is in sync with en-US.
(In reply to comment #17)
> What about SeaMonkey and Calendar products?

I talked with Robert Kaiser about something like this for SeaMonkey. He said that he need to know which format they should offer. 

(In reply to comment #18)
> There is no support for l10n versions which should be the key feature of such
> feed.

I think info about actual version is good enough for us. Based on this information we can create download links and link to release notes.
Ok, I see resolving this bug before we have confirmation that the source is stable was premature. Lets ask.

(In reply to comment #18)
> There is no support for l10n versions which should be the key feature of such
> feed.

I believe that would be *_primary_builds.json, which is the file I am using as of today on mozilladanmark.dk.
Status: REOPENED → NEW
(In reply to comment #14)
> Tomer Cohen wrote in
> http://groups.google.com/group/mozilla-community-sites/msg/d40846c447057832
> that the requested feed already exists.

Yea, you should probably neither hit the svn web heads to update the JSON files (they are not set up for serious traffic), nor use undocumented files you find on the mozilla{,messaging}.com web sites (they can break at any time).

However, it should probably be easy to set up a site somewhere (product-details.mozilla.org, maybe?) that will periodically SVN up and offer the JSON feeds for public consumption. Also, we should make a small set of drop-in APIs (for PHP, Python) that'll conveniently consume these feeds.

clouserw, what do you think?
(In reply to comment #22)
> (In reply to comment #14)
> > Tomer Cohen wrote in
> > http://groups.google.com/group/mozilla-community-sites/msg/d40846c447057832
> > that the requested feed already exists.
> 
> Yea, you should probably neither hit the svn web heads to update the JSON files
> (they are not set up for serious traffic), nor use undocumented files you find
> on the mozilla{,messaging}.com web sites (they can break at any time).
> 
> However, it should probably be easy to set up a site somewhere
> (product-details.mozilla.org, maybe?) that will periodically SVN up and offer
> the JSON feeds for public consumption. Also, we should make a small set of
> drop-in APIs (for PHP, Python) that'll conveniently consume these feeds.
> 
> clouserw, what do you think?

I'd rather have a real app that interacts with the build servers to get numbers but if this is a high priority it'd work.  We could have it build the JSON files itself, or even have hudson build them and commit them back to SVN.
(In reply to comment #24)
From reading your link it seems to only be some Python code, and it still uses http://svn.mozilla.org/libs/product-details/json/

Does that mean there was a policy change? Are we now allowed to do HTTP requests to the SVN server on a regular basis? For example every 5 minutes, or maybe even on each pageview? Or are we required to do the HEAD+GET based caching which you describe? I would rather avoid doing any caching at all, as that involves storing something somewhere on the webserver.

By the way Python code does not help much if you are on a webhotel only supporting PHP, and you are using it for e.g. WordPress, which is also written in PHP.
As a side note, http://www.seamonkey-project.org/seamonkey_versions.json has been existing for a bit mirroring the Firefox and Thunderbird versions mentioned here and providing the most-current SeaMonkey versions.
(In reply to comment #25)
> (In reply to comment #24)
> From reading your link it seems to only be some Python code, and it still uses
> http://svn.mozilla.org/libs/product-details/json/

That is true.

> Does that mean there was a policy change? Are we now allowed to do HTTP
> requests to the SVN server on a regular basis? For example every 5 minutes, or
> maybe even on each pageview? Or are we required to do the HEAD+GET based
> caching which you describe?

I suggest so. Refreshing frequently is fine (though I can't imagine an application that needs 5-minute refresh time of this data, except maybe mozilla.com). Refreshing on every page view, however, is a really bad idea, for all people involved.

> I would rather avoid doing any caching at all, as
> that involves storing something somewhere on the webserver.

Don't do that. Ever. Except in the rare instance when the remote solution is intended to be a high-availability resource (Google's AJAX libs, for example: http://code.google.com/apis/ajaxlibs/), you shouldn't delay the user's experience to wait for remote data, especially not data that changes every few weeks at best.

> By the way Python code does not help much if you are on a webhotel only
> supporting PHP, and you are using it for e.g. WordPress, which is also written
> in PHP.

That is true. I did, however, show a way to do it, which for example should enable someone to write a Wordpress extension with similar functionality. To my knowledge, Wordpress has a "cron" functionality that could be used to mimick what I've been doing in Python.
(In reply to comment #26)
> As a side note, http://www.seamonkey-project.org/seamonkey_versions.json has
> been existing for a bit mirroring the Firefox and Thunderbird versions
> mentioned here and providing the most-current SeaMonkey versions.

This is great. I'll add that to the product-details Readme file. Is there an explanatory page I should link to, or should I just drop the json link in there?
So if we are allowed to download http://svn.mozilla.org/libs/product-details/json/
at a regular cron-like interval (through not on every page view), I would say this bug is fixed, isn't it?
What about links in Comment #14? I suppose these are preferred over SVN, aren't they?
(In reply to comment #29)
> So if we are allowed to download
> http://svn.mozilla.org/libs/product-details/json/
> at a regular cron-like interval (through not on every page view), I would say
> this bug is fixed, isn't it?

I suppose so, unless there's anything else you are looking for at the moment?

I talked to IT about it shortly, and as long as you are considerate about the amount of updates you run, pulling from SVN is not a big deal. If you use an SVN checkout and drop ``svn up`` into your crontab, or if you use a library like the one I wrote, is up to you.

If anyone feels inclined to write a Wordpress plugin, by the way, I think everybody here would like them to share it.

(In reply to comment #30)
> What about links in Comment #14? I suppose these are preferred over SVN, aren't
> they?

Yes. (Though that still doesn't mean you should include these links directly into Client-side JavaScript that'll be pulled by every visitor of your site.)
Calling this fixed, but please reopen if we missed something here.
Assignee: gandalf → fwenzel
Status: NEW → RESOLVED
Closed: 14 years ago14 years ago
Resolution: --- → FIXED
I am not sure I fully understand. I currently use this code:

$time = @filemtime('./cached/firefox_primary_builds.json');
if (!$time || $time < time() - 5*60) {
    $data = file_get_contents('http://www.mozilla.com/includes/product-details/json/firefox_primary_builds.json');
    if ($data) {
        file_put_contents('./cached/firefox_primary_builds.json', $data);
    }
}

Is there anything I chould change? Should I use the SVN url instead?
(In reply to comment #28)
> (In reply to comment #26)
> > As a side note, http://www.seamonkey-project.org/seamonkey_versions.json has
> > been existing for a bit mirroring the Firefox and Thunderbird versions
> > mentioned here and providing the most-current SeaMonkey versions.
> 
> This is great. I'll add that to the product-details Readme file. Is there an
> explanatory page I should link to, or should I just drop the json link in
> there?

Just drop the link there, it's both modeled after those I found for Firefox and Thunderbird and, I think, pretty self-explanatory.
r68194.

(In reply to comment #33)
> Is there anything I chould change? Should I use the SVN url instead?

Hm, it seems that in PHP you'd need to use fsockopen() to send If-Modified-Since headers. As long as you are aware that when mozilla.com changes, you'll need to change your code, I would probably not worry about it.

Though I'd say every five minutes is probably a little on the frequent side: If you only did it every hour, you'd probably still be fine. But that's up to you.
(In reply to comment #35)
> Though I'd say every five minutes is probably a little on the frequent side: If
> you only did it every hour, you'd probably still be fine. But that's up to you.

I have made it a little less frequent, but not an hour. Previously when we updated it manually, it usually didn't take more than half an hour for some of our community members to mail me that I needed to update our links.
You need to log in before you can comment on or make changes to this bug.