1140538 - publish JSON feed of amo contents

Status: RESOLVED WORKSFORME

(addons.mozilla.org Graveyard :: Public Pages, defect)

Description

[Dave Herman [:dherman]]

As mentioned in bug 1074960 comment 38, it would be good to have a standard published feed of the current contents of amo. In particular my `samo` tool needs to extract an amo index:

    http://github.com/dherman/amo

Thanks,
Dave

Comment 1

[Kris Maglione [:kmag]]

If that tool currently only searches for specific strings, is there some reason you can't use the add-ons MXR tree?

Comment 2

[Andy McKay]

"Current contents" could cover rather a lot of things. We are rewriting AMO with an API in mind and if you are looking for add-ons data, as Kris says, using MXR makes sense.

Closing based on those changes unless you've got more feedback about what you are after.

Comment 3

[Kris Maglione [:kmag]]

The Cuban community has been maintaining a mirror of the public contents of AMO for users whose bandwidth constraints prevent them from accessing them directly.

When we stopped mirroring our CDN data to FTP, I started manually maintaining a list of the latest public versions of add-ons and their download URLs. Since we migrated to AWS, I can't continue to update that list, so it would be really helpful if we could provide the data directly from AMO instead.

Something as simple as a cached JSON feed, updated once a day-ish, along the following lines should do:

{
  "addons": {
    "<slug>": {
      "status": "<status>",
      "url": "<url>",
      "created": "<creation-time>",
      "versions": {
        "<latest-version-number>": {
          "created": "<creation-time>",
          "files": [
            {
              "created": "<creation-time>",
              "status": "<status>",
              "platform": "<platform>",
              "filename": "<filename>",
              "download_url": "<download-url>"
            },
            ...
          ]
        }
      }
    },
    ...
  }
}


I was using something along the lines of the following to update the old list:

SELECT
    a.id AS addon_id,
    CONCAT('https://addons.mozilla.org/addon/', a.slug, '/') AS addon_url,
    f.created AS file_created,
    CASE f.platform_id
    WHEN 0 THEN 'Any'
    WHEN 1 THEN 'All'
    WHEN 2 THEN 'Linux'
    WHEN 3 THEN 'OS-X'
    WHEN 4 THEN 'BSD'
    WHEN 5 THEN 'WinNT'
    WHEN 6 THEN 'Solaris'
    WHEN 7 THEN 'Android'
    WHEN 8 THEN 'Maemo'
    ELSE '?'
    END AS file_platform,
    CONCAT('https://addons.cdn.mozilla.net/user-media/addons/', a.id, '/', f.filename) AS file_url
FROM
    addons a
INNER JOIN
    versions v ON a.current_version = v.id
INNER JOIN
    files f ON f.version_id = v.id
WHERE
    a.status IN (4, 8, 9) AND
    NOT a.inactive AND
    a.is_listed AND
    ((a.status IN (8, 9) AND f.status = 8) OR
     (a.status = 4 AND f.status = 4))
ORDER BY
    f.created DESC
;

But obviously I'd prefer an ORM-based solution for AMO code.

Comment 4

[Andy McKay]

There's an API: https://developer.mozilla.org/en-US/Add-ons/AMO/Add-ons_manager_API, there's are multiple RSS feeds: https://addons.mozilla.org/en-US/firefox/extensions/format:rss?sort=users - do those not provide the information needed?

Comment 5

[Kris Maglione [:kmag]]

I don't think that either of those allow someone to get a list of all of the add-ons on AMO. The add-on manager API definitely only returns add-ons with full review, and I'm pretty sure they both limit the number of results they'll return.

Comment 6

[Yunier J.]

I see the solutions that @andym says but it not solved my problem beacuse the API only show the firts 10 approbed addo-ns, and the RSS channel.
The API provide the data that I need but with few addons.

I think that idea of @kmag is the best because return all data that I need, and I can develop an advanced system that update add-ons and let the WP installation.

Comment 7

[Yunier J.]

Hi friends, it would be possible?
The timeline for signed add-ons has been scheduled to Firefox 46.

Comment 8

[Andy McKay]

We'll do this through the new API as noted in this bug on github: https://github.com/mozilla/addons-server/issues/1878.

Closing this issue, follow the github issue for more information. Making it worksforme because there isn't a resolved gofollowitongithub status.

Comment 9

[Andy McKay]

Noting for the author and others that the APIs have landed and are now accessible, you should be able to search and paginate through all the add-ons on AMO to build out a list.