873190 - Plan and document a bulk export dump

Status: VERIFIED FIXED

(Marketplace Graveyard :: API, defect, P1)

Description

[Wil Clouser [:clouserw]]

Our API is a great resource for quickly getting up to date information for our app catalog.  However, for some applications (eg. search engines), it's more useful to get all the data than it is to have to up to the minute.  Continuously scraping our paginated API is time consuming and costly.

In the tradition of fine marketplaces everywhere we'd like to create a curated dump of our entire app catalog and make it available to partners or other folks who are interested in manipulating our data in bulk.

The best example I know of is Apple's Affiliate Resources (http://www.apple.com/itunes/affiliates/resources/documentation/itunes-enterprise-partner-feed.html).  That page explains what it is, has schema diagrams, and details all the fields - perfect.

This bug is about writing our own equivalent document.  For v1 I'd like to, at a minimum, get Apps, Ratings, and Categories into the doc, but I haven't thought about it past that - no doubt there are other great things to add too.  It doesn't need 100% of everything right now, we gotta ship, but let's get a good set.

I'd like to get daily dumps into a single JSON file.  Let's hear more thoughts on this as we go.

Comment 2

[Andy McKay]

https://github.com/mozilla/zamboni/commit/a1f02c

I just noticed "into a single JSON file" I don't think that's a great idea for when the data gets really, really big its easy to iterate over files. So this gives all the data as a tarball people can access.

Comment 3

[Wil Clouser [:clouserw]]

This links to -dev instead of prod.  A mistake?

Comment 4

[Andy McKay]

Won't be on prod until the push, then I'll add links for prod to the docs.

Comment 5

[Wil Clouser [:clouserw]]

CCing e.me.  This exist on -dev at the moment, will be in production at the end of this week.

http://firefox-marketplace-api.readthedocs.org/en/latest/topics/export.html

Comment 6

[nati]

Thanks!

(In reply to Wil Clouser [:clouserw] from comment #5)
> CCing e.me.  This exist on -dev at the moment, will be in production at the
> end of this week.
> 
> http://firefox-marketplace-api.readthedocs.org/en/latest/topics/export.html

Comment 7

[krupa raj[:krupa]]

The link to the development server in the docs gives me  a 404

I tried https://marketplace-dev-cdn.allizom.org/dumped-apps/tarballs/2013-05-31.tgz

What am i doing wrong?

Comment 8

[nati]

"will be in production at the end of this week."

(In reply to krupa raj 82[:krupa] from comment #7)
> The link to the development server in the docs gives me  a 404
> 
> I tried
> https://marketplace-dev-cdn.allizom.org/dumped-apps/tarballs/2013-05-31.tgz
> 
> What am i doing wrong?

Comment 9

[Wil Clouser [:clouserw]]

(In reply to krupa raj 82[:krupa] from comment #7)
> The link to the development server in the docs gives me  a 404
> 
> I tried
> https://marketplace-dev-cdn.allizom.org/dumped-apps/tarballs/2013-05-31.tgz
> 
> What am i doing wrong?

That's tomorrow.  Try yesterday's date.

Comment 10

[krupa raj[:krupa]]

awesome! Also, i was completely convinced yesterday was 05/31 :)

verified by downloading https://marketplace-dev-cdn.allizom.org/dumped-apps/tarballs/2013-05-30.tgz

Comment 11

[Wil Clouser [:clouserw]]

In production: https://marketplace.cdn.mozilla.net/dumped-apps/tarballs/2013-05-30.tgz

Is that going to work for importing into e.me's index?

Comment 12

[nati]

That should do, but if you could link e.g. current.tgz to the latest tgz, that would be awesome.

Thanks,
Nati

Comment 13

[Wil Clouser [:clouserw]]

Cool, I filed bug 878881 for that.  If you run into any bugs or need more/different info in the dumps just file another bug for us and we'll fix it up.  Cheers.