877526 - [osumo] [tracker] Add API to kitsune's wiki package in order to get data out from SUMO

Status: RESOLVED FIXED

(support.mozilla.org :: General, defect)

Description

[Shuhao]

An API needs to be created in the kitsune project in order to get the data out of SUMO to be downloaded. Some initial ideas are:

 - Bundles (all articles) are generated and rendered every <day/week/month?>
 - Diffs are generated for smaller time frame (day, per edit, etc.)
 - API to download the latest bundles to build initial offline caches
 - API to download diffs for updating offline caches

Since we cannot reasonably store bundles and diffs from all the generations indefinitely, we should only store the latest bundles and diffs that goes up to a certain time. If the user's local content is too outdated[1], we replace it with a new bundle.

[1]: The term "too outdated" is loosely defined. I think we should take a look at the amount that the KB has changed rather than just setting a timeout.

More comments to come if necessary.

Comment 1

[Shuhao]

A good point is offline images.

While at this point I'm not sure how big all the images are, it is probably quite big. The user should be able to opt out of downloading these images or have some sort of click to download (or queued to be downloaded when there is network) functionality (which means we need the correct sizes to be downloaded with the API).

Comment 2

[Shuhao]

Here's the client side database format. The web api should return all of this to avoid any computations client side:

    https://github.com/shuhaowu/osumo/blob/master/docs/database.md

Comment 3

[Shuhao]

This work is currently in progress and can be viewed at:

    https://github.com/shuhaowu/kitsune/tree/offline-sumo

Comment 4

[Shuhao]

Merged here: https://github.com/mozilla/kitsune/commit/cf8ccaa583b258a038f68ee9140a45b2f834988a