Closed Bug 1200291 Opened 4 years ago Closed 4 years ago

Implement simple package registry ("catalog") for downloadable content

Categories

(Firefox for Android :: General, defect)

All
Android
defect
Not set

Tracking

()

RESOLVED FIXED

People

(Reporter: sebastian, Assigned: sebastian)

References

(Blocks 1 open bug)

Details

Quote from bug Bug 1194338:
"Implement a simple package registry on the device. This keeps track of which versions of the fonts are installed, allows us to check hashes, and update when necessary. This will be where we track all DLC; for now just fonts."
Status: NEW → ASSIGNED
Depends on: 1197720
I have been working on a first version of this registry to connect to the download service in bug 1197720. I'd like to discuss some implementation details to make a good decision.

1) At first I wanted to have a local SQLite database to track available and installed assets. But after deciding to download single assets instead of compressed archives (bug 1197720) it felt more compelling to use the filesystem for that (e.g. does the file exist? does it have the correct checksum? -> asset is installed -> else download/replace). Apart from asset deletion this is simple and it avoids inconsistencies between database and local filesystem. It might depend on how much the server side component will do: A "delete asset" instruction could be part of the downloaded asset list too. This heavily depends on how we structure the list of assets and if the server is just a file storage or will have business logic.

2) Currently my implementation pulls the following list for testing: http://people.mozilla.org/~skaspari/packages/catalog
* id: Unique ID to identify the asset
* url: URL to download the asset
* checksum: Checksum of the asset. In this first iteration also used to determine if this version is newer than the local version.
* size: Will be used to decide whether the asset can be stored locally before downloading anything

I have been thinking about more properties that are not used right now. Some make sense, others are nice to have and some are questionable:
* optional: Boolean flag indicating if this is "required" for a browser experience. The idea was to use this to rank assets if there's not enough disk space to download them all. In a later version this could be assets that are only downloaded if confirmed by the user?
* version: (int) This property is more interesting when storing the data in a database and downloading compressed archives of multiple assets because the archive is deleted after extracting (Can't compare checksum of archive). There's the risk of inconsistencies between the version we recorded as installed and the actual file on the filesystem.
* destination: Right now we have only fonts but other assets will need to be copied to different locations. But this might change in future versions so we probably don't want to have this in the list of assets.
* action: Depending on the asset we might need to do different things with it: "copy" to a location, or "extract" to a location. Or even "delete" to remove an asset that is no longer needed.

3) Different assets for different app versions: This is something we don't need now but it's very likely that the list will change based on the app version. This could be part of the asset list and the client decides whether it should download the asset or the server could return a different list for different versions (and the version is in a header or in the url or ..).

4) Adding to 3): Will it be needed to version the schema of the list? .. so that old versions of the app will ignore new asset list versions.

5) For the current set of fonts it makes sense to download them all. But looking at the upcoming assets like dictionaries there will be a decision involved whether the asset is needed for this specific installation. Should this decision be made by the client solely or should we encode this into the asset list? For example:
"requires": {
  "locale": "hu_HU"
}
Flags: needinfo?(rnewman)
3) Different assets for different app versions: This is something we don't need now but it's very likely that the list will change based on the app version. This could be part of the asset list and the client decides whether it should download the asset or the server could return a different list for different versions (and the version is in a header or in the url or ..).

I think a single list would be simpler to manage for both sides, but it means that you pull data you don't really care about. But I don't think it's really a problem.

> 4) Adding to 3): Will it be needed to version the schema of the list? .. so that old versions of the app will ignore new asset list versions.

We usually add a version number in the server's endpoint as a prefix, e.g. /1/ , so as far as the server is concerned it's easy to do. However that leads me to another question : will we need to maintain the older versions assets going forward or just freeze them everytime the schema is changed ? that can add some complexity to the update flow. I'd vote for a freeze if it's doable.
I splitted this bug into two: This one here is tracking the work needed for a local representation of the package registry / catalog. We might use this component in Nightly without the server component. The work needed to download/synchronize/update catalog of assets from the web is now tracked in bug 1201059.
(In reply to Sebastian Kaspari (:sebastian) from comment #1)
> 1) At first I wanted to have a local SQLite database to track available and
> installed assets. But after deciding to download single assets instead of
> compressed archives (bug 1197720) it felt more compelling to use the
> filesystem for that (e.g. does the file exist? does it have the correct
> checksum? -> asset is installed -> else download/replace). Apart from asset
> deletion this is simple and it avoids inconsistencies between database and
> local filesystem. It might depend on how much the server side component will
> do: A "delete asset" instruction could be part of the downloaded asset list
> too. This heavily depends on how we structure the list of assets and if the
> server is just a file storage or will have business logic.

Okay, after actually taking the time to read about and play with Kinto, it doesn't make any sense to use the filesystem only. We want to (partially) download/update the catalog and save the installation state along with it..

(In reply to Sebastian Kaspari (:sebastian) from comment #1)
> * destination: Right now we have only fonts but other assets will need to be
> copied to different locations. But this might change in future versions so
> we probably don't want to have this in the list of assets.
> * action: Depending on the asset we might need to do different things with
> it: "copy" to a location, or "extract" to a location. Or even "delete" to
> remove an asset that is no longer needed.

The "Fennec OTA Updates" document mentions "type" and "kind" and they seem to be much more valuable here than "destination" or "action".
(In reply to Sebastian Kaspari (:sebastian) from comment #4)

> Okay, after actually taking the time to read about and play with Kinto, it
> doesn't make any sense to use the filesystem only. We want to (partially)
> download/update the catalog and save the installation state along with it..

Yes, but bear in mind that we might need a concept of invalidation. If we fail to load an asset that we thought we downloaded, it might be because it was deleted. The filesystem doesn't completely leave the equation. Future.

> The "Fennec OTA Updates" document mentions "type" and "kind" and they seem
> to be much more valuable here than "destination" or "action".

Yeah, this is more "dispatch to the right custom handler" than "define a DSL for processing".
Flags: needinfo?(rnewman)
(In reply to Sebastian Kaspari (:sebastian) from comment #1)
> I have been working on a first version of this registry to connect to the
> download service in bug 1197720. I'd like to discuss some implementation
> details to make a good decision.
> 
> 1) At first I wanted to have a local SQLite database to track available and
> installed assets.

How about in-memory data that we flush to a JSON file? No need for SQL here.

(See e.g., the ProfileInformationCache.)

> it felt more compelling to use the
> filesystem for that…

We'll need a catalog regardless. I think a verification step will be needed, and that will necessarily touch the filesystem, but at the very least we need some place to record "yes, I intend to have this package installed".


> * optional: Boolean flag indicating if this is "required" for a browser
> experience. The idea was to use this to rank assets if there's not enough
> disk space to download them all. In a later version this could be assets
> that are only downloaded if confirmed by the user?

Isn't this implied by the 'kind' and application context? That is: the locale files for your current locale are required, hyphenation dictionaries never are, fonts never are.

And the error case needs further (subsequent) thought. If we can't download the files due to lack of space, we have more serious worries. If they keep getting deleted, we shouldn't re-download them. Etc.


> 3) Different assets for different app versions: This is something we don't
> need now but it's very likely that the list will change based on the app
> version. This could be part of the asset list and the client decides whether
> it should download the asset or the server could return a different list for
> different versions (and the version is in a header or in the url or ..).

This is really about compatibility, and we will definitely need to address this if we ever get to locales.


> 4) Adding to 3): Will it be needed to version the schema of the list? .. so
> that old versions of the app will ignore new asset list versions.

Always version formats and protocols.


> 5) For the current set of fonts it makes sense to download them all. But
> looking at the upcoming assets like dictionaries there will be a decision
> involved whether the asset is needed for this specific installation.

I don't think we need to capture dependencies between assets, and I don't think the manifest itself should decide what's required. All of these are decided by the client code itself.
Summary: Implement simple package registry for downloadable content → Implement simple package registry ("catalog") for downloadable content
This has been implemented as part of bug 1197720. The client to update the catalog is going to be implemented in bug 1201059.
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.