1288330 - Evaluate: Running fathom locally vs. metadata back-end service

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect, P1)

Description

[Sebastian Kaspari (:sebastian; :pocmo)]

Sooner or later we want to extract metadata and meaning from web pages. Right now there are two obvious choices:

* Running fathom locally
* Querying the page metadata service (which is using fathom internally)

Fathom:
https://github.com/mozilla/fathom

page-metadata-service:
https://github.com/mozilla/page-metadata-service/


* What are the advantages / disadvantages of both approaches?
** CPU, battery consumption, network traffic, online/offline, retry/fallback, more complex rules and classification, visited pages vs. unknown pages
* Are there other (maybe temporary) options (embedly)?
* How much effort is it to implement those solutions?
* Which one can get us implementing features sooner?

Comment 1

[Sebastian Kaspari (:sebastian; :pocmo)]

Hey ahunt, I think you've already started to evaluate some of those things?

Comment 2

[Richard Newman [:rnewman]]

Note that pages behind login/access control — everything from Facebook, to TheClymb and other membership purchase sites, to forums — are hard to process in a hosted service. That alone makes me think that content extraction has to occur on the device to some degree.

Comment 3

[Andrzej Hunt :ahunt]

(In reply to Richard Newman [:rnewman] from comment #2)
> Note that pages behind login/access control — everything from Facebook, to
> TheClymb and other membership purchase sites, to forums — are hard to
> process in a hosted service. That alone makes me think that content
> extraction has to occur on the device to some degree.

I came to a similar conclusion that we should do as much as possible on device, even with just the following reasons (i.e. the login/access control aspect is an additional advantage)

- no additional data usage
-- To be extra clever we should save/cache the page image (which is likely to be the bulk of data usage)
- no need to send history outside of the device
- faster / immediate availability (as opposed to waiting for remote extraction)

We will need a separate solution for unvisited pages (suggestions, and also synced bookmarks), but either way it looks like we definitely need local processing, since the bulk of what we show will be previously visited pages.

(I'm preparing some more detailed notes on all of this, but TL;DR: is local processing is best for most of the content we need, and we probably need some form of hosted service for suggestions.)

Comment 4

[Sebastian Kaspari (:sebastian; :pocmo)]

Backlog until we need it :)

Comment 5

[Sebastian Kaspari (:sebastian; :pocmo)]

Attachment: Bug 1288330 - Extract website metadata and make it available. r?

This patch introduces WebsiteMetadata.jsm which imports fathom and page-metadata-parser.
The code has been slightly modified to not depend on more node libraries.

On DOMContentLoaded the module will extract the metadata asynchronously and send it with
a 'Website:Metadata' event.

Review commit: https://reviewboard.mozilla.org/r/77844/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/77844/

Comment 6

[Sebastian Kaspari (:sebastian; :pocmo)]

I filed new bugs for the first implementation:

* Bug 1301715 - Extracting metadata
* Bug 1301717 - Storing metadata
* Bug 1301718 - Storing the image(s)