1297703 - Wrap page-metadata-parser/fathom.js in WKUserScript for extracting metadata from web pages

Status: RESOLVED FIXED

(Firefox for iOS :: General, defect, P1)

Description

[Stephan Leroux [:sleroux]]

Now that we know that we can run fathom.js locally by injecting it into the web context, we can start expanding on this to build a local metadata parser on iOS.

By using webpack, we're able to inline all of the node dependencies that fathom.js requires [1]. This inline also removes any traces of ES6 syntax - allowing the code to run in iOS's older Javascript context.

The next step will be to pull in :jkerim's page-metadata-parser ruleset that sits on top of fathom which defines extraction rules [2]. Combining these two should allow us to extract the meta data needed for Activity Stream uses.

We'll also need to load the Javascript through a WKUserScript object in a similar way we handle extracting favicons. By doing so, we'll be able to pass the extracted metadata from the web context into the native layer for storage and rendering on the A-S panel. 

At this point, there are a few questions:

1. How do we want to persist this metadata? Should this live in our version of the places DB or in a separate cache?

2. Related to 1., how would we modify our existing queries and data layer to mix in meta data content for rendering top sites/highlights?

3. How does inlining all of the node/ES6 dependencies effect performance on page load? Webpacking fathom.js spits out 25k lines of Javascript which sounds like a lot to add on top of every page.

Finally, we've discussed whether we want to do this processing locally or through a web service. Considering that desktop has hopes of running it locally and the network constraints on mobile, running the extraction logic locally would be ideal.

[1] https://github.com/mozilla/fathom/pull/24
[2] https://github.com/mozilla/page-metadata-parser

Comment 1

[Stephan Leroux [:sleroux]]

Attachment: Link to Github pull-request: https://github.com/mozilla/firefox-ios/pull/2093#attch-to-bugzilla

Broke out my DB work for this patch and kept this PR to just including the metadata parser and adding the plumbing in place. If you want to see the metadata being generated, place a breakpoint on the TODO line in Profile.swift!

Comment 2

[Brian Nicholson (:bnicholson)]

Comment on attachment 8791720 [details]
Link to Github pull-request: https://github.com/mozilla/firefox-ios/pull/2093#attch-to-bugzilla

LGTM!

Comment 4

[Stephan Leroux [:sleroux]]

master https://github.com/mozilla/firefox-ios/commit/6e6fffb80d6c345c699f689add717831443a398d