1265511 - Link clickers can submit URLs to a metadata extraction server to add to room TOC

Status: RESOLVED INCOMPLETE

(Hello (Loop) :: Client, defect, P2)

Description

[Chris Rafuse :crafuse]

Link clicker adds URL to room through form or data submission.   Server to respond with fetched URL page meta-data to be displayed in about:TOC page.

Flow

Link clicker clicks on Room URL link and supported browser initializes room controls and current state.

Link clicker hits [+], enter a URL: http://example.com

Link clicker hits submit button to add the page. Sends request to server to add the URL to the room.

Server response sends extracted URL meta-data to link clicker and generator if connected( part of room data when connected next).

Link clicker's Add URL request response is added to the TOC data and page tile.


Requirements
Server needs to identify link clicker connected to roomToken through public channels.
-or- 
A WebRTC connection be established to make the secure transaction.

Link clicker has established secure connection and/or encrypted session with server.

Meta-data extraction server needs access to Internet sites for meta data extraction.

Design
Link Clicker Add Url to Room Client
Room initiation with roomToken and secure connection is established and assumed to be available.   

On click event within the TOC add site URL to room, sends request to server for response meta-data and confirmation that the site was added.
GET request format:
 https://someservice.hello.firefox.com/summarize?url={url}

PUT format:
PUT firebase/?orderBy=”date”&equalTo=url
{url: “http://dronewarehouse.ca/?target=main” }

Site Meta-Data Extraction
Modules that handle site meta-data property extraction:
	[og:]title or similar title categories (ie. Twitter:title)
	[og:]description 
	og:image  - may not be present – find image or use title 

future module support cases or other data capture values
	og:video
	og:url
	og:site_name
	og:locale
	og:type

Sample of meta data properties

…
  <base href="http://dronewarehouse.ca/" />
  <link rel="canonical" href="http://dronewarehouse.ca/" />
…
<link rel="shortcut icon"  href="images/simplecms/favicon.ico" type="image/x-icon" />
...
<meta property="og:title" content="Drone Warehouse Canada" />
<meta property="og:type" content="website" />
<meta property="og:url" content="http://dronewarehouse.ca/" />
<meta property="og:site_name" content="Drone WareHouse" />
<meta property="og:description" content="We are authorized to resell the following:DJI - Yuneec - 3DRobotics - Lumenier - Storm - More Coming Soon
Shipping free for orders over $500. We ship from Canada!(Prices are in USD as default - Choose CAD for canadian pricing at top menu)" />
<meta property="og:locale" content="en_US" />
<meta property="fb:app_id" content="480900238768525" /><!-- Google Analytics -->

Meta-tractor
The meta-extractor service or server would need exclusive access to loop server to responsive and out of channel of other loop work loads.
An asynchronous server would need to wait for URLs to respond and return results of other URL meta-data extraction requests.   

Example (different use case) of parsing body:contentcheck.js (uses Domino); PageMetadata query: PageMetadata.jsm#93

Persist meta-data to room remote storage and optionally capture anonymous URL usage.

Response(s) for request of add URL to room
If link generator is active in or connected to the same room, update the generator's local room data with newly added URL. 

Link clicker's response returns JSON data or other object representational data formats.
Example of URL meta-data payload:
{
	url: “https://whispersystems.org/blog/whatsapp-complete/”, 
	title: “WhatsApp's Signal Protocol integration is now complete”, 
	image: [{url: “https://whispersystems.org/blog/images/whatsappcomplete2.png”}]
}
Other properties may be added to increase clarity and value of TOC URL tile to the user.


Link clicker's Add URL response is added to the TOC and meta-data is displayed in the site tile.

Comment 1

[Chris Rafuse :crafuse]

Attachment: Loop server PR on Github

in progress working review to validate approach.

This should retrieve the page data and start searching the content for meta and links.

timeout issue with some sites:
Use HTTPs Requester GET
http://localhost:5000/meta-data/https%3A%2F%2Fwww.mozilla.org/

will return page data after prompt or error has returned on console.

The Async should be waterfall or serial but seems it is not implemented.  Node 4 maybe needed for support?  

Does this make sense for handling a number of meta-data extraction requests?

And as a blocker, I could not pass data between the async functions so moved it to the first function(finder).  The pagemetadata module gives a standard metadata output, still working on the output details.  getAttribute in getLinks function does not work as it maybe from the FF jsm code this function came from.

Just need some validation on this approach.

Comment 2

[Chris Rafuse :crafuse]

Worked it out and now formatting json output with typed meta data objects and required property values.  Image output still needs work. use og:image if defined or use screen shot service/package.

Comment 3

[Mark Banner (:standard8)]

Comment on attachment 8758058 [details] [review]
Loop server PR on Github

Not intending on looking at this just at the moment.

Comment 4

[Mark Banner (:standard8)]

Support for Hello/Loop has been discontinued.

https://support.mozilla.org/kb/hello-status

Hence closing the old bugs. Thank you for your support.