1278677 - Use Embed.ly for temporary metadata collection

Status: RESOLVED FIXED

(Hello (Loop) :: General, defect, P1)

Description

[Ian Bicking (:ianbicking)]

Until we get some of the other metadata stuff fully stood up, it will be nice to see a table of contents with some good data in it.

The easiest way is to use the Embed.ly API to get some data.  This should be done on PAGE ADD TIME (not at ToC render time).  You'll make a request to:

  GET https://api.embedly.com/1/extract?url={url}&key={api_key}

API key has been provided to some people on the team via email.

I propose we hardcode the API key in the client for now.  It has a fairly low limit (5000 requests per month), but this is a very temporary approach.

Note then that when you add a page you'll do something like:

function getPageMetadata(url) {
  return new Promise((resolve, reject) => {
    let result = {url: url};
    let req = new XMLHttpRequest();
    req.open("GET", `https://api.embedly.com/1/extract?url=${encodeURIComponent(url)}&key=${encodeURIComponent(api_key)}`);
    req.onload = function () {
      if (req.status != 200) {
        reject(req);
        return;
      }
      let extracted = JSON.parse(req.responseText);
      result.title = extracted.title;
      result.images = extracted.images;
      result.favicon_url = extracted.favicon_url;
      //maybe result.embedly = extracted ?
      resolve(result);
    };
  });
}

Docs are here: http://docs.embed.ly/docs/extract

The .images property is fairly rich, and some stuff (like entropy) we're unlikely to do ourselves, but we can keep it for now.

Comment 1

[GitHub Autolander Bot]

Attachment: [loop] mancas:bug1278677 > mozilla:akita

Comment 2

[Fernando Campo (:fcampo)]

Comment on attachment 8761121 [details] [review]
[loop] mancas:bug1278677 > mozilla:akita

left couple of questions on GH. code looks good, needs error treatment and tests. 

Or maybe we can avoid it, being a temporary solution and close deadline (although I'm afraid we will forget about it when exchanging for final solution if we don't make them now)

Comment 3

[Adam Roach [:abr]]

Ian -- if I read this correctly, we are planning to send every URL visited by Hello users to embedly as they visit them. Is this right?

Comment 4

[Mark Banner (:standard8)]

(In reply to Adam Roach [:abr] from comment #3)
> Ian -- if I read this correctly, we are planning to send every URL visited
> by Hello users to embedly as they visit them. Is this right?

That's correct, however this is temporary on our Akita branch and only going to be used for demos/initial development - it won't be released to users (for which we're planning alternate options).

Comment 5

[Dan Mosedale (:dmosedale, :dmose)]

I'd say just add an XXXakita comment with a bug number for writing the tests and error handling.

Comment 6

[Dan Mosedale (:dmosedale, :dmose)]

Comment on attachment 8761121 [details] [review]
[loop] mancas:bug1278677 > mozilla:akita

At this point, this wants Fernando's questions answered and probably just a spin-off bug filed before a final review/landing.

Comment 7

[Manuel Casas Barrado [:mancas]]

Attachment: Snackbar demo

Comment 8

[Fernando Campo (:fcampo)]

Comment on attachment 8761121 [details] [review]
[loop] mancas:bug1278677 > mozilla:akita

Thanks for adding the error handling.
And cool feature with the snackbar :) Let's wait for UX approval before merging it

Comment 9

[Pau Masiá [:Pau]]

Comment on attachment 8761515 [details]
Snackbar demo

It's a good start. Thanks, Manu!

Comment 10

[Manuel Casas Barrado [:mancas]]

Landed in akita: https://github.com/mozilla/loop/commit/aeb3cf69d5ff1b74ad21056257d902141c0ca38f