1882524 - Improve page text extraction

Status: RESOLVED DUPLICATE of bug 1990609

(Core :: Machine Learning: General, enhancement)

Description

[Tarek Ziadé (:tarek)]

The current Readability lib in Firefox will extract text that includes headers and footers, which leads to a lot of noise when you want to perform inference on the web page

example (try reader mode) : https://edition.cnn.com/2023/09/27/australia/australia-seismic-blasting-whales-climate-intl-hnk/index.html

let's see if we can improve this

prior art: https://github.com/mozilla/fathom

Comment 1

[Tarek Ziadé (:tarek)]

See https://github.com/mozilla/readability/issues/89

Comment 2

[Tarek Ziadé (:tarek)]

Looks like CNN implements the article schema markup.

this snippet works on that page to extract the textual content:

const jsonLdScripts = document.querySelectorAll('script[type="application/ld+json"]');


jsonLdScripts.forEach(script => {
  try {
    const data = JSON.parse(script.innerText);
    console.log(data["articleBody"]);
  } catch (error) {
    //
  }
});

I suppose this could be leveraged by Readability to remove the clutter, as any text not in this article body should probably not be displayed.

Comment 3

[Tarek Ziadé (:tarek)]

Readability is already parsing that structure but does not grab that specific field in its metadata

added https://github.com/mozilla/readability/issues/846

Comment 4

[Greg Tatum [:gregtatum]]

We have the page extractor component now, let's track work there.