1332920 - Allow MAF (Mozilla Archive Format) to be a WebExtension

Status: UNCONFIRMED

(WebExtensions :: Compatibility, task, P5)

Description

[Danny Lin]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0
Build ID: 20161208153507




Expected results:

"We want to reiterate our commitment to our add-on development community, and will work with you in porting extensions, designing new APIs, and creating innovative new add-ons that make Firefox great." (Kev Needham,https://blog.mozilla.org/addons/2015/08/21/the-future-of-developing-firefox-add-ons/).

MAF (https://addons.mozilla.org/en-US/firefox/addon/mozilla-archive-format/) is an addon that can save a web page into a zip pack as .maff and allows Firefox to read them. Currently it cannot be implemented via WebExtension.

Comment 1

[Danny Lin]

APIs we need (and current WebExtension can't do):

1. An API that supports full frame/iframe DOM access (so that MAF can save the current page, including the current DOM content in their frames and/or iframes)

2. An API that can manipulate a zip file (add, update, extract, etc)

3. An API that can save a file to the local disk.

4. An API that can load a file from the local disk.

Comment 2

[Tom S [:evilpie]]

1. Sounds like bug 1214373
2. There are various JS libraries that support zip files, I don't think we need an API for that
3. Either download.downloads or something like https://stackoverflow.com/questions/36139329/how-do-i-programmatically-download-a-file-created-on-the-file-with-a-firefox-web/36151977#36151977
4. <input type=file>

Comment 3

[Andy McKay]

Or also bug 1246236.

Comment 4

[Andy McKay]

Wouldn't just implementing https://developer.chrome.com/extensions/pageCapture be the way to go?

Comment 5

[Danny Lin]

(In reply to Tom Schuster [:evilpie] from comment #2)

What you mentioned does not fit the requirement of MAF.

What MAF works is that:
(1) The user opens a .maff file via Firefox, so that the maff file's location is appeared in the address bar and Firefox knows it.
(2) Firefox blocks the normal file loading flow so that there is not a "Open via" dialog prompt.
(3) Firefox unzips it to a temporarily local disk directory (e.g. the "%temp%" directory in Windows)
(4) Firefox goes in the directory, walk in each sub-directory, and opens the index file defined in the index.rdf.
(5) When the maff tab was closed or when Firefox is shut off, delete the temporarily directory created by (3).
We need a local disk saving at (3) and (4) and a local disk loading at (3) and (5). 

The 4. (or likely, drag and drop API) only allows Firefox to read a file that is specifically picked by the user via <input type=file> or via drag and drop. It cannot programmatically read a specific local file given its "file:" URL address. AJAX approach is also not available since Firefox addons are not allowed to read local files via AJAX.

Likewise, the 3. does not allow Firefox to access the temporarily directory on the filesystem. The saving process of a download via download.downloads is shown in the download bar, and could be interrupted by the user, which are not desired for MAF which attempts to save temp files programmatically and uninterruptibally. As for the approach in the link, it does not work for multiple files.

Comment 6

[avada]

(In reply to Danny Lin from comment #5)
The saving process of a download via download.downloads
> is shown in the download bar, and could be interrupted by the user, which
> are not desired for MAF which attempts to save temp files programmatically
> and uninterruptibally. 

Actually this is how it works now and how it always worked as far as I know. If you save a large enough page you can simply interrupt it, because it takes several seconds.

Though I don't think this is an issue. It might even come handy if you decide you don't want the page.

Comment 7

[avada]

(In reply to Danny Lin from comment #5)
> (In reply to Tom Schuster [:evilpie] from comment #2)
> 
> What you mentioned does not fit the requirement of MAF.
> 
> What MAF works is that:
> (1) The user opens a .maff file via Firefox, so that the maff file's
> location is appeared in the address bar and Firefox knows it.
> (2) Firefox blocks the normal file loading flow so that there is not a "Open
> via" dialog prompt.
> (3) Firefox unzips it to a temporarily local disk directory (e.g. the
> "%temp%" directory in Windows)
> (4) Firefox goes in the directory, walk in each sub-directory, and opens the
> index file defined in the index.rdf.
> (5) When the maff tab was closed or when Firefox is shut off, delete the
> temporarily directory created by (3).
> We need a local disk saving at (3) and (4) and a local disk loading at (3)
> and (5). 
> 
> The 4. (or likely, drag and drop API) only allows Firefox to read a file
> that is specifically picked by the user via <input type=file> or via drag
> and drop. It cannot programmatically read a specific local file given its
> "file:" URL address. AJAX approach is also not available since Firefox
> addons are not allowed to read local files via AJAX.
> 
> Likewise, the 3. does not allow Firefox to access the temporarily directory
> on the filesystem. The saving process of a download via download.downloads
> is shown in the download bar, and could be interrupted by the user, which
> are not desired for MAF which attempts to save temp files programmatically
> and uninterruptibally. As for the approach in the link, it does not work for
> multiple files.

So all of this doesn't seem to be happening. So is there any chance of native support off MAFF/MHTML? ( Bug 1346513 )

Comment 8

[avada]

(In reply to Andy McKay [:andym] from comment #4)
> Wouldn't just implementing
> https://developer.chrome.com/extensions/pageCapture be the way to go?

Firefox doesn't even support viewing MHTML files. So saving them wouldn't help much.

(Used to be the only major browser to lack MHTML support, but now Edge also lacks it, for the time being)

Comment 9

[Ed Morley [:emorley]]

Adding tag so this bug appears on https://arewewebextensionsyet.com/

Comment 10

[timlepes]

If this can't be implemented as a Web Extension, then why not bake the functionality back into Firefox proper? This is a feature that a fair number of people want.  So what does it take to move this "feature" to the other side of the API "wall" and become a feature of Firefox itself?

Comment 11

[timlepes]

More precisely:  Can this bug's Product or Component be updated to make this a feature request for Firefox, or does a separate bug need to be opened to make the request?

Comment 12

[YF (Yang)]

(In reply to timlepes from comment #10)
> If this can't be implemented as a Web Extension, then why not bake the
> functionality back into Firefox proper? This is a feature that a fair number
> of people want.  So what does it take to move this "feature" to the other
> side of the API "wall" and become a feature of Firefox itself?

Priority: -- → P5 = the lowest priority, no resources to implement it, unless the community to make a patch.

> If this can't be implemented as a Web Extension, then why not bake the functionality back into Firefox proper? 

This is Bug 1346513.

Comment 13

[avada]

(In reply to Andy McKay [:andym] from comment #4)
> Wouldn't just implementing
> https://developer.chrome.com/extensions/pageCapture be the way to go?

So, is this happening in the realm of possibility in the near future?
Because one thing's for sure: we'll, be left without the possibility to view save or even view saved archives when FF57 hits.

(BTW it's called page capture. Hopefully that doesn't mean you could only save pages and have to view them in another browser...)

Comment 14

[Andy McKay]

It's not currently on the roadmap for the WebExtensions development team, but contributors patches would be more than welcome.

Comment 15

[Danny Lin]

The addon Web ScrapBook can already capture a web page and save it as a maff file, and view a maff file. However, due to the restriction of WebExt it currently cannot view a local maff file directly (you have to open the viewer page and  pick the maff file), and some feature would not work fine in the viewer.

Comment 16

[avada]

(In reply to Danny Lin from comment #15)
> The addon Web ScrapBook can already capture a web page and save it as a maff
> file, and view a maff file.

Thanks for the tip! Sounds interesting.

> However, due to the restriction of WebExt it
> currently cannot view a local maff file directly (you have to open the
> viewer page and  pick the maff file), and some feature would not work fine
> in the viewer.

That's too bad. Sadly there's nothing for Viewing (much less saving) webpage archives for ff <57 and with multi-process enabled.
Neither of the addons I know of (MAF, UnMHT) were upgraded to work with e10s.

Comment 17

[BugBot [:suhaib / :marco/ :calixte]]

The meta keyword is there, the bug doesn't depend on other bugs and there is no activity for 12 months.
:mixedpuppy, maybe it's time to close this bug?

Comment 18

[John McCormick [:JohnBoyTheGreat]]

The need for native support of archive file extensions in Firefox is huge. Many people want to save web pages for many different reasons. They keep important records, they retain important information, they save things that interest them, they save things that they believe will interest others...the list of reasons is endless.

The only thing that Firefox supports is "Web Page, complete", which is a terrible way to save a web page for most people. Saving this page, for example, creates SIXTY-ONE (61) files and/or folders.

Regular people don't know how to deal with such groups of files so they can send a web page. It should be simple for the average user, not a complex mess.

There are a few extensions ("Save Page WE" and "Singlefile" are the most prominent) which will save single-file web pages. However, this is also a problem, since those pages are saved as "page.html". The average person doesn't know the difference between a single-file HTML and standalone HTML, so they will likely be confused by the difference. Also, users shouldn't have to load up an extension to do something that is an important and common use of a web browser--to save pages offline for future use (i.e., archive them).

As a matter of convenience and to prevent the loss of information when moving these files around, the "Web Page, complete" method is close to useless. It is extremely inconvenient to move hundreds of files and folders to relocate a few web pages. Important files can be easily lost if a user fails to move the folder as well.

Firefox should at the very least support the MHTML/MHT format, since it is used by all major browsers except Firefox (and I don't count anything Mac, so Safari doesn't count).

I also think that Firefox should support the MAFF format which was creatd by Mozilla in the first place! It adds compression to single-file archiving, which is a fantastic idea.

Firefox ought to be MORE compatible with file formats, not less. The only value of Firefox among a field of other browsers is it's feature set. When it has fewer features than the competition, it fails to be useful any longer. I have stayed with Firefox since the very beginning. I like the many extensions which make Firefox an incredible browser! If that feature was dropped, I would jump ship in a heartbeat. (I almost did when they dropped the traditional extension API, which was an incredibly stupid and short-sighted decision. Many of my favorite extensions stopped working and there are no replacements even today.)

Firefox should support:

(1) MAFF
(2) MHTML/MHT
(3) Webarchive
(4) WARC (Internet Archive's Web ARChive format, for entire sites), if possible, perhaps for archiving tab sets as well.
(5) HTMLD (HTML Directory, also for sites), if possible.