Open Bug 1246236 Opened 8 years ago Updated 3 months ago

Implement local filesystem read/write access.

Categories

(WebExtensions :: General, task, P3)

Product:
WebExtensions
Component:
General
Type:
task

Tracking

(Not tracked)

People

(Reporter: billjm05, Unassigned)

References

(Blocks 3 open bugs)

Details

(Whiteboard: [design-decision-approved] triaged) 

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36

Steps to reproduce:

N/A


Actual results:

N/A


Expected results:

N/A
Status: UNCONFIRMED → NEW
Ever confirmed: true
See Also: → 1215028
Whiteboard: [design-decision-needed] triaged
I'm confused. How will this differ from the Filesystem API implementation of bug 1265767?
It's undecided at this point.
This seems like a good idea with multiple use cases and there's been a few suggestions how we could do it. The first stage would probably be something like:
* implement the Blink Filesystem APIs (as noted in bug 1265767)
* make it a permission that a developer has to request
* limit the reading and writing to an add-on specific sandbox in the developers profile

Once we've got that done, we could move on to the next stage which would be allowing an add-on to read and write from any directory that the user would approve. Because it would require user approval we'd need some UX and security review to ensure that we can limit malicious activity.
Whiteboard: [design-decision-needed] triaged → [design-decision-approved] triaged
Blocks: webext-port-mega
We'd at the very least need required permission support for this. Its probably an optional permission so add as depending on that one.
Component: WebExtensions: Untriaged → WebExtensions: General
Depends on: 1197420
Priority: -- → P2
What permit is required? when it will be implemented for the Nightly?
Any news or updates on this CRITICAL issue of WebExtensions?

A lot of useful addons CAN'T be made without ability to read/write local real filesystem.
Like: SessionManager, Mozilla Archive Format, DownloadThemAll, Places Maintains and etc.

The feature maybe limited to some folder (e.g. add-on's folder), or with user's manual approval or user's folder choice or something else if needed, but dropping the whole thing is obviously not a right decision, we don't want another quite dull and limited half-a-browser like Chrome, do we?

So, the question is - When should developers expect filesystem support for the-only-recommended WevExtensions?
I mean, it's one of the most desired feature (according to votes!) for WebExtensions, that always was in Firefox and is required for many tasks. Is this feature decided to be added soon?
Hi,
I have just started contributing to Mozilla and would like to work on this bug. It would be really great if I was given pointers as to how to start working on it.
Thank you :)
Suhan thanks for wanting to help out. We are happy to have contributors for bugs, but if you are just starting contributing to Mozilla then you might want to start with some easier bugs. Check out this link: https://wiki.mozilla.org/Add-ons/Contribute#Improve_add-ons_in_Firefox, which links to the onboarding and bugs that are mentored and recommended for your first bug.
Blocks: 1118289
See also bug 1323414.
See Also: → 1323414
webextensions: --- → +
Dropping down to ? for Firefox 57.

At this time we think it makes sense to:

* implement a streaming API for downloads, this is basically the proposal on bug 1323414 and would allow add-ons that stream a large amount of data (for example Video Download Helper) to be ported over to WebExtensions. More bugs will be filed for this.

* bug 1331618 allows extensions to store up to 20gb per file into indexeddb and use that as a virtual file system and should meet most storage systems. We might need to add in a library to make this easier to use, or re-use one that already exists.

Most use cases we've found so far would fit into one of the two use cases above. So for the moment I'm dropping this down until we can find more use cases.
webextensions: + → ?
(In reply to Andy McKay [:andym] from comment #12)
> Most use cases we've found so far would fit into one of the two use cases
> above. So for the moment I'm dropping this down until we can find more use
> cases.

For my add-on withExeEditor *1, it is essential that this bug is FIXED by Firefox 57.
withExEditor creates a temporary file in the OS's temporary folder for source view, selection view, and text editing, and passes that file path to the text editor.
And when text editing, after user edits text, add-on gets that temporary file and update it in the web page.

Apart from this bug remaining, almost all the functionality has already been migrated to WebExtensions (not released yet).

*1 https://addons.mozilla.org/en-US/firefox/addon/withexeditor/
I presume withExeEditor uses native messaging to communicate with a local editor?  If so, the native application could do the filesystem reading and writing...
(In reply to Andrew Swan [:aswan] from comment #14)
> I presume withExeEditor uses native messaging to communicate with a local
> editor?  If so, the native application could do the filesystem reading and
> writing...

Pass whole content instead of file path to the host?
(In reply to Kazumasa Hasegawa (Kazz) from comment #15)
> Pass whole content instead of file path to the host?

Yes.  Are you worried about the size?  Its already data that's been moved across the network and is moving to a file, an extra inter-process copy doesn't seem like a problem...
(In reply to Andrew Swan [:aswan] from comment #16)
> Yes.  Are you worried about the size?  Its already data that's been moved
> across the network and is moving to a file, an extra inter-process copy
> doesn't seem like a problem...

Thanks. I'll think about that.
It seems to need total redesign again...
With the proposed approach, how would one go about implementing addons in the vein of ASF (Automatic Save Folder) which need to circumvent the target folder restrictions described in Chrome's download API docs? 

(Or would an addon developer just have to ask non-Windows users to set the default download location to the filesystem root and warn Windows users that it's constrained to a single drive?)
We would like to create a Firefox extension for Kiwix, based on https://github.com/kiwix/kiwix-html5/
It's an offline wikipedia reader, that reads its content inside big local files (and they can be very big : http://download.kiwix.org/zim/wikipedia/?C=S;O=D ). These files are usually copied from another media (or downloaded separately).
It already works by simply opening the local HTML file. But the user needs to manually select the file(s) each time (through an "input type=file").

A browser extension with such filesystem API would allow us to :
- remember the selected file(s), and re-open them automatically
- "package" the application in a more user-friendly way than a local directory where the users needs to open an HTML file

The streaming API would be useless for us.
A local sandboxed storage might technically work, but does not seem adequate to our use-case : the files will probably be too big for that, and the user will (I suppose?) be forced to download them from within the extension itself (instead of copying them offline).
Direct local file system access is useful in may cases:

1. Save addon data in a Dropbox / Google Drive / Box folder so that they can be synced across the web automatically.

2. Specify an "input" and an "output" directory and perform a batch convert.

3. Flush HTML + CSS + JS files in a specifically folder dynamically so that they can be opened from the  browser.

These effects cannot be achieved by FileSystem API, indexDB API, streaming API, or downloads API.
>Andy McKay: So for the moment I'm dropping this down until we can find more use cases.
What does "dropping this down" mean exactly (literally)?

This is this critical issue, WebExtensions can not be offered as the only recommended replacement if they are still so limited.

As for use cases, here one more: Addon wants to store some files with folders hierarchy in folder/pass, that was set/chosen by user.
It could be result of some parsing of page, some script's downloads or math evaluations, doesn't matter.

The previous extensions API allowed this, and the current - doesn't. It should be fixed, it's critical for such addons, you take what worked and replace with something less functional, it can't end well.

Local file system access IS MANDATORY, users and developers said that far more than once. It can be limited by permissions, by user manual approval as offered in commends, but it MUST be, otherwise Firefox will become another dull chromish-browser.
Local file system access IS MANDATORY. Thank you for your time.
> Local file system access IS MANDATORY. Thank you for your time.

I have to agree. At its core, there are two gaping holes in the new extension API which have me scrambling to arrange a Firefox exit strategy:

1. The ability for extensions like Automatic Save Folder, downThemAll!, Greasemonkey, savetexttofile, ScrapBook, and TiddlyFox to integrate with the filesystem without going through the aforementioned crippled APIs

2. The ability for Classic Theme Restorer and Stylish to revert enough of Australis to at *least* match Chrome's level of UI native-ness.

Greasemonkey and savetexttofile should be simple fixes with some kind of external helper, so I'm not too worried about them. (I can't be the only userscript developer who iterates on scripts by opening the installed script and using `<Ctrl+S> <Tab> <F5>` to apply changes and my experiments say that it should be possible to force no-dialog saving so an inotify-based helper can pick up the output of a savetexttofile successor and append it to the actual destination file.)

Automatic Save Folder and downThemAll will be more painful, but I can probably replace them by writing a WebExtensions shim which essentially hijacks all Ctrl+S functionality and pipes the resulting stuff to JDownloader or KGet. (Possibly requiring an XGrabKey+XTest helper to remap my muscle memory if it doesn't have permission to universally hijack Ctrl+S)

ScrapBook may be OK if the raw on-disk format of Firefox's IndexedDB is documented enough for me to do I/O on it from outside the way I can with ~/incoming/ffox_scrapbook/

It's TiddlyFox and Classic Theme Restorer that are the big problems.

TiddlyFox is already a life-support patch for the XPConnect-based save code in TiddlyWiki Classic and, while I really don't want to have to invent some kind of HTTP daemon whose only purpose is to open up a hole in Firefox to restore the ability for an HTML+JS quine to move foo.html to foo.html.backup and then write new foo.html and foo.xml (RSS) files, if I must then I must.

As for Classic Theme Restorer and the XUL style functionality in Stylish, well, let's just say that Chrome is closer to the browser I want than Australis would be without CTR. (The only use I have for "lightweight" theming is to recolor Chrome so it matches my Qt/GTK+ theme.)

Without Classic Theme Restorer, my exit strategy is looking like it'll be "migrate as much as possible to Chrome-compatible WebExtensions, migrate the rest to external stuff like the Privoxy HTTP proxy server, install those extensions in Chrome, and then rely on Chrome's UI being closer to what I want while I wait for something like QWebEngine to implement WebExtensions support so it's feasible for me to maintain my own browser frontend."
...and I just realized that I went off on a tangent there. My apologies.

My original intent in bringing CTR into the mix was to use it as part of an argument as to the degree of importance filesystem-accessing extensions play in my workflow, but I can't remember exactly how.
After a quick review of Automatic Save Folder, it seems it does not need full filesystem access, it just needs an API to downloads to decide where the file should go. The API could be passed the originating domain or the file name and decide where it gets written. There's some nuance there like, it shouldn't allow overwriting files and there might be permissions we need to prompt the user about there, but seems doable.

Similarly for saving to a drop box folder, again the ability to run downloads to a specific location, or generate them using downloads.download API gets them into your DropBox folder. Then you can perform any processing of files on the native file system using nativeMessaging.

DownThemAll is not planning on moving to WebExtensions as far as I know (although I'm not the author of that add-on so please don't take that as gospel). 

SaveTextToFile can use the downloads API to write a file to a location of the users choosing.
(In reply to Andy McKay [:andym] from comment #24)
> After a quick review of Automatic Save Folder, it seems it does not need
> full filesystem access, it just needs an API to downloads to decide where
> the file should go. The API could be passed the originating domain or the
> file name and decide where it gets written. There's some nuance there like,
> it shouldn't allow overwriting files and there might be permissions we need
> to prompt the user about there, but seems doable.

That'd definitely be doable for ASF. Heck, a proper WebExtensions API for determining destinations like that would actually be *better* for my use because it would turn ASF into little more than a visual programming frontend for an API I could easily and comfortably script directly.

(There are various situations where ASF falls short currently because it doesn't support matching against selectors in the DOM and I'd like to implement things like "On 'Save Image As...' on this list of *booru-style image boards, scrape the tags sidebar and map recognized combinations of tags to download folders based on priority ordering.")

> 
> Similarly for saving to a drop box folder, again the ability to run
> downloads to a specific location, or generate them using downloads.download
> API gets them into your DropBox folder. Then you can perform any processing
> of files on the native file system using nativeMessaging.

Without overwrite, I have no reason to save into the Dropbox folder. ~/Dropbox isn't inside ~/incoming and, as a Linux user, I see the reliance on native messaging as a MAJOR step backward since I'll probably have to maintain forks of various extensions myself to get Linux support.

...so, again, I'd probably just resort to some hack like setting my default downloads folder to to ~/.downloads_dispatch/ and then using `mount --bind` to outwit any "don't follow symlinks" mechanism.

As for TiddlyFox, Without the ability to either rename or overwrite arbitrary files, it'd be almost useless as anything more than a life-support harness for TiddlyWiki Classic. (TW5 already falls back to opening a save dialog every time you click "Save") ...so it would most definitely have to rely on a nativeMessaging helper without an API for that.

(I suppose I could contribute one to them if it comes to that, given that Rust makes it easy to cross-compile to a fully statically-linked musl-libc target. Still, speculatively bundling dozens of different helpers for platforms like my arm-unknown-linux-gnueabi with X11 (my OpenPandora) is inferior to Just Working™ on any platform as soon as someone cross-compiles Firefox itself... but, then, for my OpenPandora, it'd probably just be easier to keep an old Firefox build around.)

> 
> DownThemAll is not planning on moving to WebExtensions as far as I know
> (although I'm not the author of that add-on so please don't take that as
> gospel). 

Yeah. DTA was sort of loud about that.

Given that Giorgio Maone is more receptive, I'm hoping that FlashGot with either JDownloader or KGet will prove to be a suitable replacement. (Using the "legacy" JDownloader API which doesn't require a My.JDownloader account to add URLs is a pre-requisite for JDownloader being considered suitable.)

I've already got a Chromium->JDownloader extension using the "legacy" API, so that's covered if I do decide that Firefox's demographic targeting has left me behind.

> 
> SaveTextToFile can use the downloads API to write a file to a location of
> the users choosing.

The reason I mentioned SaveTextToFile is that I use it in Append mode only and I don't know of any way to append to files using the downloads API.

(In fact, I use it in append mode so habitually that I forgot to mention that because I forgot it supports anything else.)

Give me an "append to file" API and I wouldn't mind spending the 15 minutes or less to replace SaveTextToFile with a webextension that hard-codes my preferred settings and displays a context menu entry that captures the selected text.
In bug 1338780 I gave two scenarios of usage, where necessary overwrite the files and full access to the file system at least in a special directory.

It should not be implemented with restrictions writed above. Need implementation, writeAtomic relevant in the current context with full access to the extension directory
I don't get what problem!
 - First of all we should define protocol for this (local://|fs://);
 - Second there must be unified paths on all platforms, so not this "C:\\...", "/home/...", but this "local://shared/...", "local://secure/..."
 - Accordingly to the second point there must be logical to physical paths mapping, which user can setup in about://settings
 - In manifest file (of webext) there should be field where developer define all files which he intent to use, for example: "fs_perms": {"read":["local://etc/1.txt"],"write":["local://etc/1.txt"],"exec":["local://etc/", "python", "npm"]}. Any attempts to use not declared file should throw exception, and as option send the claim to AMO.
 - In AMO should be insistent block which in plain language tells what fs permissions this webext is require. And when xpi is downloaded an user push the install button, the should be another warning? for example popup with "Omg! This extension attempt to override your /etc folder" (in this stage we already know physical paths which ext will use.
 - Developers should can make self folders where they will store files easy accessible to user. For example "local://books/", "local://pdfs" etc. For this in fs_perms there should be additional field "create". When user install the extension there should be info alert which confirm creation of listed folders in desktop dir of user.
 - There should be "local://tmp/" which points to the system/user tmp dir.
 - There should be limits settings in about://settings page. If developer attempt to overcommit limits then he must specify in AMO webext home page that user should increase self limits otherwise his webext will crash when rich limits.
 - If webext require to read system files, then developer should do this through native-messaging to some system utils.
 - There should be opportunity to encrypt files in "local://secure/" folder. For example now it will be unencrypted, but we will explain in api docs that files in this folder can be encrypted in the future and developer should put to this folder only that files which should be readable only from his webext and none of others exts/programs/utils. Or he can provide sha/rsa/... keys and files in this folder will be encrypted transparently.
 - Each webext has own "local://secure/" folder. So should be dynamic mapping.
 - Should be opportunity to use other protocols. For example "ftp", "sftp", "webdav", "http", "custom"
> I don't get what problem!

In my opinion, you did not read the comments above. For example, #27.
Extensions need to write and to write in files with unknown in advance names.
Yes I read this thread. But from security view override any file in system is ugly. As alternative we can give opportunity to make personal folder for each extension where it can create any what it want names of files and read and write to. For example folders created with "create" property of fs_perms field has this omitted perms {"read": ["local://folder/*"], "write": ["local://folder/*"]}. Or we can accept specify folder as subject to write permissions ({"write":["local://etc/"]}) then in this folder extension can do create rename delete files operations.
Again, that doesn't help with cases like these:

1. The whole purpose of the TiddlyFox extension is to allow TiddlyWiki HTML files to rename and overwrite themselves

2. SaveTextToFile's append mode is predicated on allowing the user to define an arbitrary file path which will be appended to by selecting some text and then choosing a context menu entry.
First of all TiddlyWiki not designed to be webextension. If you want, you can use localStorage for this purpose (not append themself but store changes to the some of storage).
Secondary if you want to make extension based on TiddlyWiki, you can change sources that: In extension package html contain iframe which sourced to the created by extension folder, then in this extension you define command on "Ctrl+s" or define page action or browser action which read the iframe content and saves it to the folder. And voila!
Ok, I was having a hard time making sense of your writing before, but now it's gotten so bad that I'm very worried I've misunderstood you somewhere.

That said, here's my best attempt...

> First of all TiddlyWiki not designed to be webextension. If you want, you can use localStorage for this purpose (not append themself but store changes to the some of storage).

Completely unacceptable. I can't turn my localStorage into a URL like http://ssokolow.com/story_ideas.html (a hobby TiddlyWiki of mine) simply by running `rsync`, nor can I drop it onto a thumbdrive for anyone to read with whatever browser they have on hand.

> Secondary if you want to make extension based on TiddlyWiki, you can change sources that: In extension package html contain iframe which sourced to the created by extension folder, then in this extension you define command on "Ctrl+s" or define page action or browser action which read the iframe content and saves it to the folder. And voila!

I'm not entirely sure what you're trying to say here, but I'm pretty sure that I'd have to stick to versions of Firefox that allow me to disable extension signing enforcement in order to accomplish it.

All in all, I get the impression that you're saying something equivalent to "No, we refuse to support Encrypted Media Extensions and you're not allowed to switch to Chrome for your NetFlix either." It's ridiculous and an impractical demand to make.

(And, sure enough, if WebExtensions-only Firefox won't meet my needs, I'll grab the last pre-WebExtensions-only ESR, disable updates, and use that. "Defining the problem away" is no way to build a tool that people actually need to use to get work done.)
Why you are want that TiddlyWiki will be webextension? Just save it where you want and open it when you want by double click on *.html. What problem? It was designed for such use. And no one will take away that opportunity.
When I click "Save", TiddlyWiki Classic is supposed to save changes. It has no fallback mechanism for presenting a download dialog like TiddlyWiki 5.

It used to save via XULConnect. When that was taken away, they wrote TiddlyFox, a XUL extension which un-breaks the save feature and allows TiddlyWiki 5 to do it without the Save As dialog.

Mozilla wants to kill off XUL extensions and replace them with WebExtensions.

The only way TiddlyFox could allow saving via WebExtensions, currently, is to present a download dialog. 

It's unacceptable for me to have to meddle with a download dialog every time I click Save or to store my TiddlyWikis in some special Firefox-specific place, rather than in the project folders where they currently reside alongside related things.
Cool it guys, we all want Firefox to be good, flexible, powerful and secure.

Let's try to agree on statement that there CAN BE extensions that NEED to write to local filesystem. The must-have requirement that should be fixed in memory, otherwise this issue will exist for years!

Maybe it should be limited to some fixed folder, maybe with some permission-ask, maybe even with manual allowance via about:config, maybe this extension should be hidden by default on addons.mozilla.org, but let's fix that such extensions CAN and MIGHT be, agree?

If you agree, than the API to write to local filesystem in some folder is mandatory.

PS. The issue was created more than a YEAR ago, any hope that someone will take responsibility for this task, mozilla-guys?
PS2: And I agree with Stephan, that saving dialog is not acceptable for some tasks, because the extension might save the results in batch of files, and what the user should do, click 150 times? So the future solution should not be limited to per-file-manual-user-clicking thing like saving dialog.
You will can map for example "local://tiddlyFox/" logical folder (where tiddlywiki future webext will use) to the your "C://Projects/Wiki/somethingelse" physical folder by two ways:
1. In about:config make record "webext.folders_mapping.tiddlyFox" = "C://Projects/Wiki/somethingelse" (its not real key, it's for example);
2. Make the system symlink from path where Firefox will want to store files appurtenant to the "local://tiddlyFox" to the needed destination.

You will do it only once and none of confirmation dialogs will be showed when this extension will be write to this folder (save your session). Except only one confirmation dialog (for creating new folder and report what fs actions this ext will do in your pc) when you install this extension.
(In reply to jamy023 from comment #38)
> You will can map for example "local://tiddlyFox/" logical folder (where
> tiddlywiki future webext will use) to the your
> "C://Projects/Wiki/somethingelse" physical folder by two ways:
> 1. In about:config make record "webext.folders_mapping.tiddlyFox" =
> "C://Projects/Wiki/somethingelse" (its not real key, it's for example);
> 2. Make the system symlink from path where Firefox will want to store files
> appurtenant to the "local://tiddlyFox" to the needed destination.
> 
> You will do it only once and none of confirmation dialogs will be showed
> when this extension will be write to this folder (save your session). Except
> only one confirmation dialog (for creating new folder and report what fs
> actions this ext will do in your pc) when you install this extension.

Not very helpful, because:

1. A lot of users (especially Windows users) have no idea what a symlink is.

2. I'd just end up using `mount --bind` to point `local://tiddlyFox` at `/` in a way that any kind of "protect you from yourself" symlink checks wouldn't notice.

2a. The only common root for /media/ssokolow/16GB_Flash, /home/ssokolow/src/ssokolow.com/story_ideas.html, /home/ssokolow/Documents/Critical/tiddlyme.html, and /home/ssokolow/Documents/writing/Non-Fiction/ is the root of the filesystem.

2b. I don't feel like babysitting a bunch of symlinks as I create, move, and rename TiddlyWikis.

3. Running TiddlyWiki off a thumbdrive using a PortableApps copy of Firefox is not uncommon, and that means you can't rely on the drive letter to remain constant. (TiddlyFox just pops up an "Allow saving?" dialog the first time you open up a TiddlyWiki on an unrecognized path)
> 1. A lot of users (especially Windows users) have no idea what a symlink is.

This users actually have symlink opportunity. This users simply will use original folder which Firefox will made for this extension. Developers of extension can teach this users how map folder to the needed destination through about:config in his AMO webext home page. Or even how to make symlink in Windows.
 
> 2. I'd just end up using `mount --bind` to point `local://tiddlyFox` at `/`
> in a way that any kind of "protect you from yourself" symlink checks
> wouldn't notice.

If you run Firefox as root then maybe true. But usually the system's acl will reject any attempts to override system files. Then extension will not work. You will not get any profit of this trick :)

> 2a. The only common root for /media/ssokolow/16GB_Flash,
> /home/ssokolow/src/ssokolow.com/story_ideas.html,
> /home/ssokolow/Documents/Critical/tiddlyme.html, and
> /home/ssokolow/Documents/writing/Non-Fiction/ is the root of the filesystem.

I don't understand what this mean actually. Maybe you can use that:
ls $HOME/tiddlyFox/
  ssokolow.com -> /home/ssokolow/src/ssokolow.com/
  Critical -> /home/ssokolow/Documents/Critical/
  writing -> /home/ssokolow/Documents/writing/

> 2b. I don't feel like babysitting a bunch of symlinks as I create, move, and
> rename TiddlyWikis.

Otherwise you will hosting malware software (If webext's will have full filesystem privileges). Remember, webext's will run from your user (ssokolow), and theoretically have choice to look at any file in your home folder. Do you want that other (not tiddlyWiki) extension for example some advertise blocker or so, as payment for itself, collect info about you. Your conversations, bookmarks, web navigation history, passwords, keys from your ssh logins to the ssokolow.com host, bank moneykeepers keys, photos, etc is right paying for that freedom which you beg from us? 
 
> 3. Running TiddlyWiki off a thumbdrive using a PortableApps copy of Firefox
> is not uncommon, and that means you can't rely on the drive letter to remain
> constant. (TiddlyFox just pops up an "Allow saving?" dialog the first time
> you open up a TiddlyWiki on an unrecognized path)

And what problem? Just don't use manual mapping settings an lookup extension's data in most ease place -> Desktop or Home dir. Any desktop os propagate homedir path. And you can (if you want) tell Firefox to use different home dir, for example on thumbdrive. Developers of portable distribution of Firefox can do it for you.
> I don't get what problem!
> - First of all we should define protocol for this (local://|fs://);


FireFox must implement very simple functions. The first requirement of security is simplicity.
You offered so intricate scheme that they themselves do not understand that it does not implement the use case in comments #27 .
Or I did not understand. That is, your scheme is so complicated that some of us do not understand what is happening.


From intricate schemes started the main security problems in architecture.

Therefore, it is easier and safer make implementation the standard and simple operation. Which now supports FireFox.
If you want to restrict addons, the user could be asked to give permission for an extension.


The file operation should be implemented through a specially designed file API. Security operations should be implemented through a specially designed security features.
(In reply to fdsc from comment #41)
> > I don't get what problem!
> > - First of all we should define protocol for this (local://|fs://);
> 
> 
> FireFox must implement very simple functions. The first requirement of
> security is simplicity.

Who saying that it's not simple? You? This is very simple and secure scheme.

> You offered so intricate scheme that they themselves do not understand that
> it does not implement the use case in comments #27 .
> Or I did not understand. That is, your scheme is so complicated that some of
> us do not understand what is happening.

The scheme like my is used for a long time. Remember chrome protocol. And even now in webextensions for access to extensions self files used similar scheme (web-extension://) the difference is it using some hash key for folder name, but mine scheme using human readable folders name. If you don't understand mine scheme how you will go forward? Maybe I can explain how my scheme is working and how it rich a safety? Give me a full usecase and I give you description how you can develop this with my scheme of FS API.

> From intricate schemes started the main security problems in architecture.

Again: my scheme is not intricate. If it not clear for you it's does not mean that it not clear for others.

> Therefore, it is easier and safer make implementation the standard and
> simple operation. Which now supports FireFox.

As I explained before, Firefox using similar operations for a long time. And even more difficult.

> If you want to restrict addons, the user could be asked to give permission
> for an extension.

Many users don't understand what happened and scared if some dialog windows appearing unexpected. And then they stopped touch the computer and call the support. Or simply accept all and doesn't worry.

> The file operation should be implemented through a specially designed file
> API. Security operations should be implemented through a specially designed
> security features.

Examples please. What security features?
I think you need to add a bug 1215059 in blocks
Flags: needinfo?(billjm05)
> This is very simple and secure scheme.

Once again, if your scheme is simple, it does not satisfy the use case from Bug 1338780
(In reply to fdsc from comment #44)
> > This is very simple and secure scheme.
> 
> Once again, if your scheme is simple, it does not satisfy the use case from
> Bug 1338780

> 1. The user should be able to provide detailed logs if he gets an error and developer cannot be repeated.

local://shared/logs/ for logs

> 2. A developer wants get full user setting the additions. It is easiest to implement if it's possible to export the settings to the file.

local://secure/settings.json

>USE CASE 2
It is possible to transfer add-on settings between different versions of the add-on or between different user profiles (to set up similar profiles).

local://shared/tiddlyFox/settings.json

I think if developer a real developer he can explain to the user how to find ".mozila/firefox/webext_shared_data" folder in his home folder if he searching files from local://shared/ logical folder. If user needs files from local://secure/ folder then he should find folder of his firefox profile and here he can find "webext_secure_data" folder where places folders with human readable names equals to extension names. And it's intricate for you?

And this scheme of File Api eliminates of differences in different platforms as well as originality of technology allow developer to not worry about verry old versions of firefox, maybe even not worry about of absents of about:debugging
> local://shared/logs/ for logs

And in order to destroy the entire world you want local://destroyTheWorld

Do you think or just write the names?

If you are not a developer or you have no qualification, let's not clutter this bug with useless arguments?


> I think if developer a real developer he can explain to the user how to find

I think if the developer is a real developer, there is no need to explain to the user how to do the actions that user do not want to do.
> And this scheme of File Api eliminates of differences in different platforms as well as originality of technology allow developer to not worry about verry old versions of firefox, maybe even not worry about of absents of about:debugging

It is not the schemes, but that addition is not able to write and read files. This scheme is just useless, as it does not implement file operations.
(In reply to fdsc from comment #46)
> > local://shared/logs/ for logs
> 
> And in order to destroy the entire world you want local://destroyTheWorld

If developer of extension is declare in manifest that he want to create destroyTheWorld folder then he can write to or read form. The user when will install this extension will see that this extension wants ot create that folder on his desktop.
 
> Do you think or just write the names?
> 
> If you are not a developer or you have no qualification, let's not clutter
> this bug with useless arguments?

I am a developer. And you my dear friend, come up the unreal usecases.

> > I think if developer a real developer he can explain to the user how to find
> 
> I think if the developer is a real developer, there is no need to explain to
> the user how to do the actions that user do not want to do.

This your idea that user must send logs and settings to the developer if extension is throw exceptions.

(In reply to fdsc from comment #47)
> > And this scheme of File Api eliminates of differences in different platforms as well as originality of technology allow developer to not worry about very old versions of firefox, maybe even not worry about of absents of about:debugging
> 
> It is not the schemes, but that addition is not able to write and read
> files. This scheme is just useless, as it does not implement file operations.

It is concept. Which we can use. If you my dear friend don't get yet, then I explain you that new webextension is much more different from old addons and as proof I can give you a quote: 

http://webextensions-experiments.readthedocs.io/en/latest/new.html

There are some basic principles around WebExtensions APIs as they currently stand. Any API that wants to get merged into Firefox would need to meet these principles.

    Security: the API should not expose an unacceptable risk to the end user.
    Privacy: similar to above.
    Performance: APIs should be async by default and focus on not causing jank, hangs or any sort of bad performance in Firefox.
    Multi-process: all APIs need to be multi-process aware.
    Useful: APIs that land in Firefox should be useful since all APIs have a maintenance burden.
    High level: WebExtensions provide a known public API layer on top of Mozilla code, allowing the underlying code to change quicker and easier.
    Alternatives: Do alternatives exist for example, WebAPIs?

Compared to legacy Firefox add-ons there is a significant sandbox around WebExtension APIs that is very restrictive. Access to priviledged APIs within Firefox, preferences, arbitrary File System access, socket access and so on are restricted (at the time of writing this documentation). APIs that break out of this sandbox will need very clear justification for why.
> This your idea that user must send logs and settings to the developer if extension is throw exceptions.

How this log will be recorded by the extension?
Write the example of code that write to log


> need to meet these principles.

First of all, these APIs should function. If someone thinks that some feature is dangerous, the user must choose if you want it or not. The user and not the developer who chooses.

The most secure API, that which does not exist.
I now write the browser without any extensions. Or to do without a browser. After all, it will be safer.
(In reply to fdsc from comment #49)
> > This your idea that user must send logs and settings to the developer if extension is throw exceptions.
> 
> How this log will be recorded by the extension?
> Write the example of code that write to log

1) function someOuterFunction(){
  //some code here
  try {
    //some code that may not work
    dangerousMethod();
  } catch (error) {
    fs.appendFile('local://shared/logs/tiddlyFox/erros.log', error.message);
  }
  //some code here
}

2) function someOuterFunction(){
  //some code here

  //some code that may not work
  dangerousMethod().then(success, function(error){
    fs.appendFile('local://shared/logs/tiddlyFox/erros.log', error.message);
  });
  //some code here
}

3) function someOuterFunction(){
  //some code here

  //some code that may not work
  var result = dangerousMethod();

  if(result == undefined) {
    fs.appendFile('local://shared/logs/tiddlyFox/erros.log', runtime.lastError);
  }
  //some code here
}

4) runtime.onUnhandledException = function(error) {
     fs.appendFile('local://shared/logs/tiddlyFox/erros.log', error.message);
}

More questions?

> > need to meet these principles.
> 
> First of all, these APIs should function.

First rule -> don't harm
Second rule -> help him if can don't break the first rule

> If someone thinks that some feature is dangerous, the user must choose if you want it or not. The user
> and not the developer who chooses.

Who knows is it danger or not? Nobody! It can work fine and malicious code can sleep long time, meanwhile this extension will acquire big group of users. And when user will see confirm dialog, he look at extension home page and find: "over thousand downloads, five stars, no abuse reports" - ok! this is good extension! Confirm!

> The most secure API, that which does not exist.

Do you propose that we are choose:
  or make most secure API (that which does not exist)
  or make most unsecure API (that you beg from us)
?

We are choosing middle. 

> I now write the browser without any extensions. Or to do without a browser.
> After all, it will be safer.

You can still use legacy bootstrapped add-ons if you don't like webextensions
> More questions?

It dawned on me. You suggest to make the file API with the limited special scheme "local". But the API is generally the same.
And the user can install the appropriate directories and local directories on your computer.

But do this in FireFox and so it was convenient. That is someone everyone needs to realize, so much so that the user chooses his need. And that it could, if user necessary, to remove all restrictions, as if the extension works with the whole filesystem.

Otherwise it can be very uncomfortable for the user.


> You can still use legacy bootstrapped add-ons if you don't like webextensions

As I understand it, the SDK functionality will be disabled in November.


> It can work fine and malicious code can sleep long time

The user have risks when installing application on your computer. For example, installation a licensed game with a safe driver or even a network card driver.
With them, the situation is much worse.


> Who knows is it danger or not? Nobody!
1. AMO gallery editors for gallery extensions.
2. User. User takes risks when installing any software on his computer. How he decides is his business. We only need to give him information about what he may really need it. The decision is the one who manages risk. Manages the user.
The developers of FireFox don't have to solve everything for everyone. People don't like it. People want to live in self mind. No Mozilla mind.


Working with the file system can be limited by means of the operating system if the user is afraid. I do for certain applications.
>> And that it could, if user necessary, to remove all restrictions, as if the extension works with the whole filesystem.

Usage scenario: any save files: for example, simple encryption and file conversion.


That is, the user must still have the ability to set "local" scheme for this extension as if it was on the entire file system. I am afraid, however, it will be difficult to properly implement. But it will allow the user to find a balance between security and convenience.
(In reply to fdsc from comment #51)
> But do this in FireFox and so it was convenient. That is someone everyone
> needs to realize, so much so that the user chooses his need. And that it
> could, if user necessary, to remove all restrictions, as if the extension
> works with the whole filesystem.

> Otherwise it can be very uncomfortable for the user.

If user so need to use this danger opportunity, then he can:
1. stick with "Nightly" or "Firefox developer edition" and install webexperiment (which has privileged API access) and install the webextension which will requests to webexperiment for privileged actions.
2. Some portion of time developers can use embedded webextensions. This is hybrid of legacy bootstrapped addon with new webextension in one single addon. This type of privileged add-ons we will support as long as possible so the developers have time to port their addons to new API (and migrate data of course). Two main difference is that you can't use XUL and can't use "chrome" module API.
 
> > You can still use legacy bootstrapped add-ons if you don't like webextensions
> 
> As I understand it, the SDK functionality will be disabled in November.

I am described the situation above.
 
> > It can work fine and malicious code can sleep long time
> 
> The user have risks when installing application on your computer.

On my computer? Or maybe on self computer?

> For example, installation a licensed game with a safe driver or even a network
> card driver.
> With them, the situation is much worse.

It's not the Mozilla problems. They only want to deliver better product, like Chrome/Opera developers.

> > Who knows is it danger or not? Nobody!
> 1. AMO gallery editors for gallery extensions.

No resources for that. You will be volunteer? Even that, who knows, maybe you friend of extension developer and purposely miss the danger extension.

> 2. User. User takes risks when installing any software on his computer. How
> he decides is his business. We only need to give him information about what
> he may really need it. The decision is the one who manages risk. Manages the
> user.

Many users don't read just confirm. And then "Omg! This browser so slow! It eat all of my RAM! It hang my system....". Again if some restriction is redundant then we away it. But at this moment you don't give me real justification that full filesystem access is very important for the main part of extensions. Example about tiddlyWiki is very custom, but I am resolve it for you. Example about logs and settings too. Do you have more usecases where full fs access very need?

> The developers of FireFox don't have to solve everything for everyone.
> People don't like it. People want to live in self mind. No Mozilla mind.

I think it's you trying to solve "everything for everyone". Who says that this concept don't cover all usecases? Or you trying to solve just self usecase and never mind about security of others?
 
> Working with the file system can be limited by means of the operating system
> if the user is afraid. I do for certain applications.

No! The reality is so that one system almost nevermind on ACL's (if you specially don't turn on this, what don't do 90% of users of this system), and other type of modern system use same user profile for all applications that user run (any app can see data produced by other app, and even modify). Little better situation with this in modern mobile systems, but we are talking about desktop systems now (if interesting, then I can give the cue that they use similar concept which I propose there).
(In reply to fdsc from comment #52)
> >> And that it could, if user necessary, to remove all restrictions, as if the extension works with the whole filesystem.
> 
> Usage scenario: any save files: for example, simple encryption and file
> conversion.

Ok! I will describe an this usecase. Developer make extension which have page with form on it. In this form there is <input type="file" html component. And declare in manifest file of his extension that will be write to "local://tmp/megaencryptapp.out". Then when user open extension he must simply push on input=file button, then system's choose file dialog appears. When user choose and confirm, then content of selected file will appear for js scripts of extension through DOM File API (https://developer.mozilla.org/en/docs/Web/API/File). Then extension do the job on it. When it done, it write result to the "local://tmp/megaencryptapp.out" and open next page where will be written "Success! You can save the result right now by clicking on this <a href="local://tmp/megaencryptapp.out">link</a>". When user click on this link the standard open/save dialog appear and user can save result where would he like. And voila!

> That is, the user must still have the ability to set "local" scheme for this
> extension as if it was on the entire file system. I am afraid, however, it
> will be difficult to properly implement. But it will allow the user to find
> a balance between security and convenience.

Completely don't get whole paragraph.
Hello fdsc and jamy,

Thank you for your information, I think we have enough information to properly evaluate this proposal at the next meeting.

There is no reasons for you to continue arguing the points anymore, so I encourage you to end this discussion, as it doesn't seem very productive anymore.  Thank you.
I can just echo fdsc, many plugin depend on file access, TiddlyWikibeing one of them. We need some sort of solution not to loose many useful plugins.
The easiest way to do this would probably be using the chrome.fileSystem app api. Firefox already added support for those filesystem interfaces for web folder upload. It may be possible to convince Chrome to enable that api for extensions.

https://developer.chrome.com/apps/fileSystem
webextensions: ? → ---
I have been keeping an eye on this topic for months. There are many occasions that are not covered by what was suggested earlier. 

As an example, I have an addon that requires some sort of file read (only READ and only for IMAGES) from ContextMenu action. I have been waiting to migrate to WebExtension, once an API becomes available.

It converts images to Data:URI. There is no problem with remote images but when the image is local (from file system) there is no method to read it in WebExtension in order to convert it to data:URI.

For example, user drops an image to the browser window. At the moment, my addon can read and convert it and send the data.
Another example is when user opens a local file (eg HTML) which includes a local image. That image will have a local src.

It is worth noting that in my cases, addon does not need access to unknown arbitrary files. I only needs access to files that are already opened by the user and displayed by the browser.

For my own use, I would be happy with any method that allows me to read that file, even if it is from the browser windows itself or browser cache.


A further limitation can be imposed by limiting file access by type/extension as well as separate read and write permissions.
I would only need READ and only for IMAGE types.

Also .. ref bug 1266960
(In reply to erosman from comment #58)
> For example, user drops an image to the browser window. At the moment, my
> addon can read and convert it and send the data.

This is you can achieve even now by DOM File API (https://developer.mozilla.org/en/docs/Web/API/File)
Working example is http://jsfiddle.net/z3JtC/4/

> Another example is when user opens a local file (eg HTML) which includes a
> local image. That image will have a local src.

This is not problem, the canvas "src" property can be path relative to the html file path. Don know about windows firefox builds, but in my linux firefox 52.0.1 working fine.
 
var canvas = document.getElementById('myCanvas');
var context = canvas.getContext('2d');
var imageObj = new Image();

imageObj.onload = function() {
  context.drawImage(imageObj, 69, 50);
};
imageObj.src = 'darth-vader.jpg';
> This is you can achieve even now by DOM File API (https://developer.mozilla.org/en/docs/Web/API/File)
Working example is http://jsfiddle.net/z3JtC/4/

That is not the intension. The intension is NOT to use file input by the user to OPEN the file but use ContextMenu on an already opened image.

Using New File() with local file (ie file:///.....jpg) will result in error.

Similarly, using canvas fails with imageObj.src = 'file:///....darth-vader.jpg';

I have even tried fetch and XHR and all fail since access to local file of all these APIs is blocked via match patterns for "<all_urls>"
Ok. I see. I am even had experiments with your example yesterday. In webext scripts canvas.toDataURL throw securityError even for image not local but from https. Local images simply produce dataUrl with empty picture (black square). So you only one exit -> when user click on context menu item, you need open new tab (or popup window) where will be html page (from extension) opens in normal (not webext) mode. There you can place script like I am give you in comment 59 (not jsfiddle). This scheme will work
A filesystem API for WebExtensions is absolutely necessary. We just have to make sure each extension only gets access to the paths the user wants it to. Maybe this will require an additional sophistication of the permission system, but it's necessary. Many add-ons which prevent users from switching to the Chrome-likes depend on the filesystem I/O.
I'm planning to make a privacy-focused add-on which would enable users not only to delete their browsing data, but to also shred it (i.e. overwrite it with random bytes) so that it's unrecoverable. This feature is wanted by many users choosing to access sensitive data through their browser. Unfortunately I cannot do this through the currently presented WebExtensions APIs, and I don't feel like using legacy deprecated technologies which are going to get discared soon. So I will be watching this issue closely so as to be notified when I can get to work.
(In reply to ashpool from comment #62)
> I'm planning to make a privacy-focused add-on which would enable users not
> only to delete their browsing data, but to also shred it (i.e. overwrite it
> with random bytes) so that it's unrecoverable. This feature is wanted by

It's too much for webextension. I think you should don't wait this feature. You probably better to make standalone application which do the job and which user will install in operating system if he need this. For example with electron.io. For example I am as a user will worry about how this (privacy-focused) extension can break my privacy, than about my browsing data will remain in current device, because I am trust Mozillas guys more than someone who published some extension in extension store.

P.S. You should understand that extensions is cool, but main purpose of browser is sites, and browsing should be secure for user
Session Manager with 250k users also needs this API because it stores saved sessions in a separate folder in Firefox profile. Also, http://forums.mozillazine.org/viewtopic.php?p=14730663#p14730663
>At this point things don't look good. Web Extenstions don't give access to the API that Session Manager needs to actually save or restore sessions, let alone anything else it does. The only thing Web Extensions currently allows is reopening closed tabs from that current browser session. There is no way to "save" the session state and restore it after the browser has been closed and reopened
It looks like the developer of this addon didn't try to request the needed APIs for some reason so I decided to leave a comment in this thread in order to pay attention to the fact session manager will die after FF57 is released if the needed APIs are not implemented.
(In reply to ajfhajf from comment #64)
> Session Manager with 250k users also needs this API because it stores saved
> sessions in a separate folder in Firefox profile. Also,
> http://forums.mozillazine.org/viewtopic.php?p=14730663#p14730663
> >At this point things don't look good. Web Extenstions don't give access to the API that Session Manager needs to actually save or restore sessions, let alone anything else it does. The only thing Web Extensions currently allows is reopening closed tabs from that current browser session. There is no way to "save" the session state and restore it after the browser has been closed and reopened
> It looks like the developer of this addon didn't try to request the needed
> APIs for some reason so I decided to leave a comment in this thread in order
> to pay attention to the fact session manager will die after FF57 is released
> if the needed APIs are not implemented.

This is not true at all! I think that developer of this extension dont't request this API because clearly understand that he make a mistake when decided to store data directly to the file system. This is job for databases. And this feature we already have, and he can start to port his extension
> I think that developer of this extension dont't request this API because clearly understand that he make a mistake when decided to store data directly to the file system


I think he clearly understood what he wants.
He is not alone keeps the options of add-on in the file system. And, as has been indicated, there are other use cases.

Databases are generally unsuitable for a normal production use.
(In reply to fdsc from comment #66)
> > I think that developer of this extension dont't request this API because clearly understand that he make a mistake when decided to store data directly to the file system
> 
> 
> I think he clearly understood what he wants.
> He is not alone keeps the options of add-on in the file system. And, as has
> been indicated, there are other use cases.
> 
> Databases are generally unsuitable for a normal production use.

Any who use filesystem to store data should adopt their add-ons to use other solutions. There are two choices:

- they adopt their add-ons
- they not adopt and world lost their addons forever

Time of super-privileged extensions is gone. Who is against, that can use old versions or alternative forks. Filesystem will be, but not full and restricted. So we should turn on creative thinking and don't be afraid to redesign old goods
(In reply to jamy023 from comment #67)
> (In reply to fdsc from comment #66)
> > > I think that developer of this extension dont't request this API because clearly understand that he make a mistake when decided to store data directly to the file system
> > 
> > 
> > I think he clearly understood what he wants.
> > He is not alone keeps the options of add-on in the file system. And, as has
> > been indicated, there are other use cases.
> > 
> > Databases are generally unsuitable for a normal production use.
> 
> Any who use filesystem to store data should adopt their add-ons to use other
> solutions. There are two choices:
> 
> - they adopt their add-ons
> - they not adopt and world lost their addons forever
> 
> Time of super-privileged extensions is gone. Who is against, that can use
> old versions or alternative forks. Filesystem will be, but not full and
> restricted. So we should turn on creative thinking and don't be afraid to
> redesign old goods

...or, in cases where we end users don't like that arrogant "we know better" attitude, we can concoct our own filesystem-access APIs using either the native messaging API or a localhost HTTP daemon.

That's what I plan to do if TiddlyFox can't retain its ability to grant single-HTML-file TiddlyWikis the ability to copy file://$CURRENT_URL to file://$CURRENT_URL.backup, overwrite file://$CURRENT_URL, and then overwrite file://$CURRENT_URL_MINUS_EXT.xml if an RSS feed is requested.

(If I have to implement it myself, I'll probably go the HTTP route so I can whip up something small and simple with a UAC-esque UI using Rust, cross-compile to all the major platforms, and then let a single install service multiple browser extensions without a ton of config-file fiddling.)
(In reply to Stephan Sokolow from comment #68) 
> ...or, in cases where we end users don't like that arrogant "we know better"
> attitude, we can concoct our own filesystem-access APIs using either the
> native messaging API or a localhost HTTP daemon.
> 
> That's what I plan to do if TiddlyFox can't retain its ability to grant
> single-HTML-file TiddlyWikis the ability to copy file://$CURRENT_URL to
> file://$CURRENT_URL.backup, overwrite file://$CURRENT_URL, and then
> overwrite file://$CURRENT_URL_MINUS_EXT.xml if an RSS feed is requested.
> 
> (If I have to implement it myself, I'll probably go the HTTP route so I can
> whip up something small and simple with a UAC-esque UI using Rust,
> cross-compile to all the major platforms, and then let a single install
> service multiple browser extensions without a ton of config-file fiddling.)

Native message API is better. Localhosted HTTP daemon is like use sledgehammer for nailing.

And again you can store data in the db and have button with "Export to file". Only when button will be pressed then will be opened SaveAs dialog where user can select path where he want to save this wiki. The only one trouble I see is how to continue work when user next time open this file from file explorer (file://...). And from this case there is solution: you can inject content script to "<all_urls>". Then when in time of export to file you should place <meta name="GENERATOR" content="TiddlyWiki"> tag in <head> as flag means that content script should work with this page. Next is up to your fantasy.
This is all easy but for some reason you can't see this and make horrible solutions in Rust and localhosted http daemons, or beg us full filesystem access.
> Native message API is better. Localhosted HTTP daemon is like use sledgehammer for nailing.

That depends on how you're evaluating it. 

The Native Messaging API is clearly designed with an undercurrent of "make this API a prohibitive hassle to use unless the external component is primary enough to justify having its own full-blown installer."

My evaluation is based on end-user experience and, as such, I'd see the HTTP option as a superior choice because a single, simple install could produce an unofficial API extension that many add-ons (if necessary, unsigned and running via Developer Edition or un-branded builds) could share with little bother.

(Basically, if need be, I'm willing to reinvent an improved version of the popup permission dialog that TiddlyWiki originally had via the XULConnect API for filesystem access.)

> And again you can store data in the db and have button with "Export to file"....

You don't understand my needs.

A TiddlyWiki is a self-contained, structured document, more like a MS Word document in how it's saved, loaded, and shared. Try to imagine how much hue and cry there would be if an update to MS Word suddenly complicated the opening, saving, and sharing of documents like that.

Heck, TiddlyWiki 5 already has a Save As... dialog as a fallback.

Even if I were willing to risk losing up to 24 hours of progress to a saving bug not being countered by wiki_name.html.backup (that did almost happen once) and even for the wikis where I'm not generating an RSS feed, popping up a Save As... dialog every time I hit Save is completely unacceptable.

(Also, TiddlyWiki isn't the only thing I'm thinking of which needs this kind of functionality... it's just the one that's hardest to hand-wave away by saying "Screw it. I'm turning it into a native app. I can embed Blink in PyQt if need be and use py2exe to bundle it all up for thumbdrive use on the go.")
(In reply to Stephan Sokolow from comment #70)
> (Basically, if need be, I'm willing to reinvent an improved version of the
> popup permission dialog that TiddlyWiki originally had via the XULConnect
> API for filesystem access.)

I am understand this.

> > And again you can store data in the db and have button with "Export to file"....
> 
> You don't understand my needs.
> 
> A TiddlyWiki is a self-contained, structured document, more like a MS Word
> document in how it's saved, loaded, and shared.

MS Word is only file format. Opening in MS Office suite.

> Try to imagine how much hue and cry there would be if an update to MS Word suddenly complicated the
> opening, saving, and sharing of documents like that.

I am remember that for working with .doc/.docx docs I must "OPEN" it in Office suite then "SAVE" (export from RAM to Disk) it then I can share it. What wrong in my solution? (Thing that you call wiki_name.html.backup can be implemented in localStorage)

> Heck, TiddlyWiki 5 already has a Save As... dialog as a fallback.
> 
> Even if I were willing to risk losing up to 24 hours of progress to a saving
> bug not being countered by wiki_name.html.backup (that did almost happen
> once) and even for the wikis where I'm not generating an RSS feed, popping
> up a Save As... dialog every time I hit Save is completely unacceptable.

Not every time, but only when you going to share (or save to thumbdrive) this wiki. All other time changes will be stored to the database and any time you can comeback and continue work with wiki.
For example you export your wiki today (12:00 4/28/2017) then you continue modify this wiki but without exports (all changes collecting in localStorage) then suddenly the electric power goes out (16:00 4/28/2017) until next day (12:00 4/29/2017). You going to continue work with this wiki -> open exported document, content script injected to it, checks if this doc is tiddlywiki exported document (through meta tag), then if true check maybe there is new version of this wiki in localStorage, time of opened doc is 12:00 4/28/2017 but it see that in localstorage there is newer version 16:00 4/28/2017 so it immediately change the content of this doc to the new content from localStorage (not in file but only in DOM). And in some place you can show message says that time to re-export this wiki to file. So saveAs dialog will appear very rarely only before real export to file. Even if user close the browser he can do it without fear that his work will be lost all there, wait for him.
I think now you clearly understand needed workflow.

> (Also, TiddlyWiki isn't the only thing I'm thinking of which needs this kind of functionality...

TiddlyWiki is very alternative case IMHO. The idea is good but it's value in today world is exaggerated by you. It's cool toy, nothing more. Lets imagine how much time we will lost if this bugzilla would work on TiddlyWiki. I see only one case for it is personal organizer, but then it's not wiki. Wiki is knowledge sharing system. How you can share knowledge from file://...? No way! If you host it in http:// then you better to use normal wikis, because yes you (as owner of host) share knowledge to the people, but they can't share their knowledge through it so this is not wiki, this is static site.

> it's just the one that's hardest to hand-wave away by
> saying "Screw it. I'm turning it into a native app. I can embed Blink in
> PyQt if need be and use py2exe to bundle it all up for thumbdrive use on the
> go.")

I see you have much time, you are lucky :)
(In reply to jamy023 from comment #71)

> TiddlyWiki is very alternative case IMHO. The idea is good but it's value in
> today world is exaggerated by you. ... Wiki is
> knowledge sharing system. How you can share knowledge from file://...? No
> way!

You are correct that this isn't a very popular use case right now, but I don't believe you are seeing the utility of Stephan's examples.  There is a lot of being done to share data directly, filtered and processed through lightweight pieces of code; end users host their own copies of data and share in ways outside a centralized site, eg. IPFS.  Don't save in a DB and install complicated programs to map back and forth; work directly on the data.  This is an different paradigm, and it'll be great if Mozilla can stay relevant in that world... and even better if Mozilla designs a good solution and becomes the tool of choice.
(In reply to trent from comment #72)
> You are correct that this isn't a very popular use case right now, but I
> don't believe you are seeing the utility of Stephan's examples.  There is a
> lot of being done to share data directly, filtered and processed through
> lightweight pieces of code; end users host their own copies of data and
> share in ways outside a centralized site, eg. IPFS. 

I'm not against. Let's they can share! I just explaining to Stephan how to achieve this without direct file access. 

> Don't save in a DB and install complicated programs to map back and forth; work directly on the
> data.  This is an different paradigm, and it'll be great if Mozilla can stay
> relevant in that world... and even better if Mozilla designs a good solution
> and becomes the tool of choice.

You don't get me. There is a couple of DB included in browser. Not needed install any soft, just make TiddlyWiki firefox web extension little bit different from official TiddlyWiki (other words adopt to extension environment)

P.S. Firefox have enough features for comfort working with TiddlyWiki. But you (don't you trent) beg more and more like how Firefox is created exclusively for you or for desktop utility applications.
(In reply to jamy023 from comment #67)
> (In reply to fdsc from comment #66)

> Any who use filesystem to store data should adopt their add-ons to use other
> solutions. There are two choices:
> 
> - they adopt their add-ons
> - they not adopt and world lost their addons forever
>

I repeat. Database completely unusable. At many years. You leave only one choice: to cease writing the workers extensions. Because databases add-on is always off.
The defects are not closed for the year. Then it close. Sometimes without a well correcting. And then again it wait a year. And again correct. And again, new defects appear, and wait a year.
All this time, that is years and years, extensions are unable to work normally.
 
> So we should turn on creative thinking and don't be afraid to
> redesign old goods

The impression is that this is not need Mozilla for the authors of extensions but need authors for the Mozilla. This is not so.
(In reply to Stephan Sokolow from comment #70)

> A TiddlyWiki is a self-contained, structured document, more like a MS Word
> document in how it's saved, loaded, and shared.

It may be wise to abandon the browser altogether?
I see Mozilla porridge will not weld.
(In reply to jamy023 from comment #73)

> But you (don't you trent) beg more and more like how Firefox is created
> exclusively for you or for desktop utility applications.

Yes, FireFox made specifically for him. And specifically for me. And specifically for many people who ask for this functionality. And it is already done. FireFox have it.

But many, many and many jamy023 don't want it. They do not like. These jamy023 so much that here are quite a lot. And they, of course, smarter. Of course, it is for them FireFox has been exclusively created. We respect most many jamy023 and bow before them on his knees. Let them spoil FireFox, which used to be allowed everything, but now it becomes useless dummy.

Oh yeah, I forgot. There are a lot jamy023 who will use FireFox. But we, miserable meaningless singles, will not use it.
(In reply to fdsc from comment #75)
> (In reply to Stephan Sokolow from comment #70)
> 
> > A TiddlyWiki is a self-contained, structured document, more like a MS Word
> > document in how it's saved, loaded, and shared.
> 
> It may be wise to abandon the browser altogether?
> I see Mozilla porridge will not weld.

Also not feasible.

it's not negotiable that the TiddlyWiki must viewable in read-only mode on a machine with restricted application execution such as a corporate workstation, simply by tossing it on a thumbdrive or a GitHub Pages wiki and loading it in any modern browser.

Abandoning the browser for editing may be an option though, since it started requiring the TiddlyFox extension ages ago anyway. It all depends whether it's easier and more robust to:

1. Replace the browser altogether using something like NW.js (formerly node-webkit) or Qt's QWebEngine

2. Write a special HTTP daemon which serves the wiki up and grants it special filesystem-manipulation powers via a ReST API

3. Write a special HTTP daemon which grants multiple arbitrary Firefox extensions filesystem-manipulation powers by reinventing part of XULConnect user experience over an HTTP API.

To be honest, #1 isn't very appealing because of how I like to cram things together into a single tabbed window and I'm likely to do #3 rather than #2 out of sheer spite... but then I've always had that anti-authoritarian type of personality that George Carlin characterized as "Lead, follow, or get out of the way? I obstruct!"
(In reply to fdsc from comment #74)
> I repeat. Database completely unusable. At many years. You leave only one
> choice: to cease writing the workers extensions. Because databases add-on is
> always off.
> The defects are not closed for the year. Then it close. Sometimes without a
> well correcting. And then again it wait a year. And again correct. And
> again, new defects appear, and wait a year.
> All this time, that is years and years, extensions are unable to work
> normally.

The situation is changed very soon. All improvements for having one stable api.


> > So we should turn on creative thinking and don't be afraid to
> > redesign old goods
> 
> The impression is that this is not need Mozilla for the authors of
> extensions but need authors for the Mozilla. This is not so.

False. Mozilla and authors of extensions all both work for simple users (on same side). Today cyber security is very important, because some people play with TiddlyWiki, but other some even don't know about it and even don't want, and both using Firefox, and if for first (tiddlywikis funs) pc is tool for only sharing knowledge, but for others pc is tool for work where they keep secrets or money. We must protect the interests of most users. Just understand this

> But we, miserable meaningless singles, will not use it.

Then your place will be taken by others. And someone other will make extension for TiddlyWiki. Maybe it be useless, but it's more than nothing.

> 1. Replace the browser altogether using something like NW.js (formerly node-webkit) or Qt's QWebEngine
> 2. Write a special HTTP daemon which serves the wiki up and grants it special filesystem-manipulation powers via a ReST API
> 3. Write a special HTTP daemon which grants multiple arbitrary Firefox extensions filesystem-manipulation powers by reinventing part of XULConnect user experience over an HTTP API.

Using included in browser database is bad, and using out of browser localhosted app is good? Right? (facepalm)
NOTE: this is the second (and last) polite request, please pay attention.


    Hello fdsc, jamy, and Stephan,
 
    Thank you for your input, again.  As I already said, we have enough 
    information to properly evaluate the proposal.

    There is really no reasons for you to continue arguing the points
    anymore, so I _strongly_ encourage you to end this discussion, as it
    now seem totally unproductive.  Thank you.
(In reply to Oleksandr from comment #81)
> Tell me please, will WebExtensions support file renaming, writing to files
> and other operations with files?

No.
Hello:

I'm stepping in as a Moderator to remind you that commenting on bugs is subject to Bugzilla's Etiquette and Mozilla's Contributor Guidelines.

Arguing or disputing a decision made by a person working on a bug is a violation of our guidelines, so is abuse and bullying. 

Remember that people are making decisions about how to deploy limited resources against a deadline.

If you wish to advocate for or against a Web Extensions API, the place to do that is on the mailing list, not on this bug. 

https://mail.mozilla.org/listinfo/webextensions-support

Thank you.
Is there a reason for this list to only be available to list members? I'd think it'd be a good idea to allow add-on authors to find answers to their questions in the archives instead of letting them stumble around blind like it has happened in this bug.
Flags: needinfo?(ehumphries)
Blocks: 1322442
FWIW, ChatZilla uses direct access to local files for the following:

- Caching channel lists and some misc other data - I assume this can move to IndexedDB
- Downloading files using DCC - this needs a prompt-to-save API, which may be covered by the File/FileSystem DOM APIs
- Logging chat
- Loading scripts

For these last two, both key features, I don't see a way forward without this bug or something similar being created.

Logging chat involves creating new files in a user-chosen directory, to be written to over the course of a day, and possibly read back later. The only prompts that would acceptable here are when the user initially selects the directory.

Loading scripts involves recursively enumerating and reading files from one or more directories on startup, so again this will not work without a prompt-free means to do this on each startup.
No longer blocks: 1322442
Blocks: 1322442
(In reply to James Ross from comment #87)
> FWIW, ChatZilla uses direct access to local files for the following:
> 
> - Caching channel lists and some misc other data - I assume this can move to
> IndexedDB
> - Downloading files using DCC - this needs a prompt-to-save API, which may
> be covered by the File/FileSystem DOM APIs
> - Logging chat
> - Loading scripts
> 
> For these last two, both key features, I don't see a way forward without
> this bug or something similar being created.
> 
> Logging chat involves creating new files in a user-chosen directory, to be
> written to over the course of a day, and possibly read back later. The only
> prompts that would acceptable here are when the user initially selects the
> directory.
> 
> Loading scripts involves recursively enumerating and reading files from one
> or more directories on startup, so again this will not work without a
> prompt-free means to do this on each startup.

While I don't agree that it's sufficient, the argument has been made that all of that can be accomplished using APIs like IndexedDB.

Specifically, that the File API (ie. <input type="file"> or drag-and-drop) can allow scripts to be loaded into the DB to install them and the prompt-to-save API can be used to get logs and installed scripts back out when desired.
I think YF (Yang) was wrong here, removing the obvious problem-block because he just "thinks" it's ok to put or get "files" in or from "indexdb" instead the correct way.

Shouldn't the developer know better?


>Specifically, that the File API (ie. <input type="file"> or drag-and-drop) can allow scripts 
>to be loaded into the DB to install them and the prompt-to-save API can be used to get 
>logs and installed scripts back out when desired.
It's a bad idea to save logs only after manual user's clicks. Have you seen logs to work like that anywhere else? They ARE logs, and any other extension can also need to store some information at least in some allowed folder.

-------------------------------

Dear Firefox developers you have a lot of solutions that are good from the security perspective to release file API for extensions, e.g. a lot of ideas were mentioned in this topic. JUST DO IT.

JUST DO THE FILE API FOR EXTENSIONS, BECAUSE IT WAS ALWAYS AVAILABLE FOR DECADES IN FIREFOX.
It's necessary. And if it's not than no one will use it (and will put files in indexdb or any other creepy way), what is the problem?

Firefox is already changing from the best browser ever to a dumb one with super-limited extensions and you just pushing it further with this inadequate solution to use indexdb for files instead of files to the dir allowed once by user.

I don't want the best advanced browser to be another Chrome clone. :(

The issue is 2 years old, still unsolved, 37 upvotes and still nothing done! Except ridiculously forcing offers to use ugly walkarounds except correct way with file access.
(In reply to vleet from comment #89)
> I think YF (Yang) was wrong here, removing the obvious problem-block because
> he just "thinks" it's ok to put or get "files" in or from "indexdb" instead
> the correct way.
> 
> Shouldn't the developer know better?

I don't understand what you mean. As rewriting ChatZilla to a WebExtension, using a database instead of reading and writing files directly is a reasonable best practice, so it's not a block bug in my opinion.

> >Specifically, that the File API (ie. <input type="file"> or drag-and-drop) can allow scripts 
> >to be loaded into the DB to install them and the prompt-to-save API can be used to get 
> >logs and installed scripts back out when desired.
> It's a bad idea to save logs only after manual user's clicks. Have you seen
> logs to work like that anywhere else? They ARE logs, and any other extension
> can also need to store some information at least in some allowed folder.

Built-in a log viewer (can support save, highlight, etc.) is a viable solution. Direct reading and writing files may bring performance (IndexedDB support cache) and security issues, it is not a good practice for most add-ons.

About practice of convert ChatZilla, should be discussed in bug 1322442 rather than here.
(In reply to YF (Yang) from comment #90)
> (In reply to vleet from comment #89)
> > I think YF (Yang) was wrong here, removing the obvious problem-block because
> > he just "thinks" it's ok to put or get "files" in or from "indexdb" instead
> > the correct way.
> > 
> > Shouldn't the developer know better?
> 
> I don't understand what you mean. As rewriting ChatZilla to a WebExtension,
> using a database instead of reading and writing files directly is a
> reasonable best practice, so it's not a block bug in my opinion.
First of all, thank you for calm answer on my pushing reply.
Secondly, not everyone consider it to be reasonably bets practice, you should understand it. Why the filesystem exist available to the user at all? Why GNU/Linux and other *nix-like systems and recently Windows vouch for filesystem configs instead of some binary databases and like registers? Because it's simple, it's cleaner and more user friendly in different cases.

I'm not in position to have any strong vote here, but consider this - you're a extension developer and you want to store logs of your extension in the extensions special folder or user-allowed-once folder (like the secure could work). Why this feature souldn't be available to you? Any security risks? If yes - show these risks to us, if not - please declare to us that this mandatory feature (that was always available in Firefox!) WILL BE DONE FOR SURE.


> Direct reading and writing files may bring performance (IndexedDB
> support cache) and security issues, it is not a good practice for most
> add-ons.
For extensions that require output to the user as files, there is no faster and better way than output to files directly.

About security issues - can you please tell more? E.g. I propose it works this way:
Extension has it's special subfolder (like extensionname/files/) and it is allowed to write/read to it only after user approve during installation or activation of the plugin. But the whole folder and allowed once (optionally), not like click-every-time for every file.

And I see this security possible issues here:
1) The plugin can make some binary file or script with malicious code.
- But this code won't be executable from the filesystem perspective that support x flags, because extension is not allowed to chmod or chown obviously.
- There is nobody to run it anyway, otherwise the executing person/script could run some bash/cmd/powershell with malicious commandline command directly.
- The folder is specific (it's almost sandboxed by available to user to enter). So, it won't be like ngix's folder or something else.
So, no worries here.

2) The user puts some of his files (of make a symlink to an external folder) in this folder for some reason and the extension could read and steal the content of it.
- But it requires user to find this folder and do something with it. The only chance this happens if the user really needs the extension to work with files. So, it's necessary to make file api in this case anyway.
So no worries here too.

Any other security issues?

> About practice of convert ChatZilla, should be discussed in bug 1322442
> rather than here.
OK, I'm talking about the general point, that extensions should have this feature for sure, not only ChatZilla other, but all, that require it, FileAPI is must-have.
I agree on the general point that webextension APIs do *a terrible job* at interoperability with operating system primitives. This goes against several unix philosophies (everything is a file; one tool that does a job well and interacts with other specialized tools).
Filesystems are hugely important to keeping data organized. They provide the ability to have granular backups, integrity, durability, deduplication, compression, encryption, etc.
Siloing everything into indexedb will either require mozilla to reinvent the filesystem wheel or provide a much inferior experience.

My understanding is that durable writes (with WAL enabled) to indexedb are expensive, especially if one wanted to continuously append to a Blob (the straight-forward equivalent to O_APPEND writing to a file). While ZFS on the other hand would just CoW the last block of a file.

This very much smells like a combination of NIH and walled garden.
(In reply to The 8472 from comment #92)
> Siloing everything into indexedb will either require mozilla to reinvent the
> filesystem wheel or provide a much inferior experience.

https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Working_with_files#Store_files_data_locally_using_the_IndexedDB_file_storage_library :(

> My understanding is that durable writes (with WAL enabled) to indexedb are
> expensive, especially if one wanted to continuously append to a Blob (the
> straight-forward equivalent to O_APPEND writing to a file). While ZFS on the
> other hand would just CoW the last block of a file.

At least for ChatZilla, the logs are written whole lines at a time, so they would be separate rows in IndexedDB or similar, rather than a blob-per-file. But this is a non-trivial change for us developers to make and *especially* for users, because we would need to NIH a bunch of standard log operations like searching. It's the users I am really worried about here.
Nobody is trying to force anybody to use indexedDB when it isn't appropriate.
The filesystem certainly does make sense for some of the things in comment 87 but as mentioned in comment 88, a bunch of it is already possible today:
Chatzilla logs and downloaded files can be saved to disk using the downloads api -- you can either download them directly to the configured downloads folder or use the "saveAs" option to present the user with a file picker dialog to choose where they should go.
And as already mentioned, <input type="file"> or drag-and-drop can be used to get files into the extension.

(In reply to vleet from comment #91)
> E.g. I propose it works
> this way:
> Extension has it's special subfolder (like extensionname/files/) and it is
> allowed to write/read to it only after user approve during installation or
> activation of the plugin. But the whole folder and allowed once
> (optionally), not like click-every-time for every file.

This idea seems reasonable to me, but UX and security would need to take a pass over it to ensure that it can't be trivially abused by malware to take advantage of less savvy users.  This bug is quite long and unfocused, opening a new bug with that specific proposal would be a helpful step.

(In reply to The 8472 from comment #92)
> My understanding is that durable writes (with WAL enabled) to indexedb are
> expensive, especially if one wanted to continuously append to a Blob (the
> straight-forward equivalent to O_APPEND writing to a file). While ZFS on the
> other hand would just CoW the last block of a file.

Can you explain more about the use case you have in mind here?
(In reply to James Ross from comment #93)
> It's the users I am really worried about here.

There's more to that statement than you might have considered.

Since the extensions I'm working on are of the "I'm writing this to scratch my own itch, but I'll share them in case others may benefit" variety (as I suspect many are), my solution is to simplify development by doing two things which run counter to Mozilla's intent:

1. To simplify the develop-and-deploy process as much as possible (and to express my disapproval), I made the one-time effort to choosing a release channel and user.js config that disables signing enforcement. (Likewise, I added "--enable-easy-off-store-extension-install" to my Chrome launcher.)

That way, there's no online connectivity needed to permanently install an updated version into my browser and sending updates public will be as simple as `git push` once I reach that stage. (I'll just include basic instructions for disabling signing in the README and wash my hands of that aspect of things.)

2. If I want my Firefox to integrate with my desktop, I'm not going to jump through hoops. I'll just write a simple HTTP daemon which exposes the missing APIs, gated through local GTK+ permission prompts.

If I'm representative of even a small portion of potential Firefox extension developers, that could hurt Firefox's ecosystem a lot.
(In reply to vleet from comment #91)
> (In reply to YF (Yang) from comment #90)
> > (In reply to vleet from comment #89)
> > > I think YF (Yang) was wrong here, removing the obvious problem-block because
> > > he just "thinks" it's ok to put or get "files" in or from "indexdb" instead
> > > the correct way.
> > > 
> > > Shouldn't the developer know better?
> > 
> > I don't understand what you mean. As rewriting ChatZilla to a WebExtension,
> > using a database instead of reading and writing files directly is a
> > reasonable best practice, so it's not a block bug in my opinion.
> First of all, thank you for calm answer on my pushing reply.
> Secondly, not everyone consider it to be reasonably bets practice, you
> should understand it. Why the filesystem exist available to the user at all?
> Why GNU/Linux and other *nix-like systems and recently Windows vouch for
> filesystem configs instead of some binary databases and like registers?
> Because it's simple, it's cleaner and more user friendly in different cases.

Sorry if there is any emotional wording, I am not a native English speaker.

> I'm not in position to have any strong vote here, but consider this - you're
> a extension developer and you want to store logs of your extension in the
> extensions special folder or user-allowed-once folder (like the secure could
> work). Why this feature souldn't be available to you? Any security risks? If
> yes - show these risks to us, if not - please declare to us that this
> mandatory feature (that was always available in Firefox!) WILL BE DONE FOR
> SURE.

I understand your thinking now that you want to vote for this bug to implemented ASAP.

It is design-decision-approved for now, but the lack of an appropriate patch to reach 57. If it misses 57, the users will have to accept an breaking experience, so I think the files within database and export on demand is a good practice, it's also good for cross-browser implementations. But this thinking may be wrong, because I noticed that Chrome seems to have the file system APIs implementation.

> > Direct reading and writing files may bring performance (IndexedDB
> > support cache) and security issues, it is not a good practice for most
> > add-ons.
> For extensions that require output to the user as files, there is no faster
> and better way than output to files directly.
> 
> About security issues - can you please tell more? E.g. I propose it works
> this way:
> Extension has it's special subfolder (like extensionname/files/) and it is
> allowed to write/read to it only after user approve during installation or
> activation of the plugin. But the whole folder and allowed once
> (optionally), not like click-every-time for every file.
> 
> And I see this security possible issues here:
> 1) The plugin can make some binary file or script with malicious code.
> - But this code won't be executable from the filesystem perspective that
> support x flags, because extension is not allowed to chmod or chown
> obviously.
> - There is nobody to run it anyway, otherwise the executing person/script
> could run some bash/cmd/powershell with malicious commandline command
> directly.
> - The folder is specific (it's almost sandboxed by available to user to
> enter). So, it won't be like ngix's folder or something else.
> So, no worries here.
> 
> 2) The user puts some of his files (of make a symlink to an external folder)
> in this folder for some reason and the extension could read and steal the
> content of it.
> - But it requires user to find this folder and do something with it. The
> only chance this happens if the user really needs the extension to work with
> files. So, it's necessary to make file api in this case anyway.
> So no worries here too.
> 
> Any other security issues?
> 
> > About practice of convert ChatZilla, should be discussed in bug 1322442
> > rather than here.
> OK, I'm talking about the general point, that extensions should have this
> feature for sure, not only ChatZilla other, but all, that require it,
> FileAPI is must-have.

I presume this bug and APIs is an any directory (after approval) reading and writing instead of reading and writing under profile directory, this is not correct?

If it is the profile folder only, each add-ons will have a dedicated directory or according to apply for names?
(In reply to Andrew Swan [:aswan] from comment #94)
> Can you explain more about the use case you have in mind here?

This was about chatzilla logging. To me logs mean the files on the filesystem are written *live* into a preconfigured log folder (which obviously is not the downloads folder). So that I can |tail -f| them for example.
Prompting the user for each line is out of the question. And overwriting the whole log for each line is also highly inefficient and confuses things like |less|. O_APPEND and treating files as output streams exist for a reason.
(In reply to The 8472 from comment #92)
> My understanding is that durable writes (with WAL enabled) to indexedb are
> expensive, especially if one wanted to continuously append to a Blob (the
> straight-forward equivalent to O_APPEND writing to a file).

And that in a nutshell is why IndexedDB is always the wrong answer in Firefox. You have IDB implemented as a SQLite database with no tools or docs to access the data from outside the extension.

(In reply to Andrew Swan [:aswan] from comment #94)
> Chatzilla logs and downloaded files can be saved to disk using the downloads
> api -- you can either download them directly to the configured downloads
> folder or use the "saveAs" option to present the user with a file picker
> dialog to choose where they should go.

Sayin' it don't make it so. Please explain, step by step, how you'd append successive lines to a log file this way, and how you open the log after restart and continue appending.
I don't want to beat a dead horse, so please see comment 84.

This functionality will be implemented eventually, but not in the Firefox 57 timeframe, and likely not soon after. It's going to require a lot of careful thought to design and implement, and we don't have the time or resources for it at the moment. No amount of argument is going to change that.
(In reply to Andrew Swan [:aswan] from comment #94)
> Chatzilla logs and downloaded files can be saved to disk using the downloads
> api -- you can either download them directly to the configured downloads
> folder or use the "saveAs" option to present the user with a file picker
> dialog to choose where they should go.
> And as already mentioned, <input type="file"> or drag-and-drop can be used
> to get files into the extension.

For downloads and scripts, this can probably work, but it most certainly does not work for logs - unless you meant storing them in IndexedDB and providing a manual export-one-at-a-time function.

For those not aware, this use case involves writing to multiple files (one per chat channel), with a new file started each day and appended to throughout the day. I don't see any solution that prompts per-file to be workable here.

(In reply to Kris Maglione [:kmag] from comment #99)
> This functionality will be implemented eventually, but not in the Firefox 57
> timeframe, and likely not soon after. It's going to require a lot of careful
> thought to design and implement, and we don't have the time or resources for
> it at the moment. No amount of argument is going to change that.

Thank you very much for putting some timeline information in here! (I was partially concerned about doing an IndexedDB migration only to have a working filesystem API show up shortly after.)
(In reply to The 8472 from comment #97)
> (In reply to Andrew Swan [:aswan] from comment #94)
> > Can you explain more about the use case you have in mind here?
> 
> This was about chatzilla logging. To me logs mean the files on the
> filesystem are written *live* into a preconfigured log folder (which
> obviously is not the downloads folder). So that I can |tail -f| them for
> example.
> Prompting the user for each line is out of the question. And overwriting the
> whole log for each line is also highly inefficient and confuses things like
> |less|. O_APPEND and treating files as output streams exist for a reason.

Ah, I didn't get from the original comment that the requirement is about live logging, thanks for clearing that up.
The downloads API does not cover this situation of course, I don't have anything else to add to what Kris already said in comment 99.
(In reply to The 8472 from comment #97)
> (In reply to Andrew Swan [:aswan] from comment #94)
> > Can you explain more about the use case you have in mind here?
> 
> This was about chatzilla logging. To me logs mean the files on the
> filesystem are written *live* into a preconfigured log folder (which
> obviously is not the downloads folder). So that I can |tail -f| them for
> example.
> Prompting the user for each line is out of the question. And overwriting the
> whole log for each line is also highly inefficient and confuses things like
> |less|. O_APPEND and treating files as output streams exist for a reason.

https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Native_messaging is what exactly you want to, but don't understand it. Just mention (on the chatzilla extension page) that for "logging" feature users should install some manifest file manually. And also place the link to the Gist or so where users can download this file. And job done! If you don't understand how stdio can be redirected to the plain file this can help http://steve-jansen.github.io/guides/windows-batch-scripting/part-4-stdin-stdout-stderr.html
(In reply to jamy023 from comment #102)
> https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Native_messaging
> is what exactly you want to, but don't understand it.

Native messaging could be a temporary workaround for this application but it is not a suitable long-term solution.
(In reply to Andrew Swan [:aswan] from comment #103)
> ... is not a suitable long-term solution.

Why is not long-term? Can you explain me what this solution is missing?
(In reply to jamy023 from comment #104)
> (In reply to Andrew Swan [:aswan] from comment #103)
> > ... is not a suitable long-term solution.
> 
> Why is not long-term? Can you explain me what this solution is missing?

It isn't that something is missing, it is unnecessarily complicated and has unneeded performance overhead.
(In reply to Andrew Swan [:aswan] from comment #105)
> ... it is unnecessarily complicated ...

Complicated? At the side of webextension it is same complexity as writing to file. At the side of backend it is driven by OS pipes mechanism. What more simple solution would be? 

> ... and has unneeded performance overhead.

Can you prove that will be performance overhead? Even so, for security reasons it's not bad to little overhead. Again - it's a shared web platform, it's very important to protect main part of users than make easy life for webext developers (and also for hackers).
(In reply to Oleksandr from comment #107)
> Sorry, I do not understand the situation... :)
> 
> 1. Firefox decided to become like Google Chrome and forced all developers to
> leave XUL and go to WebExtensions

Yes. See also https://wiki.mozilla.org/Add-ons/Terminology.

> 2. WebExtensions is an almost complete copy of the API from Google Chrome

No, It is based on and draws on Google Chrome Extensions APIs, but improves. https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Chrome_incompatibilities. e.g. https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/proxy with a completely different implementation.

> 3. Google Chrome has methods for working with files:
> window.requestFileSystem || window.webkitRequestFileSystem
> 4. Google Chrome does not delete these methods and does not announce removal
> in the future
> 5. Why does Firefox not want to copy these methods from Google Chrome?

https://developer.mozilla.org/en-US/docs/Web/API/LocalFileSystem written:
"The specification is more or less abandonned, failing to get enough traction."
(In reply to jamy023 from comment #102)
> (In reply to The 8472 from comment #97)
> > (In reply to Andrew Swan [:aswan] from comment #94)
> > > Can you explain more about the use case you have in mind here?
> > 
> > This was about chatzilla logging. To me logs mean the files on the
> > filesystem are written *live* into a preconfigured log folder (which
> > obviously is not the downloads folder). So that I can |tail -f| them for
> > example.
> > Prompting the user for each line is out of the question. And overwriting the
> > whole log for each line is also highly inefficient and confuses things like
> > |less|. O_APPEND and treating files as output streams exist for a reason.
> 
> https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Native_messaging
> is what exactly you want to

Native messaging requires an external application. If one is going to install an external application then one might as well install a native chat client.

It is only useful for cases where you already have an external application and simply want to ship a companion addon for that application. The other direction, writing an addon and then requiring a native application to be installed just to get something to work that's about 2 of calls to libc functions to work is too much hassle for users.
(In reply to Oleksandr from comment #107)
> Sorry, I do not understand the situation... :)
> 
> 1. Firefox decided to become like Google Chrome and forced all developers to leave XUL and go to WebExtensions

Where you find this decision? Abandonment of XUL is just architectural reason related to transition to the multiprocess architecture (read transition to "Servo" engine).

> 2. WebExtensions is an almost complete copy of the API from Google Chrome

This because Chrome Web API is littlebit extended W3C standard API and not vice versa. 

> 3. Google Chrome has methods for working with files: window.requestFileSystem || window.webkitRequestFileSystem
> 4. Google Chrome does not delete these methods and does not announce removal in the future
> 5. Why does Firefox not want to copy these methods from Google Chrome?

"In April 2014, it was announced on public-webapps that the Filesystem API spec is not being considered by other browsers. For now, the API is Chrome-specific and it's unlikely to be implemented by other browsers and is no longer being standardized with the W3C." https://www.html5rocks.com/en/tutorials/file/filesystem/ sorry, maybe there is more trusted source, but this one explained this more clean way.
(In reply to The 8472 from comment #109)
> Native messaging requires an external application. If one is going to
> install an external application then one might as well install a native chat
> client.
> 
> It is only useful for cases where you already have an external application
> and simply want to ship a companion addon for that application. The other
> direction, writing an addon and then requiring a native application to be
> installed just to get something to work that's about 2 of calls to libc
> functions to work is too much hassle for users.

One more time: This is only example how they can deliver fullfeatured application through webextnesions if they like to. WebExtensions IS NOT full featured framework or toolkit. And if there needed to store data then do it through IndexedDB or localStrorage. If needed to send some data outside of browser then pls use NativeMessaging. I am mentioned about nativemessaging comment #102 because dude wants to see logs through "tail -f" command so I am post example which shows that this possible.
Restrict Comments: true
Just a follow on to explain why comments are now restricted on this bug. 

There were multiple requests to stop commenting on this bug, for example comment 55 and comment 79. At this point this bug has ceased to be providing useful information.

The fact that Firefox has moved away from allowing file system access in extensions is something that we know affects some add-ons and their users. However, it can't be stressed enough that file system access is a security concern and something Mozilla is trying to limit through initiatives like the sandboxing initiative: https://wiki.mozilla.org/Security/Sandbox#File_System_Restrictions. We've documented some challenges here: https://wiki.mozilla.org/WebExtensions/Filesystem.

We've provided a library for storing files in indexeddb https://www.npmjs.com/package/idb-file-storage, but for Firefox 57 there is no plan to provide more file system support in WebExtensions.

We understand that improved file system support is a desirable feature and we will continue to explore options to provide this functionality in a safe and secure manner as the Firefox team plans its future releases.
Flags: needinfo?(amckay)
Flags: needinfo?(amckay)
Priority: P2 → P3
Depends on: 1449775
Product: Toolkit → WebExtensions
Blocks: Session_managers
See Also: → 1413525
See Also: → 833792
Blocks: IndexedDB-SM
No longer blocks: Session_managers
Type: defect → task

Redirect a needinfo that is pending on an inactive user to the triage owner.
:mixedpuppy, since the bug has recent activity, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(billjm05) → needinfo?(mixedpuppy)
Severity: normal → S3
Flags: needinfo?(mixedpuppy)
You need to log in before you can comment on or make changes to this bug.