Closed Bug 797443 Opened 7 years ago Closed 6 years ago

let some webapps access the local file system (enablePrivilege equivalent)

Categories

(Core :: DOM: Core & HTML, defect)

defect
Not set

Tracking

()

RESOLVED WONTFIX

People

(Reporter: fb+mozdev, Unassigned)

References

Details

With the removal of enablePrivilege, the most widely criticized to be removed functionality was full file system access as it is sometimes required for certain webapps to access the local file system in the way they were written. 

Though I do not agree that a browser should expose file system access (it is not an OS-like platform in the traditional sense), I think we can agree on a solution that works for everyone: Browser vendor incl. security, developers, users and non-users of said apps. 

My proposal consists of 
a) a DOM API extension, 
b) a hidden boolean pref and a hidden string pref, 
c) a specific dialogue design, 
d) an optional AMO-hosted addon that can change those prefs. 


This is how I envision it to work: 

a) Local file system access is requested with "window.mozRequestLocalFilesystemAccess()" (only works in a Worker??). This writes a message to the console (something like "Do not rely on this API. You should not access the local filesystem.") and returns an instance of OS.File [1]. However, this method throws by default except certain conditions met: 

b) There is a boolean pref disabled by default named something scary like "extremely_risky_web_content_access_to_full_file_system.not_blocked". However, this pref itself does not allow web content to access the local file system. The domain from which the script calling mozRequestLocalFilesystemAccess() originates [2] MUST be added to the comma-separated pref "extremely_risky_web_content_access_to_full_file_system.allowed_sites" (wildcard not allowed but allowed from 3rd level on, e. g. *.domain.tld). 

[2] The web page containing the script that requests full file system access MUST include the script from the same domain as the web page itself as well as MUST NOT include any files from any other domain (e. g. is not allowed to include an image from evil.tld). 

c) The first time a web app from a specifically allowed domain requests local FS access, a doorhanger notification with a yellow warning sign appears asking the user (with a 3 sec wait time ??) to allow or deny the FS access (like Geolocation). If the user clicks "Allow", this decision is stored in website preferences and the request succedes. If the user clicks "Deny", the domain is automatically removed from the hidden pref and the request throws. 

The first time an allowed web app uses an OS.File API that ALTERS the file system (or reads files ??), another doorhanger notification with a yellow warning sign appears asking the user for permission to allow ALTERING files in the FS. If the user clicks "Allow", this decision is stored in website preferences and the request succedes. If the user clicks "Deny", the domain is automatically removed from the hidden pref, the flag allowing FS access in the website preferences is removed and the API call throws. 


If any condition fails, the API throws. 


d) The Mozilla team can optionally create an experimental, AMO-hosted extension which allows altering the two prefs, with a checkbox enabling local FS access as well as a field to add and remove allowed domains. No other AMO-hosted extension is allowed to change those prefs. (If a dev decides to write his own extension, he could easily integrate a FS access API on his own, bypassing AMO.) 


Telemetry should be added to track the usage of this API. This could eventually help in removing this functionality (providing a list of pages where this API is used) and could identify bad guys™ that can be added to a blocklist. 

Additionally, this functionality could also be limited to signed scripts. 

It would be helpful to land this bug in Fx 18 (or 17?) thus devs relying on local FS access can continue to use this functionality short-term. IMHO programming it should not be too complex as the actual API is already there. 


[1] I believe providing this API is sufficient: https://developer.mozilla.org/en-US/docs/JavaScript_OS.File


Do you have any suggestions or questions or objections?
Depends on: 757046
Would your proposal allow filesystem access to a web page hosted
on the local computer (i.e. opened as file://...)?

Secondly, if telemetry means that mozilla.org is notified all the times
I access one such page, then this means keeping information regarding
the surfing of users. I think this would not be acceptable.
If you only want to access files on your computer from a local webpage I'd suggest you write a simple Node.js server (e. g. with jsDAV, it's just a couple of lines). This is a much cleaner, simpler solution, plus it's easier to maintain and you don't need to worry about Mozilla removing this functionality. 

Otherwise, there could be yet another pref to allow this functionality for local webpages (but including the notifications, IMO, and remembering the acceptance per file). However, this is a bit too complex. 

For telemetry, I suggest to send only the cleartext domain name once per user. If this is too much of a privacy issue, I advise to send a hashed representation of the domain name rather than just "pinging". Of course, the doorhanger notification should provide a short sentence to explain that Fx will send a message to Mozilla containing the requester domain name.
The reason why I'd refrain from allowing *local webpages* to allow local FS access when the pref is enabled is: 

If someone sends you an HTML file e. g. via e-mail attachement, the sender can easily read out (or alter) your entire FS and send it somewhere else. Thus you are more susceptible for targeted attacks (e. g. social engineering) which is especially harmful for developers (e. g. in leading positions – yes, they are susceptible to such attacks, too) that enabled this functionality.
Hey Florian,

Nice! I really appreciate you fleshing this out in such detail.

I think Gecko is unlikely to implement such a patch natively. But your suggestion could be very easily implemented perfectly as a jetpack addon (with a content-script that injects mozRequestLocalFilesystemAccess into the appropriate scopes), and I think it would be accepted on AMO given the amount of security precautions and defense-in-depth you're proposing here.

Moreover, implementing it as an addon means that it will Just Work with all releases of Firefox, including FX17 (which is already on aurora and will soon be on beta, meaning that it's too late to implement any new features).

Are you (or someone you know) interested in writing such an addon? I or others can definitely mentor such a project.
Thank you. 

I think if such an addon will be accepted on AMO, it should be no problem to implement this completely with an addon. However if it is not accepted, I don't think it will be an appropriate solution for the devs that need it. 

May you please check back with the AMO team if this functionality packed into an addon will be accepted on AMO? 

As for writing it, I have neither the knowledge nor the time to do it (as far as I can say for the next few weeks). Someone else (who really needs it) should write this.
(In reply to Florian Bender from comment #5)

> May you please check back with the AMO team if this functionality packed
> into an addon will be accepted on AMO? 

I just CCed jorge on this bug, and he'll tell us.
I find myself deeply perturbed by these developments. I had created a HTML 5 web app that permitted game design with a focus on ease of use for users (within reason). I know that there are those who were always uncomfortable with Firefox serving a dual-purpose, but these had never carried past days and I find it curious that establishment, anti-change forces are gaining the upper hand at a progressive, change-oriented organization. There has been talk of developing add-ons/extensions to fill the need for local file access (as with "AsYouWish")... those efforts have not materialized due to severe memory leaks introduced in recent Firefox versions.

At this time, there appears to be no browser that will permit local file access by offline pages out of the box. I will not heed the misguided suggestion that users be asked to run their own private web servers to use a web app when not connected to the internet. The whole reason for making the app in Firefox was portability. Firefox was the only truly ubiquitous platform... even more so than Java, as is well known.... Now that is at an end and there is no will at Mozilla to restore it. The push to effect this platform -- the passion behind it -- has been blocked. It is more than a questionable decision... it seems anti-progressive in that it has chained the potential of the people to established interests. There must have been a motive, or motives, that have not been disclosed. It seems clear to me that those who hate Mozilla have at last succeeded in penetrating its upper echelons, to the point that the dreams of Firefox's founders are no longer served. Blake Ross' parting rebuke to Netscape could today be served just as well to Mozilla... it's gotten to the point that Mozilla can no longer be distinguished from Google, can no longer be distinguished from Microsoft. Users are no longer first... the protests made previously are apt... Mozilla's current leadership is more concerned about its reputation than anything else. Winning at all costs, even if the cost is losing. That's the road Mozilla is on today.

I bought a phone recently. Put Firefox on it right away. Was so excited that I'd finally live my dream of game design on a handheld device. Totally disappointed. Still motivated though. Will never put up with the motions of making an extension that I have to update every couple months. Will go a different road. This is war. The enemy enjoined. en garde.
There is a WebAPI (https://wiki.mozilla.org/WebAPI) that may help here: DeviceStorage (https://wiki.mozilla.org/WebAPI/DeviceStorageAPI). 

Maybe someone is interested in investigating if the current API is suffcient or if we need something here to allow more use cases. I don't know how the internals work here (e. g. what type==music resolves to, and what type==documents or type==repository or type==arbitrary resolve to). 

Either way, I guess most use cases should be covered by this API. It won't help you change system files, but that's not what Firefox is supposed to do anyway. If you need to handle multiple files for "one" document (e. g. a video editing suite project file which contains film strips,
sound, index and other files) you can bundle them into an archive (like Max OS X does) and use the Archive Web API.
That'd be all well and good, but it's not there now.
(In reply to tcaudilllg from comment #9)
> That'd be all well and good, but it's not there now.

What exactly do you mean?

I'd suggest you start accurately stating what you are missing and what you need instead of writing poetry, so we can actually continue this quest. Words won't help us, action will.
Well let's look a what the app does:
- it loads game project files, which specify a list of maps to load concurrently. For example, a game may use several different interfaces which are spread out over multiple maps. So, I need to read project files to know how many interfaces there are and on which maps they start.
- I need to read the map files as users enter and leave play areas.
- For the editor, I need to load and save project and map files. For user convenience, I also need offer the ability to load and create game character files, to promote character re-usability. Users would also benefit from an automatic HUD generator, which would be enabled by reading canvas data and saving it to PNG.
- I need configuration file storage for various reasons.

localStorage takes care of the config requirements no problem. For the other files, indexDB would work in theory but, as is a matter of portability it's impractical. Users need to be able to exchange project, map, and character files. As such, only file system access through some means will suffice. ...Actually it would probably be OK if say, there were an option that users could use to export an indexDB entry to the OS file system as an independent file and import it back in. You'd have to make an interface just for this, in Firefox.
Actually that's sounding really unworkable. Users would have to have in-depth knowledge of the app's data structure to know what they were doing when importing and exporting from databases. Anyway, I'm thinking that makers of other distributions might differ with you on enablePrivilege, and because I need that now I guess that's what I'll have to do UNLESS you guys promised to have the file system done by say, next month. Realistically, we're looking at next year before file access is restored in Firefox, at the earliest, right?
I noticed you're taking a very hard line with device storage. I advise against that... your supporters will not emulate Google. If they wanted Google's way, they would have gone that way. You are really kinda treading on people's freedoms at this point and there will be a backlash, from users at best or Anonymous at worst. The certification regime for Firefox OS is a total non-starter... hell I don't even use Google Play for most of my apps. I'll bet you lose half your market share over this to a competitor that is now not even in the rear-view mirror, but will be right on your tail before you know it. I guess you don't care... as long as you've made a niche for yourselves with some people, that's all you aspire to.
Don't go this road. Be the reckless alternative. Give users total freedom. You'll dig into Android's share and still be able to grow. If you go the road of freedom, I will support you.
The road you're going now is WORSE than MS. That is saying something.
Comments 13-15 are unproductive and off-topic.  Please refrain from violating sections 1.1 and 1.2 of https://bugzilla.mozilla.org/page.cgi?id=etiquette.html
Howdy folks,

Does my AsYouWish addon (https://github.com/brettz9/asyouwish/) meet peoples' needs here?

tcaudilllg mentioned severe memory leaks in recent Firefox versions. As noted at https://github.com/brettz9/asyouwish/issues/3#issuecomment-14213881 , there are significant improvements in memory problems my AsYouWish addon is now seeing due to incorporating the latest SDK upgrade, and it will hopefully keep improving in future Firefox/SDK versions; code reviewers are also welcome to see if there are any issues on my addon's side.

Although AsYouWish is doing things a little differently than Florian's suggestion:

* using the SDK API instead of the DOM API (as I'm not sure it would even be possible for an addon to override or add functionality to the DOM APIs let alone desirable)
* allowing you to enable or disable requests from the file protocol, or to let you whitelist specific files preventing sites from having the ability to even ask you for permission without it (unless you configure AYW to allow any site, or a site complying with your protocol whitelist to be able to ask).
* not preventing other cross-domain files from being used in the same webpage (I feel that would be unnecessarily restricting developer freedom--though I am open in theory to controls on this, if the capability can be maintained, since I know the capability is otherwise multiplying the dangers of XSS vulnerabilities)

...the only feature I see that Florian mentioned that AsYouWish does not handle now is granularity on the type of file system access (e.g., read access, delete access, etc.)--something I agree would be useful (just as I hope to see AYW implement granularity with SHARED and namespacable local storage). Feel free to start an issue on it, or if you prefer, there is the AsYouWish tracking bug 823811. AYW does support granular access to the APIs, however, so you can restrict AYW to only allowing file access, as opposed to say tab access.

I very much empathize with the statements in comment 7, "Firefox was the only truly ubiquitous platform"--including for the reason of its use of client-side JavaScript for its universality, ever-growing capabilities, and relative ease for development and deployment. When my daughter grows up a little and asks me, "How can I show a list of local files in my web program?"--if she is not learning it already in school--I hope I can give her a simple and affirmative answer. (I don't mind explaining why the ability is not available to users by default, but I hope she won't have to deal immediately with the hassle of installing a server, learning another language, etc. etc.) I also hope I don't have to say that she has to program things differently (or not at all) if her users are on a mobile device, etc.

I also empathize with the sentiment behind the statement "Will never put up with the motions of making an extension that I have to update every couple months". I really hope the SDK team and Firefox in general takes to heart the desire of developers to build applications that can work indefinitely. There seems to be this attitude that just because the currently active developers are admirably always straining to implement every latest feature or is terrified of being left behind in adopting the latest best practices and smooth APIs, that therefore it is impossible that a solid, basic web application cannot be built once and left alone. I thought HTML5's reason-for-being was about compatibility, and even there I see people talking about dropping setAttributeNode or what not just because these developers think it is not useful. Some things should be sacred, man, and non-beta APIs are, imo, one of them. It sounded like the high-level SDK APIs were meant to do this, but I thought I had seen some changes or other were taking place there on non-experimental APIs?

Anyways, I sympathize very much with tcaudilllg's frustrations, and hope we can all be productive in recognizing their source, even while encouraging practical steps beyond them.

I specially agree with the sentiment that file and database storage is a touchstone of browser fidelity to user freedom (in contrast to those seeking to confine you to their cloud), particularly when applications can share such formats readily beyond their own site, and without requiring implementation of one-off custom postMessage APIs. The power of such an ability to share data on the client-side with different consuming applications in a namespace-independent manner ought to already be demonstrated sufficiently by this humble protocol called HTTP and the power of a site-independent uniform API being amply shown by this little language called HTML. :)

But instead of being possible for any browser to freely read pages, there is a need for any page to freely read its browser, or rather, to freely read sharable data supplied in that browser regardless of source. Just because one is technically "free" currently to migrate from one feudal web lord to another (even if that lord carves out its fiefdom in a restricted part of your local machine) does not mean one is actually free unless such an ecosystem is changed to support content and data being truly portable, first by technical capability--e.g., so a developer can add arbitrary data formats or content in such a manner that other sites can be allowed to discover and read the same data without needing to opt into communicating with my specific site--and THEN by standards and optional conventions.
If Brett's extension does not cause the browser to crash, then I guess it's suitable. I imagine it could be packaged in with (unofficial) distributions even to prevent the user having to install it themselves, cause that's just a hassle. I usually leave my browser open until it crashes. Honestly it's just about the only program on my PC I use nowadays, other than notepad and a few paint proggies (and if it were possible I wouldn't even use those). I really think there is an opportunity here to obliterate the practice of native app design completely and make the OS an invisible underpinning to the browser. I saw this on a Linux distro a few years ago and I liked the concept. But we're not to the point yet where everyone is always connected, and working with localhost is a pain for developers and users alike. We need device storage access and it needs to be suitable to user requirements.

I take umbrage that my advice was implied to have breached etiquette. This is not the response I expect as a consumer of Mozilla product. Please show more respect and courtesy to me and other Mozilla consumers in the future.
(In reply to tcaudilllg from comment #18)
> I take umbrage that my advice was implied to have breached etiquette. This
> is not the response I expect as a consumer of Mozilla product. Please show
> more respect and courtesy to me and other Mozilla consumers in the future.

Telling us that we're worse that Microsoft and we're going to be targeted by Anonymous is not "advice", it's a great example of a pointless comment.
(In reply to tcaudilllg from comment #18)
> If Brett's extension does not cause the browser to crash, then I guess it's
> suitable. 

I haven't seen any crashes. An issue was reported in which opening a window 100 times increased memory usage to make things unusable, but that was before the SDK release, and on my machine at least, there is only a doubling of memory (and I hardly ever open more than a couple windows in a session anyways, preferring tabs).

My request from Mozilla (beyond a review for potential distribution on AMO), is that, if, in the vein of Bobby's encouraging words to offer to "mentor such a project", Mozilla can keep in the back of its collective mind our desire to continue wrapping the SDK APIs without breakage for our own users, and just in general the idea that people are interested to use the APIs from websites, that would be a big help.
As soon as the SDK is integrated into the browser, I guess there won't be any big changes to its API anymore (except new APIs). However, from time to time it is necessary to change APIs to integrate a necessary change in the program architecture: E. g. while introducing per-window private browsing, the existing APIs needed to be replaced as they are incompatible with the new approach. Same applies to async-ing the Places library: To make Firefox be more responsive, thus to remain competitive, the synchronous APIs had to be removed (those weren't in the SDK). The SDK tries to remedy the problem of changing, internal APIs, however this is impossible prevent. 

I can't speak for Mozilla here, but that's my perception. 

Concerning tcaudilllg's problem: this can absolutely be solved by already present APIs without requiring the AWY hack. For your app, you are required to specify a new data format nonetheless, so why not use some ZIP'ed JSON + data fragments? Mozilla Popcorn works similar, too!
This appears to be fruitless. I'll make a decision based on my own interests whether or not to implement the offered remedies, as all of them are inferior to the prior method (which was itself markedly imperfect). My passion for Firefox is gone... how far you have fallen, Mozilla. My part in this conversation is over.
I have implemented my apps using AsYouWish to gain access to the local
filesystem. It works perfectly: I can read and write files much the
same as native apps can do.
I have just downloaded/tested the demo files for AYW on Github. Simple storage demo and file demo. Firefox 19, opened just after download from the downloads manager. AYW was installed (restart not required).

Results: "Error: denied"
For the sake of getting accepted to the AMO site, I added the condition that the addon be at the (nearly) strictest security settings by default. You should open the AsYouWish dialog to configure it more loosely or as desired, or otherwise it won't even be able to ask you for permission. Please report an issue at AsYouWish if you are not able to get it working beyond that advice--it should be working, most certainly with the provided demos. And thank you for everyone who has tried it out.
Firefox -> Options: check the item "Add-on bar". An icon will appear on the bottom-right corner of the Firefox window. Click on it: the AsYouWish config
panel will be displayed. There you can tell that your app is entitled to ask
permission to access the local filesystem. Then, when you open the app the
first time, a popup will appear asking you to grant the permission.
If you grant it, then the app will be allowed to access it (and your decision
will be remembered).
I just clicked the check in the AYW properties to permit unconditionally all allowed protocols (file and https have been specified), and that appears to have broken the extension.

Now instead of processing the script, both pages produce the following error:

"page error: [Exception... "Not enough arguments [nsIContentPrefService.getPref]"  nsresult: "0x80570001 (NS_ERROR_XPC_NOT_ENOUGH_ARGS)"  location: "JS frame :: resource://gre/modules/XPIProvider.jsm -> jar:file:///C:/Documents%20and%20Settings/Tony/Application%20Data/Mozilla/Firefox/Profiles/xk6s25mx.default/extensions/jid1-9Ddk7IgYYaBCAw@jetpack.xpi!/bootstrap.js -> resource://jid1-9ddk7igyyabcaw-at-jetpack/addon-sdk/lib/toolkit/loader.js -> resource://jid1-9ddk7igyyabcaw-at-jetpack/asyouwish/lib/SitePrefsManager.js :: get :: line 30"  data: no]"
Second column of the panel: grab current website, then: allow on visit.
I don't see anything like that. I just see text fields where I'm supposed to specify javascript arrays of the sites to allow. No one but systems admins can be expected to use this.
I'm sensing a conflict of interest here... if Mozilla is relying on its apps store (essentially an add-ons depot) to fuel adoption of its mobile OS, what incentive do they have to enable users to get the same functionality from web sites?
> Firefox -> Options: check the item "Add-on bar". An icon will appear on the
> bottom-right corner of the Firefox window. Click on it: the AsYouWish config
> panel will be displayed. There you can tell that your app is entitled to ask
permission to access the local filesystem. Then, when you open the app the
first time, a popup will appear asking you to grant the permission.
If you grant it, then the app will be allowed to access it (and your decision
will be remembered).

Nevermind I see it.
And it still doesn't work. I still get that error. I've done everything right, but, that error.

Gonna make my own build using parts from Firefox 14 and 19.
Please also use the XPI file from https://github.com/brettz9/asyouwish/ . The one on AMO does not work with the current version of Firefox (but it is still there so that I can get a preliminary review without being reset many slots in the queue).
How exactly do I access the AsYouWish API in my app?
I managed it. Thanks Brett. This solution, which gives access to the entire add-ons SDK, is quite powerful. I was able to fix my broken XPCOM code that had relied on enablePrivilege with two lines of code. My game maker works again. :D On Windows, at least... still doesn't work on Android, and neither does AsYouWish: Android flat-out refuses to install it.
Component: DOM: Mozilla Extensions → DOM
Seems like AsYouWish addon fixes most issues that arose with the removal of enablePrivileges. WONTFIXing, please reopen if you feel the need to implement my proposal. 

Anyone else longig for full file system access, look out for WebAPIs and a new FileSystem API proposal in the works (though I don't know if this will ever give web developers full file system access).
Status: UNCONFIRMED → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
Component: DOM → DOM: Core & HTML
You need to log in before you can comment on or make changes to this bug.