1247329 - Implement WebSQL as a WebExtensions interface

Status: RESOLVED WONTFIX

(WebExtensions :: General, defect, P5)

Description

[Nancy Grossman]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:44.0) Gecko/20100101 Firefox/44.0
Build ID: 20160125134036

Steps to reproduce:

Chrome implements the asynchronous API of WebSQL, which is available to Chrome extensions though it isn't an extension API per se. Firefox doesn't implement WebSQL, but add-ons have been able to connect to SQLite databases via the Storage API. The Storage API, or something like it, should be exposed as a WebExtensions interface, perhaps grafted to chrome.storage.

WebExtensions should be able to pass a filesystem path as 'name' to .openDatabase() and get back a connection to that database if it exists. This allows an existing Firefox add-on to continue using the same database after it's massaged into a WebExtension.

However, it also allows one WebExtension to fiddle with another WebExtension's data. That sounds bad, but maybe not. Add-ons can already do this, and the sky hasn't fallen. SQLite Manager does this, and that's good. 

I don't know what should happen if 'name' is not a filesystem path, or if it looks like a filesystem path but no such file exists. In Chrome the 'name' is just an opaque string that Chrome maps to a filesystem file of its own choosing, so maybe in these cases the WebExtensions implementation should do the same.

Comment 1

[Kris Maglione [:kmag]]

We definitely won't be allowing extensions access to arbitrary filesystem paths in that way.

At the moment, we aren't planning to add support for WebSQL either. The primary standards-based alternative is IndexedDB, which is also available in Chrome, so I suggest that you try migrating to that, if possible.

There's a non-zero chance that we may make WebSQL available to extensions in the future, but we'd like to avoid it if at all possible.

Comment 2

[Nancy Grossman]

(In reply to Kris Maglione [:kmag] from comment #1)
> At the moment, we aren't planning to add support for WebSQL either. The
> primary standards-based alternative is IndexedDB, which is also available in
> Chrome, so I suggest that you try migrating to that, if possible.

"Aren't planning" as in "planning not to", or as in "hadn't really thought about it"?

IndexedDB is a flat database, a collection of independent tables. It is decidedly not a substitute for a relational database that can natively retrieve structured data from 2+ tables grafted together according to well-defined relationships between fields. IndexedDB is OK for something like cookies.sqlite or formhistory.sqlite. It's not at all suitable for something like content-prefs.sqlite or places.sqlite. What you're suggesting is that an extension developer hack together an ad hoc RDMS with IndexedDB whenever they need storage with more structure than a simple table. Such a bad idea - how many will get it right the first time? Kiss data integrity goodbye. If Mozilla wouldn't do it in production code, then it shouldn't suggest that extension developers do it. (Not that we're less capable, but as a group we're not.)

I'm not keen on the WebSQL wrapper (in particular, it prevents enforcement of foreign key constraints), but if not WebSQL then there really must be some other RDMS interface available to WebExtensions. Chrome already implements WebSQL as a Web API, which some Chrome extensions already depend upon, WebSQL isn't far removed from Mozilla's own Storage API, which some Mozilla add-ons already depend upon, and the interface is well-defined with an existing implementation, which Mozilla's WebExtensions strategy depends upon. It seemed a natural, but exposing the Storage API would be fine by me.

In the context of WebExtensions, IndexedDB is no more "standards-based" than Chrome's subset of the WebSQL interface. WebExtensions itself is not "standards-based" but a subset of proprietary Chrome's APIs. In this context Chrome's APIs *are* the standard. Omitting the API simply makes for a poorer copy. (Which is not a bad thing if a better API is available.)


(In reply to Kris Maglione [:kmag] from comment #1)
> We definitely won't be allowing extensions access to arbitrary filesystem
> paths in that way.
> 

Not arbitrary paths. I had two things in mind there. First, I'd like to pass a token representing a local file chosen by the user to openDatabase() and connect to that database. For example, passing an HTML5 File object should get me at least a read-only connection to the file, and preferably a writable connection (because extensions are vetted and in some sense more trustworthy than dirty old websites).

I'd also like to persist that token, so that a user can choose a file to be the working database (as in Stylish) or choose from a MRU list (as in SQLite Manager). 

Second, unless a WebExtension can pass in a filesystem path and get back a database connection, an existing Firefox add-on will lose access to its data if it's converted to a WebExtension, because that's the way a Firefox add-on opens a SQLite database today. Not an issue for new databases, not an issue specific to SQLite databases, sorry I even mentioned it.

Comment 3

[Kris Maglione [:kmag]]

(In reply to Nancy Grossman from comment #2)
> "Aren't planning" as in "planning not to", or as in "hadn't really thought
> about it"?

Have thought about it and aren't currently planning to. WebSQL and IndexedDB
both went through the standardization process, and IndexedDB is the only one
that came out. Chrome happens to still implement it, but that doesn't really
have any relationship to its extension API.

> IndexedDB is OK for something like cookies.sqlite or formhistory.sqlite.
> It's not at all suitable for something like content-prefs.sqlite or
> places.sqlite.

A lot of people have actually been pushing to replace both Places and content
prefs with NoSQL databases. The only reason it hasn't happened yet is that
it's been beaten out by other priorities.

> What you're suggesting is that an extension developer hack together an ad
> hoc RDMS with IndexedDB whenever they need storage with more structure than
> a simple table.

I'm not suggesting that. I don't know how many extensions need relational
databases, or how many have problems that a relational DB would solve better
than IndexedDB. I certainly haven't seen many. But for the rare cases that do,
I'd probably suggest that they use a library that handles it for them rather
than roll a relational DB themselves.

> In the context of WebExtensions, IndexedDB is no more "standards-based" than
> Chrome's subset of the WebSQL interface. WebExtensions itself is not
> "standards-based" but a subset of proprietary Chrome's APIs.

WebExtensions is on the road toward standardization. IndexedDB is a web
standard. WebSQL is a failed web standard that Chrome exposes both to the web
and to its extensions.

> Second, unless a WebExtension can pass in a filesystem path and get back a
> database connection, an existing Firefox add-on will lose access to its data
> if it's converted to a WebExtension

There's a bug about creating a data migration path for add-ons being converted
to WebExtensions. We haven't decided what that's going to look like yet, but
it's going to have to be an actual migration path, not ongoing access to the
old model.

Comment 4

[tofumatt [:tofumatt]]

Hi Nancy, thought I'd chime in and address your concerns about IndexedDB.

I think you're totally right that IndexedDB's very low-level API is not a replacement for WebSQL or any relational database. I think, sadly, IndexedDB was advertised by browser-vendors as a database for developers but it's really an API to BUILD database APIs... see things like PouchDB (https://pouchdb.com/), localForage (https://github.com/mozilla/localForage), and db.js (http://aaronpowell.github.io/db.js/). They provide APIs on top of IndexedDB, including some that have very natural query APIs for people used to relational databases.

Because I don't see it as likely that Mozilla implements WebSQL, I think finding a library on top of IndexedDB is your best option, but not using it directly. IndexedDB was always intended to be a low-level API.

Comment 5

[Nancy Grossman]

(In reply to Kris Maglione [:kmag] from comment #3)
> WebSQL and IndexedDB both went through the standardization process, and
> IndexedDB is the only one that came out.

I don't believe that's relevant. The context here is WebExtensions. The W3C standards you allude to are recommendations for web-facing APIs. "We [Mozilla] think SQLite is an extremely useful technology for applications, and make it available for Firefox extensions and trusted code. We don’t think it is the right basis for an API exposed to general web content." [1] What's proposed is not a web-facing API, it's an API exposed to extensions that provides continuing access to an extremely useful technology.

> WebExtensions is on the road toward standardization.

The only cross-platform standard for extensions, a de facto standard, is Google's Platform APIs, and WebSQL is a part of it [2]. Every major platform except Internet Explorer now writes to that standard, and every one of those except Mozilla implements WebSQL. If extensions are to be substantially portable, then we need a WebExtension alternative to Chrome's WebSQL API. "[A]s far as possible, extensions written for those browsers should run on Firefox with minimal changes." [3]

Mozilla add-ons have had use of SQLite via the Storage Service for over a decade, and Chrome extensions have always had use of SQLite via its WebSQL API. "IndexedDB won the W3C cage match" isn't really a good enough reason to break apps in both camps.

> Chrome happens to still implement it,

You imply that support is only incidental and likely to end. You've a source for that?

> but that doesn't really have any relationship to its extension API.

Only because every other platform implements WebSQL as a Web API; they don't need a chrome.* SQL interface. Mozilla lacks a WebSQL API, but that deficit can be remedied by exposing the existing SQLite functionality as a browser-facing WebExtension interface.

> A lot of people have actually been pushing to replace both Places and content 
> prefs with NoSQL databases.

Pushing to replace them with IndexedDB? Are there proposed schemas, or just water-cooler banter?

The relative utility of IndexedDB vs SQLite isn't the issue here. The question is whether it's necessary or even prudent to arbitrarily force Mozilla add-on developers and Webkit extension developers to rewrite working SQL statements as Javascript code in order to recreate a functional database using IndexedDB.

> I don't know how many extensions need relational databases, or how many have 
> problems that a relational DB would solve better than IndexedDB.
Exactly. However, you could assume that add-on developers choose the storage scheme best suited to their needs, and use your privileges as a Mozilla insider to query the add-ons database to see how many add-ons use indexedDB, how many localStorage, and how many openDatabase (as a proxy for the mozIStorageService).

Words like "need", "better", or "rare" are meaningless without a metric. For example, at first blush it seems an IndexedDB store could be implemented as n+1 Arrays and n expressions for flattening the Objects in one of them to scalars in the others. What you find though is a SQLite database at its core, with foreign keys and triggers defined to protect data integrity. Profile databases don't use either feature, so that's certainly rare. Was that necessary? Is it a better solution? Maybe not, but probably so given the circumstances. You see the difficulty.

I wonder what the numbers will say.

> I'd probably suggest that they use a library that handles it for them rather 
> than roll a relational DB themselves.

Might have done, but didn't. You mean a SQL RDBMS built on IndexedDB which is itself built upon SQLite, a SQL RDBMS? That was Mozilla's advice six years ago [1]; still no joy.

> There's a bug about creating a data migration path for add-ons being converted 
> to WebExtensions.
I found it, bug 1214790, thank you. It's still a bit thin.

[1] https://hacks.mozilla.org/2010/06/beyond-html5-database-apis-and-the-road-to-indexeddb/
[2] https://developer.chrome.com/extensions/api_other
[3] https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Chrome_incompatibilities

Comment 6

[Kris Maglione [:kmag]]

(In reply to Nancy Grossman from comment #5)
> I don't believe that's relevant. The context here is WebExtensions. The W3C
> standards you allude to are recommendations for web-facing APIs. "We
> [Mozilla] think SQLite is an extremely useful technology for applications,
> and make it available for Firefox extensions and trusted code. We don’t
> think it is the right basis for an API exposed to general web content." [1]
> What's proposed is not a web-facing API, it's an API exposed to extensions
> that provides continuing access to an extremely useful technology.

W3C standards apply to a lot of things. There are a number of standards for
privileged APIs which aren't exposed to the web. There's also an emerging
standard for extension APIs, and it does not include WebSQL.

> > WebExtensions is on the road toward standardization.
> 
> The only cross-platform standard for extensions,

There is a community group which is currently working to develop a W3C
standard for extension APIs:

https://www.w3.org/community/browserext/

> a de facto standard, is Google's Platform APIs,

That's not quite the case. Several platforms currently implement different
subsets and supersets of that platform, but none are aiming for 100%
compatibility.

> and WebSQL is a part of it [2].

It's listed among web APIs that are available in Chrome, and happen to be
available to extensions. It's not part of the platform, or any standard.

> Every major platform except Internet Explorer now writes to that standard,
> and every one of those except Mozilla implements WebSQL.

Currently, Chrome/Chromium, Opera, Firefox, and Edge implement extension APIs
based on Chrome's. Of those, Chrome and Opera support WebSQL. Firefox and Edge
do not.

> If extensions are to be substantially portable, then we need a WebExtension
> alternative to Chrome's WebSQL API.

IndexedDB is the standards-based alternative.

> Mozilla add-ons have had use of SQLite via the Storage Service for over a
> decade, and Chrome extensions have always had use of SQLite via its WebSQL
> API.

They've had access to a lot of things that they'll no longer have access to.
In the future, we intend to rely as much as possible on standards-based and
maintainable APIs.

> > Chrome happens to still implement it,
> 
> You imply that support is only incidental and likely to end. You've a source
> for that?

No. I mean that Chrome continues to implement it despite it no longer being on
any standards track. There's been a general push to remove support for
non-standard APIs that are exposed to the web, but I can't speak to the plans
of the Chromium team.

> Pushing to replace them with IndexedDB? Are there proposed schemas, or just
> water-cooler banter?

In all likelihood, with something lower-level, such as LevelDB, which would
also become the new backing store for IndexedDB.

No. There are no proposed schemas. At the moment, the vast bulk of our
performance work is focused on projects like electrolysis.

Comment 7

[Nancy Grossman]

(In reply to tofumatt [:tofumatt] from comment #4)
> I don't see it as likely that Mozilla implements WebSQL

OK, to be clear, I don't expect Mozilla to implement (web-facing) WebSQL. I'm proposing to expose the Storage Service through a WebExtension interface on chrome.* (or browser.*). I suggested using the async WebSQL interfaces so that Chrome extensions can simply use 'chrome.something' as the base object for .openDatabase() in Firefox instead of 'window' as in Chrome.

> I think finding a library on top of IndexedDB is your best option

I hope not. Otherwise I'll wait for native.js, or something like it, so that I can use the Storage Service. I did stumble across a `webext-transition-api` [1], which would be fine, but I can't find it on MDN or on Bugzilla so I don't know when if ever it will reach me.

> IndexedDB [is] really an API to BUILD database APIs...

I read that too, but I don't believe it. You see, tabular data breaks indexing in IndexedDB. It can't index on `last` name in any of these data structures because you can't reach it with a key path [2]:

{dancers: [
  {first: "Isadora", last: "Duncan"},
  {first: "Martha", last: "Graham"}
]}

{dancers:
  "1": {first: "Isadora", last: "Duncan"},
  "2": {first: "Martha", last: "Graham"}
}

{dancers: [
  ["Isadora", "Duncan"],
  ["Martha", "Graham"]
]}

There's no technical obstacle to it, and bracket notation could have represented the key paths ('["dancers"][*]["last"]' or '["dancers"][][2]', e.g.), yet tabular data structures are not first-class objects in IndexedDB. Without script they're effectively a kind of Blob. How can it possibly be a basis for database APIs if tabular data is opaque to it?

[1] https://github.com/rpl/firefox-patches/tree/master/webext-transition-api
[2] https://www.w3.org/TR/IndexedDB/#dfn-steps-for-extracting-a-key-from-a-value-using-a-key-path


(In reply to Kris Maglione [:kmag] from comment #6)
> Currently, Chrome/Chromium, Opera, Firefox, and Edge implement extension APIs
> Of those, Chrome and Opera support WebSQL. Firefox and Edge do not.

You're right, I'd counted Safari; its API is not the same. OTOH Microsoft has promised extensions but not delivered. (How unlike them.)

Comment 8

[Nancy Grossman]

You are one duplicitous bastard, Andy McKay. You triage this issue here as P5 on Oct 7, and on Oct 13 you declare it a non-starter over on GitHub [1]. Which is it then, dead or alive?

The issue was raised here, please address the issue here and not out-of-band.

[1] https://github.com/web-ext-experiments/about/commit/fcdcc0f858eec64a6b1bff55280a60b207077cda#diff-d8dbe1ef014cfdcb1dc2ad0d0f093ca8

Comment 9

[Nancy Grossman]

OK, I've taken a look at localForage [2]. It implements typed key-value storage, not a relational database. They're not at all the same. I'm not looking for an arbitrary wrapper around IndexedDB, I'm looking for access to a reliable relational database, preferably with an API "sane" enough that I won't be writing `if (firefox) else` statements for the next five years.

[2] https://github.com/localForage/localForage

Comment 10

[Andy McKay]

Nancy, can I please point you at: https://www.mozilla.org/en-US/about/governance/policies/participation/ and https://bugzilla.mozilla.org/page.cgi?id=etiquette.html.

We discussed the issue in March amongst contributors in Berlin. I then entered our findings into a Wiki on April 4th: https://wiki.mozilla.org/index.php?title=WebExtensions%2FNewAPIs&diff=1128483&oldid=1125926, the conversation has continued since then and you can see multiple comment from comment 5. The move in October was a move to github from the Wiki.

Comment 11

[Nancy Grossman]

Etiquette be damned. You've made two very different determinations in two different places without referencing either from the other. Answer the question.

Comment 12

[Kris Maglione [:kmag]]

(In reply to Nancy Grossman from comment #11)
> Etiquette be damned.

That, at least, has been clear from the start.

Comment 13

[Nancy Grossman]

Answer the question.

Comment 14

[Kris Maglione [:kmag]]

I don't think I could have been much clearer:

> There's a non-zero chance that we may make WebSQL available to extensions in
> the future, but we'd like to avoid it if at all possible.
...
> Have thought about it and aren't currently planning to.
...
> There's also an emerging standard for extension APIs, and it does not
> include WebSQL.

We are almost certainly not going to implement WebSQL for extensions. There's
still a non-zero chance that we may change our minds in the future, which is
why this bug is still open, but it is very unlikely. None of that has changed
since the start of this conversation.

Andy assigned this bug P5 priority because it was in our list of unprioritized
bugs, and P5 is the lowest priority that we can currently assign it.


All of that said, if you can't keep this conversation civilized, then I don't
have anything further to say. Whatever you feelings on this matter, there's
absolutely no call for you to insult other members of my team and then rudely
demand answers. Everyone here is working extremely hard to make this project a
success, and the particular commit that triggered your insult was part of
Andy's effort to make this process as open and transparent as possible. So I'm
not going to encourage you to continue to make such unprovoked attacks by
continuing to respond to them.

Comment 15

[Andy McKay]

I've been trying to give every bug a priority following the new Bugzilla triage guidelines and P5 seems the most appropriate.

A group of contributors discussed this bug and followed up on this bug as a result. We outlined some thinking onto the wiki page. We did not close this bug for the simple reason: we don't believe that everyone on our team knows everything and we could have been wrong earlier this year when we started on the WebExtensions project and spent a few minutes reviewing this bug.

I'll repeat those comments here:

* Concerns about WebSQL as not a successful standard.
* No overwhelming reason to support this.
* There are already many libraries that exist outside of WebExtensions wrap around IndexedDB and give developers access to sane APIs. Local forage is one example.
* Concerns about maintenance of such an API.

After 9 months no-one else has commented on wanting this API and multiple people have suggested that it would be a bad idea. 

Perhaps just marking this a WONTFIX would be more appropriate at this point.