715790 - Consider allowing webapps to ship a prepopulated DB of some sort

Status: NEW

(Core :: DOM: Core & HTML, defect, P5)

Description

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

In gaia, we're running into https://github.com/andreasgal/gaia/issues/236 in which loading an IME database is causing slow-script timeouts.  There are various ways we can work around that, but Tim brings up the interesting idea of shipping a pre-populated DB.

Now, granted we can't ship a pre-setup foodb.sqlite file, but is there some intermediate format an app could ship to allow us to offload DB processing entirely to gecko?  Preferably off the main thread.  A solution that involves loading then serializing MBs of JS doesn't help things much.

Ideas?

Comment 1

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(Of course, for a shipping b2g phone we can cheat in various ways with what we pre-install.  But I'd rather have a solution that works for arbitrary web apps.)

Comment 2

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

I doubt we'll get to the point where the actual database-file format is even remotely closed to standardized anytime soon. The format for the mozilla implementation has been wildly changing and I expect it to change for some time still. And then there's the issue that the other two IndexedDB implementations out there use completely different back-ends.

So the best thing I can think of is to add some sort of "import API" to indexedDB. This would make a lot of sense if there's some sort of commonly used data format that we can support. For example if we could add support for importing CSV files.

For CSV import I'd expect that you'd provide a Blob and a list of property names which the various columns should be imported into.

But you could fairly easily write that as a JS-library too, which imported the data in small enough chunks that you didn't risk slow-script dialogs.

Comment 3

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(In reply to Jonas Sicking (:sicking) from comment #2)
> I doubt we'll get to the point where the actual database-file format is even
> remotely closed to standardized anytime soon. The format for the mozilla
> implementation has been wildly changing and I expect it to change for some
> time still. And then there's the issue that the other two IndexedDB
> implementations out there use completely different back-ends.
> 

Yes, of course.  Trying to standardize that would be counterproductive, even.

> So the best thing I can think of is to add some sort of "import API" to
> indexedDB. This would make a lot of sense if there's some sort of commonly
> used data format that we can support. For example if we could add support
> for importing CSV files.
> 
> For CSV import I'd expect that you'd provide a Blob and a list of property
> names which the various columns should be imported into.
> 

Interesting.  I don't know the IDB API, but you're proposing something like DB.open([csvPrepopData])?  We could probably do some magic for web apps at install time, if the csv data is recognizable as DB prepop.

> But you could fairly easily write that as a JS-library too, which imported
> the data in small enough chunks that you didn't risk slow-script dialogs.

Yeah.  That forces a degraded "first run" experience on installed apps, but it's always an option.  An "just installed" intent/hook might be a more general solution to that issue.

Comment 4

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Ah, one point missing from the Github: If we just want to avoid the slow-script timeouts, we could design the entire db populating process into Web Worker with synchronized db request. But I would still prefer shipping an IndexedDB db in default profile of B2G Gaia though.

Comment 5

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Unfortunately we don't yet support indexeddb from workers...

Comment 6

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Loading the prepop data on a worker and sending it back in chunks to the main thread is a good idea.

Workers themselves are probably not that likely to get direct IndexedDB access, and even if they did it wouldn't happen for quite a while :/.

Comment 7

[Tim Guan-tin Chien [:timdream] (please needinfo)]

http://dvcs.w3.org/hg/IndexedDB/raw-file/tip/Overview.html#sync-database

WIP spec on synchronized database in workers.

Comment 8

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

The spec for synchronous IDB access in workers is actually quite mature. The problem is finding the time to implement it as it's quite a lot of work.

What I was proposing was something like:

myidbtransaction.objectStore("mystore").
  importData(BlobWithCSVData, ["propNameCol1", "propNameCol2", ""propNameCol3"]);


But if all we're trying to solve is slow-script dialogs, it's quite easy with IndexedDB to return to the event loop often enough to avoid slow-script dialogs.

Comment 9

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(In reply to Jonas Sicking (:sicking) from comment #8)
> The spec for synchronous IDB access in workers is actually quite mature. The
> problem is finding the time to implement it as it's quite a lot of work.
> 

It depends on the magical new DOM bindings, right?

> What I was proposing was something like:
> 
> myidbtransaction.objectStore("mystore").
>   importData(BlobWithCSVData, ["propNameCol1", "propNameCol2",
> ""propNameCol3"]);
> 

That seems reasonable.  Is there a way to get a "blob promise" for a network resource, that doesn't require loading it in entirety?  (Sorry, don't know that api.)

> 
> But if all we're trying to solve is slow-script dialogs, it's quite easy
> with IndexedDB to return to the event loop often enough to avoid slow-script
> dialogs.

Yes, but it's not very developer-friendly, forces folks to guess where to chop the calls, and is likely lower-performance than a batching scheme we decide on.  We'll implement a workaround that's a variant of this gaia for now.

Comment 10

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

(In reply to Chris Jones [:cjones] [:warhammer] from comment #9)
> (In reply to Jonas Sicking (:sicking) from comment #8)
> > The spec for synchronous IDB access in workers is actually quite mature. The
> > problem is finding the time to implement it as it's quite a lot of work.
> > 
> 
> It depends on the magical new DOM bindings, right?

It doesn't technically depend on it. But yes, it would be significantly easier if we had it. As with all worker stuff :(

> > What I was proposing was something like:
> > 
> > myidbtransaction.objectStore("mystore").
> >   importData(BlobWithCSVData, ["propNameCol1", "propNameCol2",
> > ""propNameCol3"]);
> > 
> 
> That seems reasonable.  Is there a way to get a "blob promise" for a network
> resource, that doesn't require loading it in entirety?  (Sorry, don't know
> that api.)

Not currently, no. It's something that I've thought quite a bit about. The main problem is that it adds a lot of extra error conditions for blob handling, and also raises the question of what happens if the server is updated while partial reads are happening etc.

Using XHR you can download a resource and get it as a blob though. That makes turning a url into a blob a pretty easy task.

> > But if all we're trying to solve is slow-script dialogs, it's quite easy
> > with IndexedDB to return to the event loop often enough to avoid slow-script
> > dialogs.
> 
> Yes, but it's not very developer-friendly, forces folks to guess where to
> chop the calls, and is likely lower-performance than a batching scheme we
> decide on.  We'll implement a workaround that's a variant of this gaia for
> now.

A library would take care of these issues I'd think.

But if CSV (or some other DB format) is common enough then an import function would make sense to me. I'll put it on the list for IDBv2.

Comment 11

[Ben Francis [:benfrancis]]

I could do with something like this for pre-populating the gallery app with photos.

The code I'm writing at the moment fetches each photo using XHR with a blob response, then saves the blobs to IndexedDB the first time the app is loaded. It would be great if these blobs could be pre-populated in the profile at the time of shipping.

Wouldn't storing blobs in CSV be problematic?

Jonas, what do you mean by "For CSV import I'd expect that you'd provide a Blob and a list of property names which the various columns should be imported into."?

Comment 12

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Just to be clear, there are generic interfaces we should add to IDB to make this easier for all web developers, shipping apps across all browsers.  That's the CSV proposal, or something along those lines.  We would use it in firefox and b2g for installed apps.

Then there's the practical issue of shipping a b2g phone to consumers, with pre-installed and pre-configured apps that have passed regulatory testing.  There, we absolutely want/have to ship gallerydb.sqlite et al. in the profile.  But there's also absolutely no way that's going to be a standard, nor should it be.

Comment 13

[Firefox Bug Husbandry Bot]

https://bugzilla.mozilla.org/show_bug.cgi?id=1472046

Move all DOM bugs that haven't been updated in more than 3 years and has no one currently assigned to P5.

If you have questions, please contact :mdaly.