1156198 - support pre-loading a Service Worker on b2g

Status: RESOLVED INCOMPLETE

(Core :: DOM: Service Workers, defect)

Description

[Noemí Freire (:noemi)]

Please refer to http://slightlyoff.github.io/ServiceWorker/spec/service_worker/index.html#update-algorithm (4.8 section)

Following the Service Workers spec the HTTP cache should be checked for requests lower than 1 day old. It is currently being bypassed:
https://mxr.mozilla.org/mozilla-central/source/dom/workers/ServiceWorkerScriptCache.cpp#95

That was also raised in https://bugzilla.mozilla.org/show_bug.cgi?id=931249#c78

Thanks!

Comment 1

[Ben Kelly [:bkelly, not reviewing]]

I agree we aren't following the spec, but can you describe how this is effecting your app?

It seems the only way this would make a difference is if you accessed the script with another network request within the 24 hour window.  This seems unlikely to me, but maybe I don't understand the use case.

Do you have many ServiceWorkers using the same script url?

Also, be aware this is a bit tricky to fix.  Our network code doesn't offer an easy way to "bypass cache only if over X age" out-of-the-box.

Comment 2

[Noemí Freire (:noemi)]

Passing the ni to Antonio who better explain the use case. Thanks!

Comment 3

[Antonio Manuel Amaya Calvo (:amac)]

What we're trying to do is to preload an app that uses a service worker on the phone. One way we used to do this was just preloading whatever we wanted to be available at boot time on the cache. Then the network accesses would just give the cached element and everything would work. When we tried this with a service worker, though, it didn't work. 

And we need this or some way or form to make the initial load of the service worker locally, or having preinstalled apps that use service workers will not work.

Comment 4

[Ben Kelly [:bkelly, not reviewing]]

So, I don't think fixing this spec issue in our code will solve your problem.  The spec says only to consult the http cache if the resources is less than 24 hours old.  That seems unlikely to always be the case for the first launch of an app.

We can make Cache API a pre-loadable thing like we used to do it.  In theory you could do this today by copying everything under:

  $PROFILE_DIR/storage/default/$ORIGIN_DIR/cache

Note, however, we expect the Cache disk format to change a bit before release.

Does this help at all?

Comment 5

[Antonio Manuel Amaya Calvo (:amac)]

The problem is not the cache API as much as the service worker itself. We can cache everything on the app (using the cache directory) save for the service worker itself. I know the spec will still leave us hanging for most cases, but I don't think the spec was thought with devices that need to work offline out of the box anyway. 

If not the strict spec, we still need some way to preload/precache the service workers.

Comment 6

[Ben Kelly [:bkelly, not reviewing]]

(In reply to Antonio Manuel Amaya Calvo (:amac) from comment #5)
> The problem is not the cache API as much as the service worker itself. We
> can cache everything on the app (using the cache directory) save for the
> service worker itself. I know the spec will still leave us hanging for most
> cases, but I don't think the spec was thought with devices that need to work
> offline out of the box anyway. 

The ServiceWorker script itself is saved in the Cache API.  The name of the Cache object is saved in the SW registration file.  You need to pre-load both of those bits.

Have you tried that?  Andrea might be able to help with pre-loading the registration file part.

Comment 7

[Antonio Manuel Amaya Calvo (:amac)]

So I've talked with Andrea this morning and it seems that wouldn't work either. On one hand apparently the way that cache works it would still try to hit the network and fail if it didn't had network (I'll let him explain that part better) and on the other hand, even if it did work, we would still have the problem than the "install" event would *not* fire the first time the application ran (and it might cause a malfunction of the app, depending on what the app did on that event).

Can we just put this LOAD_BYPASS behind a preference that we can switch at the first boot? The only time it's critical for B2G to not hit the network is at the first launch of the preinstalled apps. So we could do something like:

(on OfflineCacheInstaller.jsm):
 - Prefill the cache (as we do now)

(on FTE) 
  - For each preinstalled app that uses service workers, launch the app.
  - After we've launched all the apps, flip a preference-backed setting that controls if we're running the FTE

(on the service worker loaded)
  - if we're running the FTE (use the preference that backs the setting), then don't skip the cache. Otherwise, do as we do now.

That way it wouldn't be very pretty (since we would have to launch all the SW backed apps in the background at first boot) but it would work.

Another, slightly better way (since we don't have to launch the apps) would be having an array preference with all the SWs that have to be cached on first use, and just remove them from the prefrence as they're being launched.

Comment 8

[Ben Kelly [:bkelly, not reviewing]]

Changing the name of the bug to reflect the conversation.

Also, it seems we're getting to the point where we need necko support.  Sounds like we want some kind of pre-loaded http cache entry that BYPASS does not ignore.  Or a different BYPASS flag which can be used to allow this kind of pre-loaded entry to be matched.

Honza, what do you think?

Comment 9

[Honza Bambas (:mayhemer)]

(In reply to Ben Kelly [:bkelly] from comment #8)
> Changing the name of the bug to reflect the conversation.
> 
> Also, it seems we're getting to the point where we need necko support. 
> Sounds like we want some kind of pre-loaded http cache entry that BYPASS
> does not ignore.  

There is no way right now to tell an http cache entry to "bypass the bypass".  The flag is examined before the entry is even open, in nsHttpChannel.

But you can do the following:
- on first fetch set manually a metadata (e.g. "service-worker-network-refetch-time") on the cache entry with value of the time you want the entry no longer be reused for your SW fetches
- then, before opening any http channel for following SW loads, check the cache entry manually and optionally AsyncDoom() it based on the previously stored time at your metadata element

To open the entry see "Example of the cache v2 code doing the same thing" at http://www.janbambas.cz/http-cache-v1-api-disabled/

Comment 10

[Ben Kelly [:bkelly, not reviewing]]

I spoke with Andrea and we can probably add something to the SW registrations file indicating its a "preloaded" entry.  It can then fire the install event on first run.

We may also want to split out the pre-loaded info from the main SW registrations file into a separate file.  This would make it easier to copy around the preloaded info without conflicting with other entries which might have organically been installed.

Comment 12

[(no longer active)]

I would like to understand the overall picture of what we want to do on the Gaia side better here.

I think the ideal way to pre-install apps that use service workers on the phone is to open each app during the build process, let it run some well-known initialization code that would register the service worker using the normal navigator.serviceWorker.register() call, let the service worker run and install as a normal service worker would, then close the app, and store things such as the cache database of that origin, the service worker registration information and anything else that we need in the profile dir of the respective app.  This way, we would not need to maintain two separate code paths for the installation of the service worker, and everything should work the same for the first run that the users will see as their future runs.

Is that something which we can do as part of the Gaia build?  Does that even make sense?  If it is not possible, but you agree that it's nice, then we can discuss what we need to fix in order to make this possible.  If you think this idea doesn't make sense, then please let's discuss, I would be interested to know why it wouldn't work...

Thanks!

Comment 13

[Ben Kelly [:bkelly, not reviewing]]

I think the main problem with the approach in comment 12 is that it requires a b2g-like execution context at build time.  I think the b2g build wants to move away from a dependency on xulrunner, etc.

Comment 14

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Ben Kelly [:bkelly] from comment #13)
> I think the main problem with the approach in comment 12 is that it requires
> a b2g-like execution context at build time.  I think the b2g build wants to
> move away from a dependency on xulrunner, etc.

Gaia's primary build problems as they've related to xulrunner have been that:
- using xpcshell or firefox/b2g-desktop in xulrunner-mode like they are node.js/io.js turns out poorly since it's not a particularly supported use-case, wheels have to be re-invented/hacked up, and various mental assumptions about synchronous/async code are violated.
- the build system has not been good about dealing with users constantly hopping branches and having to re-download large builds on each branch change; in some cases (OS X), erroneous make targets resulted in large files being re-downloaded every time specific "make" targets were used

I think it would be okay to create a situation where the existing node.js marionette tooling is invoked to do something like the following:
- create a new profile
- use marionette to connect and run chrome privileged code in the firefox/mulet/b2g process that invokes code in Webapps.jsm or via navigator.mozApps.install or whatever that will trigger installation of the apps, waiting for Promises
- shuts down the process cleanly
- copies the stuff out

This could be packaged up into an npm module that presents a binary and then whatever the build system, it works.

In general this seems desirable from a risk factor perspective.  There are many people on the gaia team who can debug this going wrong, but relatively few in gaia (other than Fabrice :) who could debug a specialized service worker code path going wrong in gecko.

Comment 15

[Francisco Jordano [:arcturus] [:francisco]]

Having the code executed sounds like a good idea to me, since the logic of which caches are suppose to be used, files cached etc, lives in the application.

How we end up with that information in the profile, both ideas in the last comments talk about somehow letting the sw run, which seems to me like the best option. I think is up to the team owning the build system to decide.

Another thing that bothers me a bit, is the fact that we have customization, that is partner adding extra web apps: https://github.com/mozilla-b2g/gaia/tree/master/customization . I think that will continue working but if developers start porting their webs to sw will be interesting to support those out of the box non-gaia apps.

ni Tim, his team owns the gaia build system, to have him on the loop.

Comment 16

[(no longer active)]

For apps that do not come pre-installed on the phone, I don't think we need to do anything special (please correct me if I'm wrong) since they will need to be able to register their service workers the first time they are run.

Comment 17

[Salvador de la Puente González [:salva]]

To run the SW seems the right way to go for me but I'm not sure if this first-run should be performed during the build process or in the device. What sounds pretty clear to me is that the list of resources should end inside the HTTP cache as it's a problem of (lack of) connectivity. This way, the first time the SW runs it has the content it needs in the HTTP cache.

The problem Noemi is explaining in comment 0 is a little bit different but could be solved in the same way. By prepopulating HTTP cache with the sw as well. The specification is saying [1]:

  > Let response be the result of running fetch using r, forcing a network fetch if cached entry is greater than 1 day old.

I'm assuming the network fetch is referring to https://fetch.spec.whatwg.org/#http-network-fetch and, although I think it is right, it is missing this scenario as Antonio said in comment 5. In this case, you can not tell about how old is the entry as it was never "fetched from the network" so it is "always new" and thus, the age for that resource is 0. The spec does not tell about the case the sw age is lower than 1 day but it is not registered so we can provide a custom behaviour here and make a network or cache fetch in this case [2].

[1] http://slightlyoff.github.io/ServiceWorker/spec/service_worker/index.html#update-algorithm
[2] https://fetch.spec.whatwg.org/#http-network-or-cache-fetch

Comment 18

[Tim Guan-tin Chien [:timdream] (please needinfo)]

What Andrew said on comment 14 is pretty much what I want to say. We would like Gecko to consume something not Gecko-specific as the preload data. So it would be great if we could drop a, say, JSON descriptor, that would trigger Gecko to register a Service Worker in the cache.

However, it's unclear to me what Ben said in comment 4; What's the file structure required in |$PROFILE_DIR/storage/default/$ORIGIN_DIR/cache|? Is it Gecko specific? If it's not, this will solve the cache part of what's discussed in bug 1155619 (except for documenting it) and we can just worry about the registering Service Worker part of the work here.

Comment 19

[Ben Kelly [:bkelly, not reviewing]]

(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #18)
> However, it's unclear to me what Ben said in comment 4; What's the file
> structure required in |$PROFILE_DIR/storage/default/$ORIGIN_DIR/cache|? Is
> it Gecko specific?

Its specific to the particular version of gecko used to create it.  Its basically an sqlite db with some extra files to store Response bodies.

There is a schema version in the sqlite db.  For right now we simply wipe data if we detect an older version.  Once we enable Cache for default on release builds we will begin migrating the data for any schema changes.  That will probably happen some time in the 41 cycle.

Comment 20

[(no longer active)]

(In reply to Salvador de la Puente González [:salva] from comment #17)
> To run the SW seems the right way to go for me but I'm not sure if this
> first-run should be performed during the build process or in the device.
> What sounds pretty clear to me is that the list of resources should end
> inside the HTTP cache as it's a problem of (lack of) connectivity. This way,
> the first time the SW runs it has the content it needs in the HTTP cache.

Note that the service worker itself is cached in the "DOM Cache" (which is completely separate from the HTTP cache), and that is where we read the service worker from before executing it.  The HTTP cache is transient and should not be relied on for any actual persistence needs.

> The problem Noemi is explaining in comment 0 is a little bit different but
> could be solved in the same way. By prepopulating HTTP cache with the sw as
> well. The specification is saying [1]:
> 
>   > Let response be the result of running fetch using r, forcing a network
> fetch if cached entry is greater than 1 day old.

I'm puzzled as to why you think that part of the spec is relevant here, but from what you are asking for, you are asking for the DOM Cache to be populated here, which my suggestion above will do.

> I'm assuming the network fetch is referring to
> https://fetch.spec.whatwg.org/#http-network-fetch and, although I think it
> is right, it is missing this scenario as Antonio said in comment 5. In this
> case, you can not tell about how old is the entry as it was never "fetched
> from the network" so it is "always new" and thus, the age for that resource
> is 0. The spec does not tell about the case the sw age is lower than 1 day
> but it is not registered so we can provide a custom behaviour here and make
> a network or cache fetch in this case [2].

Conceptually the b2g situation is similar to what a normal user would get when visiting a web page that already has a service worker installed.  Hopefully that mental model would help when thinking about how what the spec says is related to what we want to do on the device.

Comment 21

[(no longer active)]

(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #18)
> What Andrew said on comment 14 is pretty much what I want to say. We would
> like Gecko to consume something not Gecko-specific as the preload data. So
> it would be great if we could drop a, say, JSON descriptor, that would
> trigger Gecko to register a Service Worker in the cache.

There is no cross browser preload data for service workers.  The way you're supposed to do that is by installing the SW, letting it handle the install event and programmatically prime its cache however it sees fit which is what I was describing in comment 12.

We can create such a JSON format too but of course that will be SW-specific, and it will also mean that someone needs to manually verify that what the gaia SWs do in their install and update events is the same as what appears in the JSON file, which makes me believe that it's technically more sound to install the service workers properly as opposed to inventing a custom JSON format or some such, and I would like to understand whether that's practical or not.

> However, it's unclear to me what Ben said in comment 4; What's the file
> structure required in |$PROFILE_DIR/storage/default/$ORIGIN_DIR/cache|? Is
> it Gecko specific? If it's not, this will solve the cache part of what's
> discussed in bug 1155619 (except for documenting it) and we can just worry
> about the registering Service Worker part of the work here.

Note that such a solution will mean that the installation step will always depend on the internal structure of the database, so if for example we change the DB schema you may need to update the build scripts manually.  And note that there is also a service worker registration file that the build system will need to manually populate.  And then there is the service worker script cache which has a dynamically generated name that you'd need to setup properly, and so on.

While all of the above are things that we can get right with some effort, comment 12 is the only robust and non-Gecko specific solution that guarantees that what gaia does will be shielded from the changes to the internal implementation.

Comment 22

[Tim Guan-tin Chien [:timdream] (please needinfo)]

(In reply to :Ehsan Akhgari (not reading bugmail, needinfo? me!) from comment #21)
> While all of the above are things that we can get right with some effort,
> comment 12 is the only robust and non-Gecko specific solution that
> guarantees that what gaia does will be shielded from the changes to the
> internal implementation.

I don't understand how comment 12 can be a non-Gecko specific solution. It depend on Gaia build script to download and launch a copy of Gecko and tell it to load apps. I agree with the benefits though but I don't think they are not attainable with alternatives (structure data and files designed to be consumed by Gecko when the new profile is created).

Comment 23

[Tim Guan-tin Chien [:timdream] (please needinfo)]

(needinfo for above comment)

Comment 24

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #22)
> I agree with the benefits though but I don't think they are
> not attainable with alternatives (structure data and files designed to be
> consumed by Gecko when the new profile is created).

It's not clear what you're proposing as an alternative or what you're specifically trying to avoid.  The new profile is created at build time.  When the device starts up for the first time the user turns it on, we don't want the device to have to churn through ingesting some specialized format and do a lot of busy-work.

So if we assume this has to be done at build-time, do you want to avoid us using Marionette?  Like have mulet have a "--installApp serviceWorkerInstallFormat.json" command line option?

Or do you want to avoid invoking Gecko entirely and have a node.js app that duplicates the entirety of the service worker DOM cache database persistence schema and storage logic and runs the service worker logic for the app in a node.js-hosted runtime context (or assumes a specific service worker framework so that the logic doesn't need to run) that generates the schema.  And then when FxOS first runs we process some minimal JSON file that makes sure the service worker logic and DOM cache logic make sure they know that service worker cache database is there.

Comment 25

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Hum, I don't think I fully understand comment 14 when I typed out coment 18; comment 14 call for launching an Gecko process to populate the Cache as well, but in an isolated npm package thus does not require the entire build system written in xpcshell.

It also sounds like my alternative will involve having people write duplicate code for the build script just to generate the caches, like a template system w/ two impl lives on server and client (and dramatically written in PHP and JS.

Given that I think I can support what Andrew said in comment 14, as an implementation proposal of idea in comment 12.

Comment 26

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Without reading the mid-air collided comment 24 I have already replied with comment 25.

(In reply to Andrew Sutherland [:asuth] from comment #24)
> Or do you want to avoid invoking Gecko entirely and have a node.js app that
> duplicates the entirety of the service worker DOM cache database persistence
> schema and storage logic and runs the service worker logic for the app in a
> node.js-hosted runtime context (or assumes a specific service worker
> framework so that the logic doesn't need to run) that generates the schema. 
> And then when FxOS first runs we process some minimal JSON file that makes
> sure the service worker logic and DOM cache logic make sure they know that
> service worker cache database is there.

Specifically to this part, yet, this is what I already have though -- the format can be as easily as a static declaration of the parameters call to Cache API. But again, I acknowledge the hard part is about generating these contents on build time, which are the logic I agree we should try not to duplicate.

Comment 27

[Salvador de la Puente González [:salva]]

(In reply to :Ehsan Akhgari (not reading bugmail, needinfo? me!) from comment #20)
> (In reply to Salvador de la Puente González [:salva] from comment #17)
> > To run the SW seems the right way to go for me but I'm not sure if this
> > first-run should be performed during the build process or in the device.
> > What sounds pretty clear to me is that the list of resources should end
> > inside the HTTP cache as it's a problem of (lack of) connectivity. This way,
> > the first time the SW runs it has the content it needs in the HTTP cache.
> 
> Note that the service worker itself is cached in the "DOM Cache" (which is
> completely separate from the HTTP cache), and that is where we read the
> service worker from before executing it.  The HTTP cache is transient and
> should not be relied on for any actual persistence needs.

Yes, I know. I just wondering if by trying to prepopulate the HTTP cache (it's not my intention to use as persistence, just for the first time), we could make something less Firefox OS - specific. It's like saying, "if it were network, you will find the SW and the needed files for populating SW caches on the network, but there is no network so we simulate there is by having these files on the HTTP cache". The problem with this approach is that the specification says we must do a network fetch but I think could be relaxed and proposed as a new use case for the spec.

Comment 28

[Ben Kelly [:bkelly, not reviewing]]

On IRC Vivien suggested creating a jsm script that runs on device boot that will populate the SW registrations and Cache from a pristine directory on flash.  So the build script would just have to populate that source pristine dir.

That seems reasonable to me if we can expose some JS chrome API to manipulate the SW registrations.

Comment 29

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Ben Kelly [:bkelly] from comment #28)
> On IRC Vivien suggested creating a jsm script that runs on device boot that
> will populate the SW registrations and Cache from a pristine directory on
> flash.  So the build script would just have to populate that source pristine
> dir.

Is this intended to be for the development cycle only (make APP=blah install-gaia), or for production/shipped devices?

Specifically, when I do "make PRODUCTION=1 GAIA_OPTIMIZE=1", I find that the following previously non-existent have the following sizes per "du -csh":
- build_stage: 115M (the unzipped but optimized stuff on the filesystem)
- profile/webapps: 35M (this is full of zips)
Which is a lot of data no matter how you slice it.

This seems unacceptable for production devices to churn through at startup unless the production build step actually means having mulet do this work on the build machine.  In that case, it's largely isomorphic to a command-line argument or using marionette, except marionette is much more capable of tracking completion status.  (Not that I particularly want us to use Marionette; but the Marionette automation tooling in general already is aware of doing things like messing with preferences so the build doesn't try to auto-update in the middle of running, knows how to detect crashes, already needs to know how to install apps, etc.)

Comment 30

[(no longer active)]

I missed the said IRC conversation, so I'm not sure what the idea in comment 28 is trying to solve.

It would probably be helpful to have a meeting with the Gaia folks who can speak to what we can/should do from the Gaia side.  I think that might help with reducing the confusion (at least mine!) here.  :-)

Comment 31

[Ben Kelly [:bkelly, not reviewing]]

You can read the IRC discussion in this somewhat noisy IRC log of #developers:

  http://logs.glob.uno/?c=mozilla%23developers&s=30+Apr+2015&e=30+Apr+2015&h=vingtetun#c1213520

Comment 32

[[:fabrice] Fabrice Desré]

Hi all, happy to see you're all having fun! ;)

A bit of history: a long long time ago, in a galaxy where people believed in appcache, we were preloading the appcache at build time for all the apps (yep, all gaia were using appcache). This was done by using xpcshell as part of the build process. Much breakage ensued wih gecko/gaia mismatches, and since we had to push the cache to the /data partition, everything was removed when doing a factory reset. Not cool. We then moved to packages for gaia and preloading the appcache at first run or after a system update for 3rd party apps that still use appcache. This code is still in the tree.

There's a nice opportunity with SW to have a single solution for gaia and 3rd party apps. My proposal is to proceed as follows:
1- ship the apps resources as we do now for appcache, in some cache directory.
2- at first gecko run, populate the SW cache with these resources, and add a `X-preloaded: true` header to the resource. I hear Andrew's concern on how long that could take on first run, and we should measure that.
3- when the app runs, it can check the X-preloaded header to update the cached version if needed.

An another option would be to do the step 2- at first run of an app. That will slow down first run for sure, but not first boot too much.

Comment 33

[Ben Kelly [:bkelly, not reviewing]]

(In reply to Fabrice Desré [:fabrice] from comment #32)
> 2- at first gecko run, populate the SW cache with these resources, and add a
> `X-preloaded: true` header to the resource. I hear Andrew's concern on how
> long that could take on first run, and we should measure that.

Seems there's a fair amount of time in the FTU wizard to do this in the background.  The SW cache could be populated in the background while the user types in their wifi password, navigates the tour, signs into FxA, etc.

Comment 34

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Fabrice Desré [:fabrice] from comment #32)
> I hear Andrew's concern on how
> long that could take on first run, and we should measure that.

I'm not just concerned about time; I'm also concerned about battery life and heat.  It's not a great omen if it seems like the FTU screens are melting the device down when the user first turns the device on.  On my Nexus 4 device (which has relatively respectable hardware) running Android 5.1 updating a bunch of apps, the device gets noticeable hotspots and I/O backlogs cause noticeable emergent jank as async tasks take longer to perform.

But arguably that's something that can be dealt with incrementally.  If we have the mechanism you propose that runs on gecko-start, we can later just reuse that same mechanism at build time.

Comment 35

[[:fabrice] Fabrice Desré]

For sure we should use the system app knowledge of where we are in the bootstrap cycle to get an experience as good as possible. Just don't forget that we also need to support builds with no FTU.

Comment 36

[Francisco Jordano [:arcturus] [:francisco]]

(In reply to Fabrice Desré [:fabrice] from comment #35)
> For sure we should use the system app knowledge of where we are in the
> bootstrap cycle to get an experience as good as possible. Just don't forget
> that we also need to support builds with no FTU.

Agree, also we can configure which app will be launched as FTU, so we cannot trust that all phones will ship with our FTU app.

Don't know if this should be the bug to talk about, but long ago we were talking task to be perform on after an OTA update, so far I think we solved this at system app level (bookmarks porting from browser app to system app), so perhaps we could go for a similar solution.

Comment 37

[[:fabrice] Fabrice Desré]

Hm, I think Marshall was going to take this one?

Comment 38

[Marshall Culpepper [:marshall_law]]

Ah yes, apologies.. I meant to assign this to myself after catching up w/ Fabrice. Francisco, do you mind if I steal this back? :)

Comment 39

[Tim Guan-tin Chien [:timdream] (please needinfo)]

It looks like the direction we are heading right now can't avoid comment 26 or will result in duplicate code if we want to avoid (pre-)start-up spin. But I don't have a strong opinion if everyone if fine with it.

Marshall, would you mind tell the build script forks on what data to generate and which directory to push the data to? It's definitely not going to be zip files in /system/b2g/webapps right?

Comment 40

[Francisco Jordano [:arcturus] [:francisco]]

(In reply to Marshall Culpepper [:marshall_law] from comment #38)
> Ah yes, apologies.. I meant to assign this to myself after catching up w/
> Fabrice. Francisco, do you mind if I steal this back? :)

No prob :)

Comment 41

[Nikhil Marathe [:nsm] (No longer reading bugmail, please needinfo?)]

Does this need to block shipping v1 by 41? Please re-add if there are reasons I don't know about.

Comment 42

[Noemí Freire (:noemi)]

Let's block then v2 and see how these b2g blocking bugs can be prioritized. Thanks!

Comment 43

[Francisco Jordano [:arcturus] [:francisco]]

Hei Marshall,

Just assigned to you, any progress here?

Comment 44

[Marshall Culpepper [:marshall_law]]

No progress yet, I'll start working it this week

Comment 45

[Tim Guan-tin Chien [:timdream] (please needinfo)]

(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #39)
> Marshall, would you mind tell the build script forks on what data to
> generate and which directory to push the data to? It's definitely not going
> to be zip files in /system/b2g/webapps right?

We'll still need this and presumably create another FxOS Gaia Build bug for it....

Comment 46

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Unless this bug also covers that.

Comment 47

[Hema Koka [:hema]]

Quick Update: Just spoke to Marshall, he has been helping out on Spark related metrics issues late last week and is back on this bug now. He will keep this bug updated as he makes progress on it this week.

Thanks
Hema

Comment 48

[Francisco Jordano [:arcturus] [:francisco]]

Hei Marshall,

just setup the target milestone as we moved from using the whiteboard to those milestones. If you don't think is going to be ready for that, please change at will :)

Comment 49

[Marshall Culpepper [:marshall_law]]

Hey guys, I apologize for the long delay, this bug now has my full attention. 

After reading the discussion here, and poking around in AppCache and ServiceWorkerCache, I have a basic plan of action for tackling this. I agree with Tim in comment 45 that we should separate this out into Gaia and Gecko bugs.

My thoughts for Gaia:
* Packaged apps that want to preload their SW cache can list those resources in an array of URLs pointed to by the webapp manifest under the key 'swcache-preload'.

I realize this is similar in concept (and maybe less flexible) to the AppCache manifest, but I keep bumping up against corner cases in my conception of trying to discover these resources automatically via a custom xulrunner build environment, or even worse in the device itself.

* The gaia build will then just need to prefetch each of the URLs in the manifest, and can avoid using the version-tied cache schema by putting them directly under a 'swcache' dir in the profile

My thoughts for Gecko:
* Expose the SW cache via the nsICacheStorageService so we can preload SW cache from Webapps.jsm (or a new module used by Webapps.jsm, doesn't matter much to me)

* At first boot, where Webapps.jsm currently fills the app cache, preload the SW cache for each app with the 'swcache-preload' and 'swcache' dir from above

My ETA for the Gaia bug would be to have a patch up early next week, and Gecko late next week, early week after, with reviews to follow.

Comment 50

[Ben Kelly [:bkelly, not reviewing]]

(In reply to Marshall Culpepper [:marshall_law] from comment #49)
> My thoughts for Gecko:
> * Expose the SW cache via the nsICacheStorageService so we can preload SW
> cache from Webapps.jsm (or a new module used by Webapps.jsm, doesn't matter
> much to me)

I don't think it belongs in nsICacheStorageService.  That is for the http cache and Cache API is a wholly separate thing.

Is there any reason you can't use the webidl chrome-only constructor added in bug 1160138?

You would basically do something like:

  Cu.import("resource://gre/modules/Services.jsm");

  var url = 'http://gaiamobile.org/fm/app/views/main/index.html';
  var uri = Services.io.newURI(url, null, null);
  // you would need to get the app principal instead of the NoAppCodebase principal here
  var principal = Services.scriptSecurityManager.getNoAppCodebasePrincipal(uri);

  var c1 = new CacheStorage(principal, 'content');

Comment 51

[Francisco Jordano [:arcturus] [:francisco]]

(In reply to Marshall Culpepper [:marshall_law] from comment #49)
> Hey guys, I apologize for the long delay, this bug now has my full
> attention. 

\o/ beter late than never!

> 
> After reading the discussion here, and poking around in AppCache and
> ServiceWorkerCache, I have a basic plan of action for tackling this. I agree
> with Tim in comment 45 that we should separate this out into Gaia and Gecko
> bugs.

+1

> 
> My thoughts for Gaia:
> * Packaged apps that want to preload their SW cache can list those resources
> in an array of URLs pointed to by the webapp manifest under the key
> 'swcache-preload'.

Sounds good to me, we can avoid a lot of corner cases, specially when we deal with shared resources and so on.

> 
> I realize this is similar in concept (and maybe less flexible) to the
> AppCache manifest, but I keep bumping up against corner cases in my
> conception of trying to discover these resources automatically via a custom
> xulrunner build environment, or even worse in the device itself.
> 
> * The gaia build will then just need to prefetch each of the URLs in the
> manifest, and can avoid using the version-tied cache schema by putting them
> directly under a 'swcache' dir in the profile

Could we potentially do a step where each app could fill, any cache, with the content in the way they want?

Let me explain that cause it sounds weird. With SW we are planning to use not just for the offline problem, but also for other problems, like pregenerated content, translations (storing content already translated to avoid l10n first time execution problems), etc. Maybe some apps would like to pre-fill the content of some cache name based on the build process, as we do right now with the per-app build steps:
https://github.com/mozilla-b2g/gaia/blob/master/apps/costcontrol/build/build.js

Do you think that's something feasible, or is going to be complicated to add?

> 
> My thoughts for Gecko:
> * Expose the SW cache via the nsICacheStorageService so we can preload SW
> cache from Webapps.jsm (or a new module used by Webapps.jsm, doesn't matter
> much to me)
> 
> * At first boot, where Webapps.jsm currently fills the app cache, preload
> the SW cache for each app with the 'swcache-preload' and 'swcache' dir from
> above
> 
> My ETA for the Gaia bug would be to have a patch up early next week, and
> Gecko late next week, early week after, with reviews to follow.

Comment 52

[(no longer active)]

FWIW, I suggest checking in with Jonas about the current plans on how to expose the app packages to the SW.  The last time we spoke to him about this, the plan was to drop in the package in the Necko cache, not the DOM cache, but the plans were in flux as of about 10 days ago.

Comment 53

[[:fabrice] Fabrice Desré]

(In reply to Marshall Culpepper [:marshall_law] from comment #49)

> * The gaia build will then just need to prefetch each of the URLs in the
> manifest, and can avoid using the version-tied cache schema by putting them
> directly under a 'swcache' dir in the profile

There is no profile directory available at build time on device, and it gets wiped when doing a factory reset. So I guess you'll have to store these files like the appcache ones, in a subdirectoy alongside the app's zip on the system partition.

Comment 54

[[:fabrice] Fabrice Desré]

Except that there will be no zip anymore of course. Or maybe we can just use the zip to preload.

Comment 55

[Francisco Jordano [:arcturus] [:francisco]]

I think Ehsan did a great comment pointing out that Jonas wants to use the necko cache.

IMO, despite that we use the necko cache to drop the package and serve the files from there, some apps could still require to prefill a cache with some content. Note that I didn't comment to use that cache for offline, but for any other usage the app could make of it.

Comment 56

[Salvador de la Puente González [:salva]]

Why to not simply run a specific service worker in the build step? We can install the specified SW in the device itself and the worker could simply complete its installation process. We could provide a ServiceWorkerWare based helper to simply add a list of files (we already have one [1]).

[1] https://github.com/gaia-components/serviceworkerware/#staticcacher

Comment 57

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

I think the easiest solution here is that we during the FTE open all pre-pinned apps in a hidden <iframe mozbrowser> and then use application logic to register the SW and put whatever we need to in the cache.

Comment 58

[[:fabrice] Fabrice Desré]

(In reply to Jonas Sicking (:sicking) from comment #57)
> I think the easiest solution here is that we during the FTE open all
> pre-pinned apps in a hidden <iframe mozbrowser> and then use application
> logic to register the SW and put whatever we need to in the cache.

I agree. The hard part is pre-pinning https resources. Last I looked at that, I didn't find a way to create the appropriate cache entry because you need to provide the security info bits and it's hard to do without hitting the network and completing the tls handshake.

Comment 59

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Once we have bug 1180092 fixed, we can figure that out.

Should we file a new bugs specifically on figuring out pre-caching and pre-opening apps during FTE?

This bug is IMO WONTFIX.