1009531 - Failing to use preloaded cache for hosted applications over https

Status: RESOLVED FIXED

(Core :: Networking: Cache, defect)

Description

[Fredrik Lanker [:lanker]]

Attachment: appcache_fallback.tar.xz

The FALLBACK directive of appcache for hosted applications in Firefox OS is not working if an application is served via https (http works fine).

When SSL is used in conjunction with appcache, nsHttpChannel::OpenCacheInputStream checks that either the security info needs to be cached (mCachedSecurityInfo) or the application needs to be loaded from cache (mLoadedFromApplicationCache). Otherwise an error is generated, resulting in the "... requires an Internet connection" popup, or if it's a debug build, an assert is triggered.

Steps to reproduce:
1. Turn off network on device
2. Install a hosted application (served via https) with a fallback cache, see attached test application
3. Try to access app

Expected result:
FALLBACK resources are rendered.

Actual result:
Error message is displayed "... requires an Internet connection". Or if it's a debug build, an assert is triggered.

The following change works for us:

--
diff --git a/netwerk/protocol/http/nsHttpChannel.cpp b/netwerk/protocol/http/nsHttpChannel.cpp
index bdf45f9..21d49b7 100644
--- a/netwerk/protocol/http/nsHttpChannel.cpp
+++ b/netwerk/protocol/http/nsHttpChannel.cpp
@@ -3409,8 +3409,7 @@ nsHttpChannel::OpenCacheInputStream(nsICacheEntry* cacheEntry, bool startBufferi
 
         // XXX: We should not be skilling this check in the offline cache
         // case, but we have to do so now to work around bug 794507.
-        MOZ_ASSERT(mCachedSecurityInfo || mLoadedFromApplicationCache);
-        if (!mCachedSecurityInfo && !mLoadedFromApplicationCache) {
+        if (!mCachedSecurityInfo && !mLoadedFromApplicationCache && !mFallbackChannel) {
             LOG(("mCacheEntry->GetSecurityInfo returned success but did not "
                  "return the security info [channel=%p, entry=%p]",
                  this, cacheEntry));
--

The idea is that if we're in a fallback it's fine to not have security info. Is this OK? Or should mLoadedFromApplicationCache be set in this case?

Comment 1

[Honza Bambas (:mayhemer)]

We persist sec info for https resources in app cache (bug 794507).  I don't have unpacker for xz to check the test case.  If this asserts something else is broken.

Comment 2

[Honza Bambas (:mayhemer)]

Can you remove && !mLoadedFromApplicationCache and check with some ssl resources?  If it regularly fails, then we still don't correctly persist sec info in app cache.

Comment 3

[Fredrik Lanker [:lanker]]

Attachment: appcache_fallback_test.zip

Sorry, attaching a zip instead.

I removed the !mLoadedFromApplicationCache check, but no change. The assert/popup happens before we've ever hit the network (it's turned off on the device)[1], so I think in this case it's OK that we don't have any cached security info?

The idea is that the fallback should be used from the time the application is installed (in this case, built together with Gaia) until the first time the device gets connected to the network.

[1] I'm not even using a real URI for the hosted application in my test.

Comment 4

[Honza Bambas (:mayhemer)]

Attachment: appcache.zip

It works for me.  I was testing with desktop Firefox Nightly debug build.  I have a local server at test.local.i7 domain with ssl.  If you put the content to /bugs/1009531-https-fallback on your server then go to https://test.local.i7/bugs/1009531-https-fallback/appcache/ it will ask for installation.  Press install.  Exit from Firefox.  Run the installed app (on Win there is a desktop icon).  I get "Fallback!", no assertions.

What Gecko version are you on?

This could also well be a bug in the JS code that performs the offline cache update/first load (it's a duplicate of the platform code...)

Comment 5

[Honza Bambas (:mayhemer)]

Michael, could this be a bug in AppCacheUtils.jsm or somewhere around?

Comment 6

[Mike Ratcliffe [:miker] [:mratcliffe] [:mikeratcliffe]]

It could be... I am on paternity leave at the moment but I can take a look when I return.

Comment 7

[Fredrik Lanker [:lanker]]

(In reply to Honza Bambas (:mayhemer) from comment #4)
> It works for me.  I was testing with desktop Firefox Nightly debug build.  I
> have a local server at test.local.i7 domain with ssl.  If you put the
> content to /bugs/1009531-https-fallback on your server then go to
> https://test.local.i7/bugs/1009531-https-fallback/appcache/ it will ask for
> installation.  Press install.  Exit from Firefox.  Run the installed app (on
> Win there is a desktop icon).  I get "Fallback!", no assertions.

A difference (besides desktop Firefox vs. b2g) is that you're online while installing the application. I'm not that familiar with the code or flow here, but I think the cache is populated at install time, and I guess that includes security info as well. My assumption is that we need network to be able to get security info?

In my case, the application is in the system image that a device is flashed with. Then the device is booted and the application is started, all the time without any network connection.

> What Gecko version are you on?

Latest on mozilla-central.

> This could also well be a bug in the JS code that performs the offline cache
> update/first load (it's a duplicate of the platform code...)

I've tried to verify that the cache is populated correctly. I can at least see an entry in /data/local/OfflineCache/index.sqlite, and it has the right type too (after bug 1002383 was merged). If there's somewhere else I can look to verify, in code or in file system, I'm happy to take a look.

Comment 8

[Honza Bambas (:mayhemer)]

(In reply to Fredrik Lanker [:lanker] from comment #7)
> (In reply to Honza Bambas (:mayhemer) from comment #4)
> > It works for me.  I was testing with desktop Firefox Nightly debug build.  I
> > have a local server at test.local.i7 domain with ssl.  If you put the
> > content to /bugs/1009531-https-fallback on your server then go to
> > https://test.local.i7/bugs/1009531-https-fallback/appcache/ it will ask for
> > installation.  Press install.  Exit from Firefox.  Run the installed app (on
> > Win there is a desktop icon).  I get "Fallback!", no assertions.
> 
> A difference (besides desktop Firefox vs. b2g) is that you're online while
> installing the application. 

When exactly I'm online?  Not clear from the comment.

> I'm not that familiar with the code or flow
> here, but I think the cache is populated at install time, and I guess that
> includes security info as well. My assumption is that we need network to be
> able to get security info?

You need to be online to even get the resources :D  But I may not understand what you mean by "install time"  Can you please be more specific?

> 
> In my case, the application is in the system image that a device is flashed
> with. 

It might be that the image has been created before we have fixed bug 794507.  Then there will be no sec info in the entries metadata.  No help here until you update the app from network or recreate the image using a cache populated with a code having bug 794507 fixed.

> Then the device is booted and the application is started, all the time
> without any network connection.
> 
> > What Gecko version are you on?
> 
> Latest on mozilla-central.

I assume the cache client (the b2g phone you can see the bug on), but not the code creating the flash image, right?

> 
> > This could also well be a bug in the JS code that performs the offline cache
> > update/first load (it's a duplicate of the platform code...)
> 
> I've tried to verify that the cache is populated correctly. I can at least
> see an entry in /data/local/OfflineCache/index.sqlite, and it has the right
> type too (after bug 1002383 was merged). If there's somewhere else I can
> look to verify, in code or in file system, I'm happy to take a look.


Check it's metadata, you need to:

* select replace(hex(MetaData), '00', '0D0A') from moz_cache where key like 'https:%';
* convert the hex values at e.g. http://www.asciitohex.com/

You should see security-info there.

Comment 9

[Fredrik Lanker [:lanker]]

(In reply to Honza Bambas (:mayhemer) from comment #8)
> (In reply to Fredrik Lanker [:lanker] from comment #7)
> > (In reply to Honza Bambas (:mayhemer) from comment #4)
> > > It works for me.  I was testing with desktop Firefox Nightly debug build.  I
> > > have a local server at test.local.i7 domain with ssl.  If you put the
> > > content to /bugs/1009531-https-fallback on your server then go to
> > > https://test.local.i7/bugs/1009531-https-fallback/appcache/ it will ask for
> > > installation.  Press install.  Exit from Firefox.  Run the installed app (on
> > > Win there is a desktop icon).  I get "Fallback!", no assertions.
> > 
> > A difference (besides desktop Firefox vs. b2g) is that you're online while
> > installing the application. 
> 
> When exactly I'm online?  Not clear from the comment.

When the application is installing, i.e., when you press the "install" button and a few seconds after that.

> > I'm not that familiar with the code or flow
> > here, but I think the cache is populated at install time, and I guess that
> > includes security info as well. My assumption is that we need network to be
> > able to get security info?
> 
> You need to be online to even get the resources :D

This is probably the cause for the misunderstanding. I'm trying to use the FALLBACK part of the application cache manifest. As far as I understand, you don't need to be online for that to work. In our case, we've never had any network before trying to start the pre-installed application, then the fallback doesn't work (since no security info could've been fetched).

This use case is probably not relevant, or even possible, on a desktop browser. But on a phone it makes sense.

> But I may not understand what you mean by "install time"  Can you please be more specific?

"Install time" for pre-installed applications on a phone seems to be sometime during the boot. At least that's when the cache is populated.

> > In my case, the application is in the system image that a device is flashed
> > with. 
> 
> It might be that the image has been created before we have fixed bug 794507.
> Then there will be no sec info in the entries metadata.  No help here until
> you update the app from network or recreate the image using a cache
> populated with a code having bug 794507 fixed.

The code I'm using contains the fix for bug 794507.
 
> > Then the device is booted and the application is started, all the time
> > without any network connection.
> > 
> > > What Gecko version are you on?
> > 
> > Latest on mozilla-central.
> 
> I assume the cache client (the b2g phone you can see the bug on), but not
> the code creating the flash image, right?

The phone is running the latest code from m-c, plus my test application, which is built with Gaia. I.e., I'm building and flashing the phone with latest from m-c.

> > > This could also well be a bug in the JS code that performs the offline cache
> > > update/first load (it's a duplicate of the platform code...)
> > 
> > I've tried to verify that the cache is populated correctly. I can at least
> > see an entry in /data/local/OfflineCache/index.sqlite, and it has the right
> > type too (after bug 1002383 was merged). If there's somewhere else I can
> > look to verify, in code or in file system, I'm happy to take a look.
> 
> 
> Check it's metadata, you need to:
> 
> * select replace(hex(MetaData), '00', '0D0A') from moz_cache where key like
> 'https:%';
> * convert the hex values at e.g. http://www.asciitohex.com/
> 
> You should see security-info there.

Great, thanks! For my test application I get:

  request-method
  GET
  response-head
  HTTP/1.1 200 OK

So no security info.

--
Trying to clarify the step-by-step instructions:

0. Get the code for b2g
1. Put your application somewhere in Gaia, e.g., in apps/
2. Build so you get a flashable image
3. Flash the phone
4. Make sure you don't auto-connect to any network at boot and that you're offline
5. Start the test application

Expected: the fallback page should be shown
Actual: The "... requires an Internet connection" popup is shown.

Comment 10

[Honza Bambas (:mayhemer)]

(In reply to Fredrik Lanker [:lanker] from comment #9)
> (In reply to Honza Bambas (:mayhemer) from comment #8)
> > (In reply to Fredrik Lanker [:lanker] from comment #7)
> > > (In reply to Honza Bambas (:mayhemer) from comment #4)
> > > > It works for me.  I was testing with desktop Firefox Nightly debug build.  I
> > > > have a local server at test.local.i7 domain with ssl.  If you put the
> > > > content to /bugs/1009531-https-fallback on your server then go to
> > > > https://test.local.i7/bugs/1009531-https-fallback/appcache/ it will ask for
> > > > installation.  Press install.  Exit from Firefox.  Run the installed app (on
> > > > Win there is a desktop icon).  I get "Fallback!", no assertions.
> > > 
> > > A difference (besides desktop Firefox vs. b2g) is that you're online while
> > > installing the application. 
> > 
> > When exactly I'm online?  Not clear from the comment.
> 
> When the application is installing, i.e., when you press the "install"
> button and a few seconds after that.

Sure I am, how else could I install the app?

> 
> > > I'm not that familiar with the code or flow
> > > here, but I think the cache is populated at install time, and I guess that
> > > includes security info as well. My assumption is that we need network to be
> > > able to get security info?
> > 
> > You need to be online to even get the resources :D
> 
> This is probably the cause for the misunderstanding. I'm trying to use the
> FALLBACK part of the application cache manifest. As far as I understand, you
> don't need to be online for that to work. 

You need to be online during the first app cache fetch (the first appcache download) to get the fallback resource that we redirect to when a resource in the namespace cannot be loaded from network (e.g. you are offline, server is down, server responses with 4xx/5xx).

> In our case, we've never had any
> network before trying to start the pre-installed application, then the
> fallback doesn't work (since no security info could've been fetched).

You don't understand correctly how the appcache works.  Appcache must be either 1) downloaded by starting the fetch using the manifest or 2) preinstalled in the image with all necessary resources - that also need to be originally fetched online from the server.  The sec info must already be there - in the flash image!

> 
> This use case is probably not relevant, or even possible, on a desktop
> browser. But on a phone it makes sense.

It works on desktop too.

> 
> > But I may not understand what you mean by "install time"  Can you please be more specific?
> 
> "Install time" for pre-installed applications on a phone seems to be
> sometime during the boot. At least that's when the cache is populated.

Aha!  So you want to say that the app is using appcache but the appcache data are not in the pre-installed image?  Then it's not a bug in appcache but in the way how preinstalled apps are populated.

> 
> > > In my case, the application is in the system image that a device is flashed
> > > with. 
> > 
> > It might be that the image has been created before we have fixed bug 794507.
> > Then there will be no sec info in the entries metadata.  No help here until
> > you update the app from network or recreate the image using a cache
> > populated with a code having bug 794507 fixed.
> 
> The code I'm using contains the fix for bug 794507.

On the client?  Or for creating the image?

>  
> > > Then the device is booted and the application is started, all the time
> > > without any network connection.
> > > 
> > > > What Gecko version are you on?
> > > 
> > > Latest on mozilla-central.
> > 
> > I assume the cache client (the b2g phone you can see the bug on), but not
> > the code creating the flash image, right?
> 
> The phone is running the latest code from m-c, plus my test application,
> which is built with Gaia. I.e., I'm building and flashing the phone with
> latest from m-c.

what exactly means "built with gaia" ?

> 
> > > > This could also well be a bug in the JS code that performs the offline cache
> > > > update/first load (it's a duplicate of the platform code...)
> > > 
> > > I've tried to verify that the cache is populated correctly. I can at least
> > > see an entry in /data/local/OfflineCache/index.sqlite, and it has the right
> > > type too (after bug 1002383 was merged). If there's somewhere else I can
> > > look to verify, in code or in file system, I'm happy to take a look.
> > 
> > 
> > Check it's metadata, you need to:
> > 
> > * select replace(hex(MetaData), '00', '0D0A') from moz_cache where key like
> > 'https:%';
> > * convert the hex values at e.g. http://www.asciitohex.com/
> > 
> > You should see security-info there.
> 
> Great, thanks! For my test application I get:
> 
>   request-method
>   GET
>   response-head
>   HTTP/1.1 200 OK
> 
> So no security info.

And what is the key?  And which code adds this data to the sqlite database?  <-- this is the most important question here!

> 
> --
> Trying to clarify the step-by-step instructions:
> 
> 0. Get the code for b2g
> 1. Put your application somewhere in Gaia, e.g., in apps/

What all the app consist of?

> 2. Build so you get a flashable image
> 3. Flash the phone
> 4. Make sure you don't auto-connect to any network at boot and that you're
> offline
> 5. Start the test application
> 
> Expected: the fallback page should be shown
> Actual: The "... requires an Internet connection" popup is shown.

Comment 11

[Honza Bambas (:mayhemer)]

(In reply to Fredrik Lanker [:lanker] from comment #9)
>   request-method
>   GET
>   response-head
>   HTTP/1.1 200 OK
> 
> So no security info.

BTW, this is the whole response head?  If so, then it's not correct.  There should be the whole response head with Content-type and Content-Length and other headers.

Comment 12

[Fredrik Lanker [:lanker]]

(In reply to Honza Bambas (:mayhemer) from comment #10)
> > In our case, we've never had any
> > network before trying to start the pre-installed application, then the
> > fallback doesn't work (since no security info could've been fetched).
> 
> You don't understand correctly how the appcache works.

That might very well be the case.

> Appcache must be either 1) downloaded by starting the fetch using the manifest or 2)
> preinstalled in the image with all necessary resources - that also need to
> be originally fetched online from the server.  The sec info must already be
> there - in the flash image!

2) is what we're aiming for here. So then maybe we need to preload some security info as well, together with the fallback content? If so, how is that done?

> > "Install time" for pre-installed applications on a phone seems to be
> > sometime during the boot. At least that's when the cache is populated.
> 
> Aha!  So you want to say that the app is using appcache but the appcache
> data are not in the pre-installed image?  Then it's not a bug in appcache
> but in the way how preinstalled apps are populated.

No, the appcache data is also in the image (but maybe we somehow need to add some security info in the image as well, see above).

> > The code I'm using contains the fix for bug 794507.
> 
> On the client?  Or for creating the image?

Both. Latest mozilla-central is used for everything I'm doing.

> what exactly means "built with gaia" ?

Please see the step-by-step instructions in my previous comment.

> > > Check it's metadata, you need to:
> > > 
> > > * select replace(hex(MetaData), '00', '0D0A') from moz_cache where key like
> > > 'https:%';
> > > * convert the hex values at e.g. http://www.asciitohex.com/
> > > 
> > > You should see security-info there.
> > 
> > Great, thanks! For my test application I get:
> > 
> >   request-method
> >   GET
> >   response-head
> >   HTTP/1.1 200 OK
> > 
> > So no security info.
> 
> And what is the key?  And which code adds this data to the sqlite database? 
> <-- this is the most important question here!

$ sqlite3 index.sqlite "select key from moz_cache;"
https://example.com/index.html
https://example.com/cache/manifest.appcache

Both entries have the same MetaData.

Regarding which code that adds it to the sqlite database, I think it's done from dom/apps/src/OfflineCacheInstaller.jsm.

> > --
> > Trying to clarify the step-by-step instructions:
> > 
> > 0. Get the code for b2g
> > 1. Put your application somewhere in Gaia, e.g., in apps/
> 
> What all the app consist of?

The application in this case is the content in appcache_fallback_test.zip.


Please note that it works fine if we're not using SSL/TLS, i.e., if the URIs in metadata.json are changed from https to http.

Comment 13

[Fredrik Lanker [:lanker]]

(In reply to Honza Bambas (:mayhemer) from comment #11)
> (In reply to Fredrik Lanker [:lanker] from comment #9)
> >   request-method
> >   GET
> >   response-head
> >   HTTP/1.1 200 OK
> > 
> > So no security info.
> 
> BTW, this is the whole response head?  If so, then it's not correct.  There
> should be the whole response head with Content-type and Content-Length and
> other headers.

Yes, that is everything in MetaData:

$ sqlite3 index.sqlite "select replace(hex(MetaData), '00', '0D0A') from moz_cache where key like '%example%';"
726571756573742D6D6574686F640D0A4745540D0A726573706F6E73652D686561640D0A485454502F312E3120323030204F4B0D0A0D0A

BTW, I'm getting the same when using plain http too (where fallback works), i.e., no Content-type, etc.

Comment 14

[Honza Bambas (:mayhemer)]

Alex, assigning to you, please see the comment bellow.


(In reply to Fredrik Lanker [:lanker] from comment #12)

Thanks for this rich comment!

> Regarding which code that adds it to the sqlite database, I think it's done
> from dom/apps/src/OfflineCacheInstaller.jsm.

http://hg.mozilla.org/mozilla-central/annotate/d6fccb7c56a8/dom/apps/src/OfflineCacheInstaller.jsm#l55

No sec info added.

This is bad...  That code, despite data is loaded from a file, must provide an actual sec info from the server.

Best is to use xhr to the server origin (actually all server origins in the manifest) and then grab the sec info from the channel.  I can provide a detailed code how to do it.

Also, the whole concept (exposing file content as a content obtained from an https:// resource) may hit the wall of bug 660749 when it's fixed.

Comment 15

[Alexandre Poirot [:ochameau]]

Sure I wrote this code and a lot of time has gone by since then and I switched to new team and project.
I'm too overloaded right now to ensure I can look at this.

Gregor, is there someone around from b2g teams to help on this?

(In reply to Honza Bambas (:mayhemer) from comment #14)
> Best is to use xhr to the server origin (actually all server origins in the
> manifest) and then grab the sec info from the channel.  I can provide a
> detailed code how to do it.

If I understand you correctly, you want to do a request when OfflineCacheInstaller code is executed?
That's not an option as this code is executed on first boot of the device, before the device has any connectivity. Also it sounds weird to me to rely on pulling some security data over network instead of shipping it after having it being reviewed.

> Also, the whole concept (exposing file content as a content obtained from an
> https:// resource) may hit the wall of bug 660749 when it's fixed.

I'm not sure we ever supported, nor want to support preloaded apps on https?

Comment 16

[Jason Smith [:jsmith]]

Do we have a test case hosted somewhere we can use to determine if this is a regression or not? I'd like to find out if this is a regression.

Comment 17

[Honza Bambas (:mayhemer)]

(In reply to Jason Smith [:jsmith] from comment #16)
> Do we have a test case hosted somewhere we can use to determine if this is a
> regression or not? I'd like to find out if this is a regression.

This is definitely not a regressions.

(In reply to Alexandre Poirot (:ochameau) from comment #15)
> If I understand you correctly, you want to do a request when
> OfflineCacheInstaller code is executed?
> That's not an option as this code is executed on first boot of the device,
> before the device has any connectivity. Also it sounds weird to me to rely
> on pulling some security data over network instead of shipping it after
> having it being reviewed.

Didn't know.  Yep, that's definitely a better option.  Still you need to get the sec info somewhere.

> 
> > Also, the whole concept (exposing file content as a content obtained from an
> > https:// resource) may hit the wall of bug 660749 when it's fixed.
> 
> I'm not sure we ever supported, nor want to support preloaded apps on https?

I'm also curious why should the content appear as served via https, actually a fake security.

However, I'm no deep expert to the app model we have, so someone else should discuss this.

Comment 18

[Gregor Wagner [:gwagner]]

Doug will find an owner here.

Comment 19

[Honza Bambas (:mayhemer)]

Doug, ask the question first please.

Comment 20

[Honza Bambas (:mayhemer)]

On the appcache code side we can do only two things:
- allow to use cache entries w/o sec info (I'm in favor, trivial)
- WONTFIX this, since the cache is actually populated broken by an external code (OfflineCacheInstaller.jsm)

Or, delegate to the author of OfflineCacheInstaller.jsm to store also sec-info within OfflineCacheInstaller.jsm.

Comment 21

[Fredrik Lanker [:lanker]]

I'll definitely leave it up to you to decide what to do here, but  

> - allow to use cache entries w/o sec info (I'm in favor, trivial)

makes sense to me too. That was what I was trying to do in the patch in comment 0 (for fallback entries, but it's probably needed for all types).

Comment 22

[Honza Bambas (:mayhemer)]

If you added a callstack first!  We could save so much time...  

I can reproduce with a simple appcache locally and modified code to not store the sec info (as in the old times).  

The problem is that the mLoadedFromApplicationCache flag is set later.  The call to OpenCacheInputStream() happens from OnCacheEntryCheck(), which doesn't set (and must not to!) that flag.

However, this is still a big hack...  This code (the condition) is there to preserve backward compatibility for caches from times we were not storing sec-info in appcache entries.  You are actually abusing this hack only.

Comment 23

[Honza Bambas (:mayhemer)]

Attachment: v1

Comment 24

[Jason Duell]

Comment on attachment 8453614 [details] [diff] [review]
v1

Review of attachment 8453614 [details] [diff] [review]:
-----------------------------------------------------------------

Looks ok to me

Comment 25

[Honza Bambas (:mayhemer)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/9fc90ba5662e
https://tbpl.mozilla.org/?tree=Try&rev=5e31fde76c91

Comment 26

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/9fc90ba5662e