1228974 - [TV][2.5] Preloaded hosted app offline support did not work

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::TV, defect, P1)

Description

[Joseph Yeh [:joseph]]

When offline, open the preloaded hosted app for the first time (the app had never online before), screen shows server not found page instead of the preloaded cached page.

It works normally if the app had online before.

STR:

1. Use preload.py to bundle webapp from a given .webapp URL (e.g. https://bits.wikimedia.org/WikipediaMobileFirefoxOS/app.appcache).

2. Put the preloaded files under gaia/external-apps/ folder.

3. add the path external-apps/* to apps-engineering.list .

4. Rebuild gaia.

5. Make sure the preloaded files appear in profile/webapps folder.

6. Make your device offline.

7. Open the app from app deck.


Actual Result: 
 - Showing server not found page

Expected Result:
 - Showing preloaded cached page

Comment 1

[Joseph Yeh [:joseph]]

Hi Rickey, are there anything wrong with the process of preloading hosted app?

Comment 2

[Fischer [:Fischer]]

Attachment: preload_appcache_example_wikipedia.zip

Hi Fabrice,

We are trying to preload hosted app with app cache by following the instructions from https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/Market_customizations_guide#Building_Prebundled_web_apps

After running DEVICE_DEBUG=1 make command, we can find our example hosted wikipedia app inside ./profile/webapps/ and it contains the cache folder having resources pointed by the app.appcache file.

However, the appcache dosen't work when we open our hosted wikipedia app 1st time without internet connected.
We expect the pre-cached content shows up but actually it shows the message of no connection.

This issue happens both on the mulet and on a real Flame phone.
In fact, we could even see the message saying the cache is pushed to the Flame phone, like:

>  push: /Users/foxbrush/Projects/tmp/tmp_gaia/profile/webapps/{599095a7-8b4b-484d-af10-87eb019cf1fc}/cache/WikipediaMobileFirefoxOS/js/lib/MobileFrontend/javascripts/filepage.js
>  -> //system/b2g/webapps/{599095a7-8b4b-484d-af10-87eb019cf1fc}/cache/WikipediaMobileFirefoxOS/js/lib/MobileFrontend/javascripts/filepage.js
>

We are guessing maybe there is something under the Gecko we are missing.

Could you please look at this issue?

Thank you

p.s:
1) Our real purpose is to use this preload hosted app with appcache technique on the TV marketplace app so user can still see something on the TV marketplace even without internet.

2) The attachment: preload_appcache_example_wikipedia.zip is the hosted wikipedia app downloaded by the preload-app-toolkit. We are using it as the test example.

Comment 3

[Junior [inactive]]

For Linux case, the appcache still doesn't work after the app was opened online and closed by window.close().
It shows only one word "error" 

Both case are with message "XXX FIXME : Dispatch a webapps-launch: undefined" from
https://dxr.mozilla.org/mozilla-central/source/b2g/components/SystemAppProxy.jsm#128

Comment 4

[Shian-Yow Wu [:swu]]

[Blocking Requested - why for this release]:

This blocks TV marketplace for 2.5.

Comment 5

[Ethan Tseng [:ethan]]

(In reply to Fischer from comment #2)
Hi Henry,
Could you specify the code of AppCache in Gecko that Fischer can debug?

Comment 6

[Junior [inactive]]

(In reply to Junior [:junior] (ooo 12/2-12/4) from comment #3)
> For Linux case, the appcache still doesn't work after the app was opened
> online and closed by window.close().
> It shows only one word "error" 
> 
This is nothing to do with this bug. The reason is |navigator.onLine| always returns true.

> Both case are with message "XXX FIXME : Dispatch a webapps-launch:
> undefined" from
> https://dxr.mozilla.org/mozilla-central/source/b2g/components/SystemAppProxy.
> jsm#128

Comment 7

[[:fabrice] Fabrice Desré]

The code in gecko that actually loads this content in the cache is at https://mxr.mozilla.org/mozilla-central/source/dom/apps/OfflineCacheInstaller.jsm

First thing to do would be to enable the debug messages at line 36

Comment 8

[Fischer [:Fischer]]

Hi Fabrice,

We found that 
1) installCache is never called @Line:211, https://mxr.mozilla.org/mozilla-central/source/dom/apps/OfflineCacheInstaller.jsm

2) The DOMApplicationRegistry object at Webapps.jsm would call OfflineCacheInstaller.installCache through the updateOfflineCacheForApp method on init call @Line:221, https://dxr.mozilla.org/mozilla-central/source/dom/apps/Webapps.jsm
However, we did not see the call of DOMApplicationRegistry.init.
As a result, the DOMApplicationRegistry.updateOfflineCacheForApp and OfflineCacheInstaller.installCache are never called from our observation.

The above 2 situations happen both during 
  - the 1st time of the app launch without internet and 
  - launch the app with internet to have appcache downloaded once and then open the app again without internet

However, during app launch we can see DOMApplicationRegistry.launch is called, @Line1666, https://dxr.mozilla.org/mozilla-central/source/dom/apps/Webapps.jsm
We also can observe the app object is created with the kind property set to 'hosted-appcache'.

Could you please look at some questions below:
1) Does the kind property with 'hosted-appcache' represent the preload hosted app is recognized as the hosted app with preloaded appcache ?

2) When would DOMApplicationRegistry.init be called ? Why we can not observe its calling ?

Thanks for your help.

Comment 9

[[:fabrice] Fabrice Desré]

DOMApplicationRegistry.init() is only called once when b2g starts. If this is the very first start or after an OTA update, we also run code to initialize the application registry, including https://dxr.mozilla.org/mozilla-central/source/dom/apps/Webapps.jsm#824 that will preload the app-cache. Are you seeing that being called on first start? You need a fresh profile.

You're right that kind == 'hosted-appcache' means that we recognized this app as having an app cache. We base that on the appcache_path presence in the app manifest though, not on the preloading step.

Comment 10

[Fischer [:Fischer]]

Hi Fabrice,

We found a ReferenceError in OfflineCacheInstaller.jsm which would cause the operation of loading metadata is unable to proceed.
Please see bug 1230091.

However, even fixing that ReferenceError, the appcache still does not show for the 1st time launch without internet.

We can see inside OfflineCacheInstaller.jsm:
- During the b2g starts, the installCache function is invoked and the metadataLoaded promise is resolved.
- After metadata loaded, then storeCache function is called.
- Then, the onCacheEntryAvailable event handler is called @https://dxr.mozilla.org/mozilla-central/source/dom/apps/OfflineCacheInstaller.jsm#58
- Inside the onCacheEntryAvailable handler, we can see the cache files are loaded, such as:
> 12-03 03:45:52.676: I/Gecko(19694): -*-*- OfflineCacheInstaller.jsm :
>   /system/b2g/webapps/{7022c093-6e5c-6b4e-aa5b-2216a18743fa}/cache/WikipediaMobileFirefoxOS/404.html
>   -> https://bits.wikimedia.org/WikipediaMobileFirefoxOS/404.html (2)
> 12-03 03:45:53.016: I/Gecko(19694): -*-*- OfflineCacheInstaller.jsm :
>   /system/b2g/webapps/{7022c093-6e5c-6b4e-aa5b-2216a18743fa}/cache/WikipediaMobileFirefoxOS/css/image/ios/toolbar-forward.png
>   -> https://bits.wikimedia.org/WikipediaMobileFirefoxOS/css/image/ios/toolbar-forward.png (2)
  @https://dxr.mozilla.org/mozilla-central/source/dom/apps/OfflineCacheInstaller.jsm#85

If we open the hosted Wikipedia app with internet, we can see logs like
> 12-03 05:13:44.614: I/Gecko(21903): [Child 21903] WARNING: Could not get disk status from nsIDiskSpaceWatcher:
>   file /home/fischer/Projects/gecko-dev/uriloader/prefetch/nsOfflineCacheUpdateService.cpp, line 319
> 12-03 05:14:02.824: I/GeckoConsole(19694): Offline cache update done, URL=https://bits.wikimedia.org/WikipediaMobileFirefoxOS/app.appcache
From the logs, it seems the nsOfflineCacheUpdateService.cpp and nsOfflineCacheUpdate.cpp try to get and update offline cache.

Then turning off internet, open the app again. We can see similar operations again.
> 12-03 05:14:28.894: I/Gecko(21933): [Child 21933] WARNING: Could not get disk status from nsIDiskSpaceWatcher: 
>   file /home/fischer/Projects/gecko-dev/uriloader/prefetch/nsOfflineCacheUpdateService.cpp, line 319
> 12-03 05:14:31.544: I/GeckoConsole(19694): Offline cache update error, URL=https://bits.wikimedia.org/WikipediaMobileFirefoxOS/app.appcache

1) Does the call of the onCacheEntryAvailable handler means the preloaded cache is found and loaded successfully ?
2) Are the nsOfflineCacheUpdateService.cpp and nsOfflineCacheUpdate.cpp in charge of loading and serving appcache for web ?

Thanks for the help

Comment 11

[Fischer [:Fischer]]

Attachment: app_cache_log_20151203_1819.txt

Comment 12

[[:fabrice] Fabrice Desré]

Your description looks correct to me. I think one issue may be that you are trying to add https uri in the cache, but you don't have the security info at this time so we don't retrieve them later properly without hitting the network.

Can you try with a site served from http instead?

Comment 13

[Fischer [:Fischer]]

Hi Fabrice,

The issue exists with a site served from http as well.
However, we didn't see the call from nsOfflineCacheUpdateService.cpp when opening the http-hosted preload app.
Besides this, we didn't see the call from AppCacheStorage.cpp either.

Where might be the entry point that gecko starts to find and load offline app cache ?

Thank you for the help.

Comment 14

[[:fabrice] Fabrice Desré]

Honza, is there any log level that we can turn on to see what's happening there?

Comment 15

[Honza Bambas (:mayhemer)]

Hmm.. there are, but may need a trained eye to overlook.  The modules are:  cache2:5,cache:5,nsOfflineCacheUpdate:5.  It's also good to add 'timestamp' (w/o any :N suffix) module to get timings in the log.

Comment 16

[Fischer [:Fischer]]

Attachment: preload_appcache_http_example_tvmarketplace.zip

Comment 17

[Ricky Chien [:rickychien] (inactive)]

STR is correct. Clear the needinfo request since Fischer is working on it. if you have further question please ni me =)

Comment 18

[Thinker Li [:sinker]]

Some random info:
 - Fischer said there is not sqlite db for appcache after building.
 - cache2 has move cache directory to $(HOME)/.cache from user's profile directory.
 - We has a new building system for b2g.

Comment 19

[[:fabrice] Fabrice Desré]

(In reply to Thinker Li [:sinker] from comment #18)
> Some random info:
>  - Fischer said there is not sqlite db for appcache after building.
>  - cache2 has move cache directory to $(HOME)/.cache from user's profile
> directory.
>  - We has a new building system for b2g.

None of that should be a problem since we don't prebuild the cache. We ship all the files in a $app/cache/ directory and recursively add them all to the cache at first run or after a system update.

Comment 20

[Fischer [:Fischer]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1dd5596e30da

Comment 21

[Fischer [:Fischer]]

Attachment: correct the generation of group ID

This patch correct the generation of group id so offline cache could be loaded with the right group id

Comment 22

[Carsten Book [:Tomcat]]

(In reply to Shian-Yow Wu [:swu] from comment #4)
> [Blocking Requested - why for this release]:
> 
> This blocks TV marketplace for 2.5.

mahe, josh i guess this need your approval

Comment 23

[Josh Cheng [:josh]]

Hi Fischer,
Please raise 2.5 Gaia uplift request for approval. Thanks

Comment 24

[Fischer [:Fischer]]

Comment on attachment 8697324 [details] [diff] [review]
correct the generation of  group ID

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): The wrong group id
User impact if declined: Preloaded offline cache wouldn't work so user wouldn;t be able to see content of TV marketplace at the 1st time launch if no internet connection.
Testing completed: Yes
Risk to taking this patch (and alternatives if risky): Low 
String or UUID changes made by this patch: None

Comment 25

[Josh Cheng [:josh]]

Comment on attachment 8697324 [details] [diff] [review]
correct the generation of  group ID

Approve for TV 2.5

Comment 26

[Fischer [:Fischer]]

Hi Carsten,
Could you please help to land this patch?
thanks

Comment 27

[Carsten Book [:Tomcat]]

landed on 2.5 as https://hg.mozilla.org/releases/mozilla-b2g44_v2_5/rev/ff08f85000b5

Comment 28

[Fischer [:Fischer]]

Hi Carsten,

The patch is not landed to the master.
Could you please help to land ?
Thanks

Comment 29

[Carsten Book [:Tomcat]]

(In reply to [:Fischer]Fischer from comment #28)
> Hi Carsten,
> 
> The patch is not landed to the master.
> Could you please help to land ?
> Thanks

yeah sure :) done !

https://hg.mozilla.org/integration/b2g-inbound/rev/4cfcce1089fb

Comment 30

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/4cfcce1089fb

Comment 31

[Mahendra Potharaju [:mahe]]

Removing NI