1143526 - [FFOS7715 v2.1] after Power off test icons of homescreen disappear

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Stability, defect)

Description

[ben.song]

Reproduce step:

1.Do poweroff test like bug 1115619 for 4~6 days.

2.Start-up the phone normally.

Ideal result:

Icons of homescreen display completely.

Actual result:

Icons of homescreen disappear

Reproduce rate: 6/7

Comment 1

[ben.song]

Dear Rex, Vance,

Recently we do poweroff test again, the problem reproduced and the rate is 3/4. I have tried to modify sqlite of homescreen, the appearance is like this. Could you help me to analyse this problem, this another problem of adb daemon process died has been resolved today.

Thanks.

Comment 2

[Rex Hung[:rhung]]

The symptom can be reproduced in power-off test with the images which include the following two patches:
1. Attachment 8554380 [details] [diff] in bug#1115619
2. Attachment 8571226 [details] [diff] in bug#1124126

After comparing the difference of homescreen DB data between this symptom and normal case, we found that:
1. The size of 1444454539veemrothilca.sqlite are different.
2. icon picture files named by number(1~15) are not found in the tested devices.
================================================================================
All icons are gone:
drwxr-x---. 3 root root  4096  3月  6 23:51 ./
drwxr-x---. 3 root root  4096  3月  6 23:51 ../
drwxr-x---. 2 root root  4096  3月  6 23:51 1444454539veemrothilca/
-rw-------. 1 root root 38912  3月  6 23:51 1444454539veemrothilca.sqlite
-rw-------. 1 root root     0  3月  6 23:51 1444454539veemrothilca.sqlite-journal

Normal case:
drwxrwxr-x 2 root root  4096  3月 13 17:47 1444454539veemrothilca
-rw-r--r-- 1 root root 45056  3月 13 17:47 1444454539veemrothilca.sqlite
-rw-r--r-- 1 root root     0  3月 13 17:47 1444454539veemrothilca.sqlite-journal
-rw-r--r-- 1 root root  6465  3月 13 17:47 1444454539veemrothilca/1
-rw-r--r-- 1 root root  6991  3月 13 17:47 1444454539veemrothilca/10
-rw-r--r-- 1 root root  5887  3月 13 17:47 1444454539veemrothilca/11
-rw-r--r-- 1 root root  5410  3月 13 17:47 1444454539veemrothilca/12
-rw-r--r-- 1 root root  7602  3月 13 17:47 1444454539veemrothilca/13
-rw-r--r-- 1 root root  7770  3月 13 17:47 1444454539veemrothilca/14
-rw-r--r-- 1 root root  6531  3月 13 17:47 1444454539veemrothilca/15
-rw-r--r-- 1 root root  6308  3月 13 17:47 1444454539veemrothilca/2
-rw-r--r-- 1 root root  6893  3月 13 17:47 1444454539veemrothilca/3
-rw-r--r-- 1 root root  7492  3月 13 17:47 1444454539veemrothilca/4
-rw-r--r-- 1 root root 30401  3月 13 17:47 1444454539veemrothilca/5
-rw-r--r-- 1 root root  9795  3月 13 17:47 1444454539veemrothilca/6
-rw-r--r-- 1 root root 41429  3月 13 17:47 1444454539veemrothilca/7
-rw-r--r-- 1 root root 41818  3月 13 17:47 1444454539veemrothilca/8
-rw-r--r-- 1 root root 41397  3月 13 17:47 1444454539veemrothilca/9

Comment 3

[ben.song]

Dear Rex,Danny,

Base on my analysis, i find in normal device after delete ***+f+app+++verticalhome.gaiamobile.org and restart, the sqlite of verticalhome would rebuild and icons is exist; But in problem device while delete ***+f+app+++verticalhome.gaiamobile.org and restart, the sqlite of verticalhome would alse rebuild but icons is still disappear.

So i think the origin of this problem isn't build process of verticalhome sqlite, but is the origin of sqlite reading icons, perhaps manifest of verticalhome. Do you think so ?

Thanks.

Comment 4

[Vance Chen [:vchen][skype:vance.lucida][[e-mail]]]

Hi Ben -

So the device cannot recovery even after rebooting?

Thanks

Comment 5

[ben.song]

(In reply to Vance Chen [:vchen][[e-mail]] from comment #4)
> Hi Ben -
> 
> So the device cannot recovery even after rebooting?
> 
> Thanks

Dear Vance,

Yes, you are right, even replacing with ***+f+app+++verticalhome.gaiamobile.org of normal device, this problem is still exist.

Thanks.

Comment 6

[yifan [:yifan][:yliao]]

1066492 & 1067720 have different reproduce steps but the same symptom. Looking for the root cause.

Comment 7

[yifan [:yifan][:yliao]]

Attachment: 1067720_fix

Hi Ben,

Could you please try the patch and see if the issue persists? Thank you!

Comment 8

[ben.song]

(In reply to yifan [:yifan][:yliao] from comment #7)
> Created attachment 8577962 [details] [review]
> 1067720_fix
> 
> Hi Ben,
> 
> Could you please try the patch and see if the issue persists? Thank you!

Dear Yifan,

OK, I would land this patch and do poweroff test to verify it.

Thanks.

Comment 9

[Rex Hung[:rhung]]

Remove my NI first since Yifan is investigating this bug.

Comment 10

[ben.song]

Dear Yifan,

After six days poweroff test, this issue reproduced and the rate is 1/2. I would upload data module of problem device soon.

Thank.

Comment 11

[ben.song]

Attachment: data_module.tar.gz

Dear Yifan,

This attachment is the file of data module. Please help me to analyse this problem.

Thanks.

Comment 12

[ben.song]

Dear Yifan,

If as we modified, while we remove directory of ****+f+app+++verticalhome.gaiamobile.org, the database would renew one, and icons should display. But After remove it, the appearance still is no icons.

So I think the origin of this problem probably isn't ****+f+app+++verticalhome.gaiamobile.org writen incompletely.

Comment 13

[ben.song]

Dear Yifan, Vance,

Base on my compare between problem device and normal device, I find the finally origin problem is in directory of /data/local/storage/persistent/chrome/idb, there is a sqlite of 444928743bpopoakbmeawr.ktss_es.sqlite, while replace with a problem 444928743bpopoakbmeawr.ktss_es.sqlite in normal device, this problem appears.

Please help me to fix this problem.

Thanks.

Comment 14

[yifan [:yifan][:yliao]]

Hi Ben,

Could you please attach both the normal and the problem sqlite file here? Thank you!

Comment 15

[yifan [:yifan][:yliao]]

Just to make sure that apart from 444928743bpopoakbmeawr.ktss_es.sqlite, is there any other sqlite file that is broken, either? This sqlite file stores user bookmarked icons to be displayed on the homescreen. I'll look into the loading part in verticalhome first.

Comment 16

[ben.song]

Attachment: normal_and_problem_sqlite.tar.gz

Dear Yifan,

This attachment contains normal sqlite and problem sqlite.

Thanks.

Comment 17

[ben.song]

Dear Yifan,

Only 444928743bpopoakbmeawr.ktss_es.sqlite and ****+f+app+++verticalhome.gaiamobile.org are broken. And if 444928743bpopoakbmeawr.ktss_es.sqlite be OK, ****+f+app+++verticalhome.gaiamobile.org would be OK,too.

In log, I find several error like this:
03-20 21:10:59.230 E/GeckoConsole(  586): [JavaScript Error: "aEvent.target.result is undefined" {file: "resource://gre/modules/DataStoreCursorImpl.jsm" line: 173}]
03-20 21:10:59.240 E/GeckoConsole(  586): [JavaScript Error: "aEvent.target is undefined" {file: "resource://gre/modules/DataStoreCursorImpl.jsm" line: 76}]

After analysing gaia code, I get that probably there is some problem while invoking save of bookmark_source in verticalhome/js/stores/item.js, it would interrupt the read process of ****+f+app+++verticalhome.gaiamobile.org.

Thanks.

Comment 18

[Vance Chen [:vchen][skype:vance.lucida][[e-mail]]]

Clean my and Danny's ni since Yifan is helping on this one

Comment 19

[yifan [:yifan][:yliao]]

There are 3 sources for the icons on the verticalhome: application, bookmark and collection. When booting the verticalhome, it'll load in the sequence stated above. After all 3 sources are loaded can it display the icons.

Tried booting with both normal and incorrect sqlite. When booting with the incorrect sqlite, homescreen icon loading stops at gaia: apps/verticalhome/js/sources/bookmark.js line 70, in the middle of the populate method. 

This is related to what Ben analyzed in comment 17. When the DataStore 'bookmarks_store' is corrupted, it should have issued a DOMError at https://dxr.mozilla.org/mozilla-central/source/dom/datastore/DataStoreCursorImpl.jsm#76 but failed like the log in comment 17. And in gaia the verticalhome app doesn't know this failure and thus doesn't execute the following callback to display icons.

I'll look into what cause the bookmarks_store corrupted, which should be the primary problem since verticalhome doesn't responsible for bookmarks_store creation.

----

Another issue is whether gecko will expose a error or not when DataStore API fails. I assume that we could use a try catch block for the error but in this case it seems the error isn't visible in Gaia. 

Hi Thinker, could you share some idea about this according to comment 17? Thank you!

Comment 20

[ben.song]

Dear Thinker, Yifan,

I find CollectionSource probably would happen the same error like BookmarkSource, pls fix it, too.

Thanks.

Comment 21

[ben.song]

ni Thinker

Comment 22

[Thinker Li [:sinker]]

I don't really understand the problem here.  But, this seems like the database of datastore is not initialized completely.  So, if it is true, I would ask why it is not initialized completely, and why we don't check it and recreate it.

Comment 23

[ben.song]

(In reply to Thinker Li [:sinker] from comment #22)
> I don't really understand the problem here.  But, this seems like the
> database of datastore is not initialized completely.  So, if it is true, I
> would ask why it is not initialized completely, and why we don't check it
> and recreate it.

Dear Thinker,

why it is not initialized completely:

Ben: Because of progressive poweroff test, It would almost poweroff at everytime during first start-up.

why we don't check it and recreate it:

I don't know this question, but during initialition verticalhome sqlite it would invoke bookmarkdatabase, the moment bring up some problem.

Thanks.

Comment 24

[yifan [:yifan][:yliao]]

(In reply to Thinker Li [:sinker] from comment #22)
> I don't really understand the problem here.  But, this seems like the
> database of datastore is not initialized completely.  So, if it is true, I
> would ask why it is not initialized completely, and why we don't check it
> and recreate it.

Yes I'm looking into this problem. I should've described the issue which I'd like to consult you more clearly.

https://wiki.mozilla.org/WebAPI/DataStore#Sync
The Data Store API doesn't provide an example for error catching, which I assume we could use try/catch block to capture a Data Store API access error. The problem here is the error log at https://dxr.mozilla.org/mozilla-central/source/dom/datastore/DataStoreCursorImpl.jsm#76 :

03-20 21:10:59.240 E/GeckoConsole(  586): [JavaScript Error: "aEvent.target is undefined" {file: "resource://gre/modules/DataStoreCursorImpl.jsm" line: 76}]

This line supposed to dispatch a DOMError but it fails at 'aEvent.target is undefined' by itself. Should I file another bug for it?

Comment 25

[yifan [:yifan][:yliao]]

After some digging I figured that the creation of a data store is in https://dxr.mozilla.org/mozilla-central/source/dom/datastore/DataStoreService.cpp#855 . At vertical homescreen start up the verticalhome app uses navigator.getDataStores to get the bookmarks_store, and the DataStoreService will create one if there's none. Once it's created the verticalhome app will get the data store reference, but verticalhome doesn't know if the data store is corrupted or not. Hi Vance, could you please help to find some one familiar in debugging Gecko code since this is urgent? Thank you!

Comment 26

[Vance Chen [:vchen][skype:vance.lucida][[e-mail]]]

(In reply to yifan [:yifan][:yliao] from comment #25)
> After some digging I figured that the creation of a data store is in
> https://dxr.mozilla.org/mozilla-central/source/dom/datastore/
> DataStoreService.cpp#855 . At vertical homescreen start up the verticalhome
> app uses navigator.getDataStores to get the bookmarks_store, and the
> DataStoreService will create one if there's none. 

Hi Yifan, one quick question here : if the DataStoreService will create a new bookmarks_store for vertical homescreen to use if there is none available, how come we still have this issue? Is that because the new bookmarks_store the DataStoreService create is still corrupted?

Also ni Fabrice, hi Fabrice, as Yifan pointed out in Comment#25, we need help from core team to further check this problem, could you help to point we to the person who might move this forward?

Comment 27

[[:fabrice] Fabrice Desré]

Baku owns the implementation of the datastore. It looks to me that the error at https://dxr.mozilla.org/mozilla-central/source/dom/datastore/DataStoreCursorImpl.jsm#76 should be properly fixed too.

Comment 28

[yifan [:yifan][:yliao]]

Thank you Vance and Fabrice. 

(In reply to Vance Chen [:vchen][[e-mail]] from comment #26)
> Hi Yifan, one quick question here : if the DataStoreService will create a
> new bookmarks_store for vertical homescreen to use if there is none
> available, how come we still have this issue? Is that because the new
> bookmarks_store the DataStoreService create is still corrupted?

I assumed that once a corrupted bookmarks_store is created, every subsequent getDataStores request will get the same corrupted one. So once it's created at start up it'll always return the corrupted data store.

Comment 29

[Andrea Marchesini [:baku]]

Attachment: dataStore patch

This patch fixes the DataStore error reporting issue. It turned out that newTxn calls the failure cb with a string and not with a IDB error object.

Comment 30

[yifan [:yifan][:yliao]]

Thank you Andrea! Unassigned myself.

Comment 31

[Andrea Marchesini [:baku]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/3851d1999bd5

Comment 32

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/3851d1999bd5

Comment 33

[ben.song]

Dear Andrea, Yifan, Vance,

The problem reproduced in recent poweroff test, and the rate is 4/4. For the cr has closed, I have created a new cr in bug1152169.

Please help me to resolve the problem, thanks.

Comment 34

[Andrea Marchesini [:baku]]

Let's debug this bug in bug 1152169.

Comment 35

[ben.song]

remove ni for vance

Comment 36

[Ryan VanderMeulen [:RyanVM]]

It's unclear to me if something needs backporting here or not still. Or did bug 1152169 fix this?

Comment 37

[Andrea Marchesini [:baku]]

I think we need this patch and bug 1152169 landed in order to fix this issue.
Should I ask for some b2g approval?

Comment 38

[Ryan VanderMeulen [:RyanVM]]

Bug 1152169 made it to 2.1/2.2 from the looks of it. Please request b2g37 approval on this at least. If B2G RelMan wants it on v2.1S too, the blocking status takes care of the approval side.

Comment 39

[Andrea Marchesini [:baku]]

Comment on attachment 8585445 [details] [diff] [review]
dataStore patch

[Approval Request Comment]
Bug caused by (feature/regressing bug #): DataStore API
User impact if declined: An JS error is shown in the console and a DataStore error is not correctly propagate to the caller.
Testing completed: no test.
Risk to taking this patch (and alternatives if risky): none 
String or UUID changes made by this patch: none

Comment 40

[Josh Cheng [:josh]]

Hi Vance,
Do we still need this patch on 2.1S?

Comment 41

[Vance Chen [:vchen][skype:vance.lucida][[e-mail]]]

Yes, please land this one on 2.1s as well

Thanks

Comment 42

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g34_v2_1s/rev/5bd7e82802e1

Comment 43

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/fb52968e0122

Comment 44

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g37_v2_2r/rev/fb52968e0122