1102972 - [Homescreen] Empty or incomplete homescreen after a full flash

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::Build, defect)

Description

[Julien Wajsberg [:julienw]]

I full flashed my Flame-kk to nightly, and after the FTU I got an empty homescreen.

Here are the error messages I have:

11-21 17:49:27.386  1108  1108 E Homescreen: Content JS ERROR: Bookmark store: Cannot get access to the Store 
11-21 17:49:27.636   206   617 W GeckoConsole: [JavaScript Error: "IndexedDB UnknownErr: ActorsParent.cpp:6088"]
11-21 17:49:28.316   206   617 W GeckoConsole: [JavaScript Error: "IndexedDB UnknownErr: ActorsParent.cpp:6088"]
11-21 17:49:28.406  1108  1108 E Homescreen: Content JS ERROR: Bookmark store: Cannot get access to the Store 
11-21 17:49:28.406  1108  1108 I Homescreen: Content JS LOG: No single variant configuration file found 
11-21 17:49:33.336  1108  1108 W Homescreen: [JavaScript Error: "TypeError: orgEntries[0].detail is undefined" {file: "app://verticalhome.gaiamobile.org/gaia_build_defer_index.js" line: 354}]
11-21 17:49:33.336  1108  1108 W Homescreen: [JavaScript Error: "uncaught exception: [object Object]"]


It worked properly after a full reboot.

Asking for a blocking status because this could happen after a normal OTA.

Comment 1

[Julien Wajsberg [:julienw]]

Actually after a full reboot I don't have all applications. For example I don't have the Dialer nor Contacts.

Comment 2

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Julien, looks like an issue with the single variant customization?  I've seen this when I got my file wrong.

Comment 3

[Julien Wajsberg [:julienw]]

I don't use any customization :/ I simply used the normal "./build.sh && ./flash.sh" process. I use a "userbuild" type build.

Comment 4

[Gregor Wagner [:gwagner]]

Ben, any ideas?

Comment 5

[Ben Turner (not reading bugmail, use the needinfo flag!)]

That IDB message is telling us that a database was invalidated and that it canceled at least one transaction that was in progress. That can either happen due to a process crash or when something tries to clear private data for an app (maybe during app uninstall?).

Comment 6

[Gregor Wagner [:gwagner]]

Julien, have you seen this again?

Comment 7

[Julien Wajsberg [:julienw]]

I think it's ok now. We can close and I'll reopen if I see it again.

Comment 8

[:gerard-majax]

I have seen it recently again.

Comment 9

[:gerard-majax]

I feel it may be a Gaia build system issue.

1. If I |./build.sh && ./flash.sh| on a previously built tree, then the described issue exposed. And after reboot I have some applications missing (Marketplace, for example)

2. If I clean the Gaia tree and then build, |make -C gaia/ clean really-clean && ./build.sh && ./flash.sh|, Homescreen is properly populated after first boot, and I have no missing app.

3. If after this I re-do step 1, then it's in the same state after FTU: no icon on homescreen, some apps missing after rebooting.

Comment 10

[:gerard-majax]

Attachment: 2014-12-19-11-29-30.png

This is a screenshot after step 3 (after reboot) in previous comment. Build was userdebug, so this is a production Gaia. Notice the apps that are missing.

Comment 11

[:gerard-majax]

Changing component per comment 9: as much as I could reproduce, it looks like everytime I |make -C gaia/ clean really-clean|, it fixes the issue.

Comment 12

[:gerard-majax]

Attachment: Log from a broken build

Comment 13

[:gerard-majax]

Attachment: Log from a working build

Comment 14

[Gregor Wagner [:gwagner]]

Lets not block on a gaia build issue. If this is an issue a user can hit during upgrade please renom.

Comment 15

[Alexandre Poirot [:ochameau]]

If you happen to reproduce it again, could you please compute a diff between the broken and working profile?

$ cp -r profile profile-broken
$ make really-clean
$ make
$ diff -r profile-broken profile

(I haven't seen any explicit error in the log)

Comment 16

[Julien Wajsberg [:julienw]]

I just got the issue by flashing the pvtbuild for Hamachi (so it's a shallow-flash, with all data removed beforehand).

That's why I think it's not necessarily a gaia build issue.

Comment 17

[Ricky Chien [:rickychien] (inactive)]

It seems to be an issue in incremental build after introducing rebuild feature in bug 969215. However, I've tested by diff profile in every incremental build. If issue can be reproduced without specified device, it will be a Gaia:Build issue.

You should try it as Alex mentioned on comment 15. Thanks!

Comment 18

[:gerard-majax]

Attachment: gaia-profile.diff

Diff generated between both gaia profile.

Comment 19

[:gerard-majax]

(In reply to Ricky Chien [:rickychien] from comment #17)
> It seems to be an issue in incremental build after introducing rebuild
> feature in bug 969215. However, I've tested by diff profile in every
> incremental build. If issue can be reproduced without specified device, it
> will be a Gaia:Build issue.
> 
> You should try it as Alex mentioned on comment 15. Thanks!

STR:

> $ make -C gaia/ clean really-clean 
> $ ./build.sh
> $ cp -a gaia/profile/ gaia-profile-ok
> $ ./build.sh
> $ cp -a gaia/profile/ gaia-profile-broken
> $ LANG=C diff -r gaia-profile-ok/ gaia-profile-broken/ > gaia-profile.diff

Comment 20

[Ricky Chien [:rickychien] (inactive)]

The installTime / updateTime in webapps.json will be different each time, so we can say two profiles are same.

Comment 21

[:gerard-majax]

(In reply to Ricky Chien [:rickychien] from comment #20)
> The installTime / updateTime in webapps.json will be different each time, so
> we can say two profiles are same.

And nothing makes you wondering why we have some applications that are missing ?

Only in gaia-profile-ok/webapps: bluetooth.gaiamobile.org
Only in gaia-profile-ok/webapps: bookmark.gaiamobile.org
Only in gaia-profile-ok/webapps: callscreen.gaiamobile.org
Only in gaia-profile-ok/webapps: default_theme.gaiamobile.org
Only in gaia-profile-ok/webapps: emergency-call.gaiamobile.org
Only in gaia-profile-ok/webapps: fl.gaiamobile.org
Only in gaia-profile-ok/webapps: fm.gaiamobile.org
Only in gaia-profile-ok/webapps: marketplace.firefox.com
Only in gaia-profile-ok/webapps: network-alerts.gaiamobile.org
Only in gaia-profile-ok/webapps: pdfjs.gaiamobile.org
Only in gaia-profile-ok/webapps: ringtones.gaiamobile.org
Only in gaia-profile-ok/webapps: video.gaiamobile.org

Comment 22

[Ricky Chien [:rickychien] (inactive)]

Which gaia commit version you used? Please update to latest gaia and try again.
I tried your STR but didn't get such missing.

Comment 23

[:gerard-majax]

fa225ad6ee36a8d9f3f3d26340ef85c70c845b58

Comment 24

[Ricky Chien [:rickychien] (inactive)]

Attachment: gaia-profile-ricky.diff

Weird...submit my gaia-profile.diff.

You cound update gonk-misc repo and others to make sure that issue happens in latest version.

Comment 25

[Ricky Chien [:rickychien] (inactive)]

Pass B2G_UPDATER=1 and B2G_UPDATE_CHANNEL=nightly-flame can reproduce it. I'm going to find out the root cause.

Comment 26

[Ricky Chien [:rickychien] (inactive)]

Sorry, I mess up something... There are some failures while running ./build.sh so that cause empty result in profile/webapps/. ni Alex again.

Comment 27

[Alexandre Poirot [:ochameau]]

Attachment: Pull request 27047

It looks like it is somewhat intermitently failing when using ./build.sh.
I don't get why it is intermittent, looks like it should just always fail on non-clobbered builds.

It fails because ./build.sh codepath remove the profile folder, whithout cleaning build_stage, so that apps appear to build uptodate in build stage and are not copied again in profile folder.

Ricky, I cleaned a lot of stuff in this patch, please review carefully.
One big refactoring I'm doing here is to avoid calling getWebapp() everywhere, whereas we already called have a valid webapp object computed!

Thanks Alex and Julien for the detailed report!!

Comment 28

[Ricky Chien [:rickychien] (inactive)]

Comment on attachment 8542202 [details] [review]
Pull request 27047

I saw some uuid apps are built in profile/webapps. (see my diffs results)

Only in HEAD^/webapps: {76dc1451-8855-40bc-ae8b-ade4db9b69a8}
Only in HEAD^/webapps: {996bfc2a-899d-4abe-894e-06858e5267dc}
Only in HEAD/webapps: in_app_pay_test.gaiamobile.org
Only in HEAD/webapps: marketplace.allizom.org.gaiamobile.org
Only in HEAD/webapps: marketplace-dev.allizom.org.gaiamobile.org
Only in HEAD^/webapps: marketplace.firefox.com
Only in HEAD/webapps: marketplace.firefox.com.gaiamobile.org
Only in HEAD/webapps: mochitest.gaiamobile.org
Only in HEAD^/webapps: packaged.marketplace.allizom.org
Only in HEAD^/webapps: packaged.marketplace-dev.allizom.org
Files HEAD/webapps/privacy-panel.gaiamobile.org/application.unzip/resources/about.json and HEAD^/webapps/privacy-panel.gaiamobile.org/application.unzip/resources/about.json differ
Files HEAD/webapps/settings.gaiamobile.org/application.unzip/resources/gaia_commit.txt and HEAD^/webapps/settings.gaiamobile.org/application.unzip/resources/gaia_commit.txt differ
Files HEAD/webapps/webapps.json and HEAD^/webapps/webapps.json differ

I think there is an incorrect logic in rebuild.js line:127.  
put "|| !webapp.profileDirectoryFile.exists()" will copy all of uuid apps into profile/webapps.

Just fix it and test it with your diff-build.sh then set r? again thanks!!

Comment 29

[Ricky Chien [:rickychien] (inactive)]

Although I guess that the profile folder is removed during incremental build, I still cannot reproduce it in my env.

Alexandre Poirot, could you reproduce such failure and make sure your patch doesn't break incremental build with ./build.sh? Thanks!

Comment 30

[Alexandre Poirot [:ochameau]]

Comment on attachment 8542202 [details] [review]
Pull request 27047

Oh... this is such a mess.
We compute the folder name from everywhere,
I'm wondering if we have some race condition on this codepath.
Here is some more work in order to ensure that the folder name used in profile is computed in one unique place, utils-xpc.js!
Also I discovered that we save the uuid mapping file in stage and we never use it...?!?

Comment 31

[Ricky Chien [:rickychien] (inactive)]

Comment on attachment 8542202 [details] [review]
Pull request 27047

It works very well on my mac after trying three times of ./diff_build.sh. And also thanks for your a bit of refactoring :)

Remember to fix nits on Github and make sure it passes on Try. Thanks!

Comment 32

[(Inactive after June) George Duan [:gduan] [:喬智]]

(In reply to Alexandre Poirot [:ochameau] from comment #30)
> Comment on attachment 8542202 [details] [review]
> Pull request 27047
> 
> Oh... this is such a mess.
> We compute the folder name from everywhere,
> I'm wondering if we have some race condition on this codepath.
> Here is some more work in order to ensure that the folder name used in
> profile is computed in one unique place, utils-xpc.js!
> Also I discovered that we save the uuid mapping file in stage and we never
> use it...?!?
See https://bugzilla.mozilla.org/show_bug.cgi?id=1058555#c19

Comment 33

[Ryan VanderMeulen [:RyanVM]]

Master: https://github.com/mozilla-b2g/gaia/commit/af2d1e31f4c4bd0438984794b0139e72ce7d45a2

Comment 34

[Alexandre Poirot [:ochameau]]

Alex, could you confirm that the build system works fine now in your daily usage?

Comment 35

[:gerard-majax]

It looks like it is fixed, I cannot reproduce the issue anymore now.

Comment 36

[Julien Wajsberg [:julienw]]

I still had an issue when flashing latest gecko+gaia for Flame-kk.

Gaia-Rev        82b534791a72ad59c78363df80459e18d35b48f8
Gecko-Rev       https://hg.mozilla.org/mozilla-central/rev/d480b3542cc2
Build-ID        20150108160401


It has also other issues now that I try to reflash it to see if it's only intermittent, I have an issue with the Settings API...

Comment 37

[Julien Wajsberg [:julienw]]

Forget previous comment, I have a change in my shallow_flash script where I don't delete user data, so maybe the phone was in that state before the flash, I don't really know.

Comment 38

[:gerard-majax]

We still have reports that this is broken on v2.2 at least: https://bugzilla.frenchmozilla.org/show_bug.cgi?id=623

Comment 39

[Ricky Chien [:rickychien] (inactive)]

Hi, do you have time to take a look? thanks =)

Comment 40

[Alexandre Poirot [:ochameau]]

(In reply to Ricky Chien [:rickychien] from comment #39)
> Hi, do you have time to take a look? thanks =)

Not really...

Comment 41

[GitHub Autolander Bot]

Attachment: [gaia] RickyChien:rebuild-profile-removed > mozilla-b2g:master

Comment 42

[Ricky Chien [:rickychien] (inactive)]

Comment on attachment 8590057 [details] [review]
[gaia] RickyChien:rebuild-profile-removed > mozilla-b2g:master

Since I cannot reproduce on my machine and according to comment 27, I submitted this patch for fixing rebuild problem if someone deletes profile in somewhere.

Patch is a simple workaround to ensure gecko build.sh doesn't break gaia build. Please take a look and verify it kindly thanks!

Comment 43

[Alexandre Poirot [:ochameau]]

Comment on attachment 8590057 [details] [review]
[gaia] RickyChien:rebuild-profile-removed > mozilla-b2g:master

I'm not used to reproduce this issue, but this modification makes sense.

Comment 44

[Ricky Chien [:rickychien] (inactive)]

Landed in master: https://github.com/mozilla-b2g/gaia/commit/c6fe4557d7e23acad07a0e1f3e543c3bb6cdcf8a

Comment 45

[:gerard-majax]

We are still getting reports for this behavior from time to time ... Is it possible there is still rare cases?

Comment 46

[Julien Wajsberg [:julienw]]

I got it with a master PVT build from last week; after the flash the homescreen was empty, then I rebooted and I missed some icons (and they were not ordered at all), notably Phone and Contacts. They were accessible when searching for them in the rocket bar.

Unfortunately I didn't try again with the same build to know if this was a build problem or a race; and I didn't look at logcat..

Comment 47

[Mathieu Pillard [:mat]]

FWIW I just had this happen a few minutes ago when flashing mozilla-central gaia+gecko using flash_pvt.py. When reflashing the same gaia+gecko, it went away and I got my homescreen icons back.