1048639 - Homescreen does not paint (just see system wallpaper)

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::Homescreen, defect)

Description

[Greg Grisco]

[Blocking Requested - why for this release]:

This is a re-open of bug 1033618 which is still happening.

Comment 1

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

hi Kevin,

We are again hitting this issue after 48 hour stability testing. 
STR will be difficult. 

Observations:
1) |adb shell b2g-info| shows that homescreen process is not running. 
2) its not a memleak.

Could you please provide us a patch which will log more details during homescreen launching and will help us to understand why it is not launched again after killed ?

Comment 2

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

This could be a dupe of bug 1038854.

Comment 3

[cafbot (PoC: ggrisco)]

Observed on: 

Device: msm8610
Gonk Version: AU_LINUX_GECKO_B2G_KK_3.6.01.04.00.000.043
Moz BuildID: 20140721000201
B2G Version: 2.0
Gecko Version: 32.0a2
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=8cb1a949f2e9650bb2c5598e78a6f24a58bbaf97
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=5f27d3ee3ccf01ac91a3efacb5e3e22ea62fd73c

Comment 4

[Kevin Grandon :kgrandon]

Thanks for the info Tapas. It sounds like we should wait for a patch in bug 1038854 and check with that patch.

Comment 5

[Preeti Raghunath(:Preeti)]

Blocks a blocker.

Comment 6

[Wesley Huang [:wesley_huang] (EPM) (NI me)]

DOes this fall under your plate?

Comment 7

[Ken Chang[:kenkai]]

According to comment 2 and 4, the patch in bug 1038854 was just landed on 8/7. We need to retest with that patch. 
Hi Greg, could you please help?

Comment 8

[Thinker Li [:sinker]]

I have no comment before getting test result of the bug 1038854.  Please ni me, again, if bug 1038854 does not fix the issue.

Comment 9

[Inder]

We have been running tests with the fixes in bug 1038854. We will update if it shows up again.

Comment 10

[Greg Grisco]

(In reply to Inder from comment #9)
> We have been running tests with the fixes in bug 1038854. We will update if
> it shows up again.

The fix for bug 1038854 has already landed and has been built into AUs since AU51.

Comment 11

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

Attachment: gecko memory report when this happens

Comment 12

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

We've ruled out anything Nuwa or process launching related.  The homescreen is definitely running.  Still investigating why it doesn't paint.

Comment 13

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

Attachment: layerdump.tar.bz2

(In reply to Kyle Huey [:khuey] ([e-mail]) from comment #12)
> We've ruled out anything Nuwa or process launching related.  The homescreen
> is definitely running.  Still investigating why it doesn't paint.


As discussed, here is the layer dump for both affected and unaffected device

Comment 14

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

Attachment: about-memory- gecko memory report from unaffected device

Hi Kyle,

As discussed , here is gecko memory report from unaffected device.

Comment 15

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

The homescreen is missing most of its DOM nodes, so this is a Gaia bug until proven otherwise.

Comment 16

[Kevin Grandon :kgrandon]

Hi Greg, Kyle,

Can you provide any of:

1 - Screenshot the problem.
2 - Video of getting into this state.
3 - Dump of the dom nodes of the homescreen (according to comment 15).

Comment 17

[Kevin Grandon :kgrandon]

4 - Also a logcat of the homescreen would be useful, didn't see one here.

Comment 18

[bhavana bajaj [:bajaj]]

Passing this to Kevin as he's looking into this issue.

Comment 19

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

> 3 - Dump of the dom nodes of the homescreen (according to comment 15).

The about:memory reports have that (via the CC dumps).  I can show you how to read it tomorrow.

Tapas has #1 and #4.

Comment 20

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

(In reply to Kyle Huey [:khuey] ([e-mail]) from comment #19)
> > 3 - Dump of the dom nodes of the homescreen (according to comment 15).
> 
> The about:memory reports have that (via the CC dumps).  I can show you how
> to read it tomorrow.
> 
> Tapas has #1 and #4.

Here is the link to download #1 and #4 from device which we debugged together for this issue.

https://drive.google.com/file/d/0B1cSMS8_GuAETF93SlpGVkM0OHc/edit?usp=sharing

Comment 21

[Kevin Grandon :kgrandon]

Attachment: Screenshot of issue

Comment 22

[Kevin Grandon :kgrandon]

I see some lines in the logcat regarding dragdrop. I don't think this actually impacts this issue, but removing the line will help us debug it, and maybe even fix it. Bug 1051061 will fix this.

Comment 23

[Kevin Grandon :kgrandon]

Tapas - we are going to try to get the patch from bug 1051061 reviewed and landed soon, but in the meantime would you want to try to test with this patch?

Also - the line numbers in the log do not match up. Could you provide us the homescreen zip file for us to debug this. The file should be located at: /system/b2g/webapps/verticalhome.gaiamobile.org/application.zip

Comment 24

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

Attachment: verticalhome.gaiamobile.org.tar.bz2

(In reply to Kevin Grandon :kgrandon from comment #23)
> Tapas - we are going to try to get the patch from bug 1051061 reviewed and
> landed soon, but in the meantime would you want to try to test with this
> patch?
> 
Thanks . I asked out internal test team to test it.

> Also - the line numbers in the log do not match up. 

As we discussed in #gaia IRC, here is the exact gaia/gecko SHA1s which we picked up for logs in Comment 20

https://www.codeaurora.org/cgit/quic/lf/b2g/mozilla/gaia/commit/?h=mozilla/v2.0&id=8cc28fd31905a0ea2b2e15d13e80a0eab2feb1ba
https://www.codeaurora.org/cgit/quic/lf/b2g/mozilla/gecko/commit/?h=mozilla/v2.0&id=f7bd772b1e42774708a4ede13b149a1706a59b25

Please note that we also cherry-picked following patches on top of above gaia/gecko:

gaia patch: bug 1050751 attachment 8470312 [details] , bug 1050423
gecko patch: bug 1008791, bug 1030112, bug 1050423, bug 1028532, bug 1047149, bug 1044322, bug 1047645, bug 1049806 attachment 8469653 [details] [diff] [review]


> Could you provide us the
> homescreen zip file for us to debug this. 
> The file should be located at:
> /system/b2g/webapps/verticalhome.gaiamobile.org/application.zip

Yes. I attached it here.

Comment 25

[Kevin Grandon :kgrandon]

Thanks Tapas. These log lines do match up and I will see if there is additional information I can pull out of the logs.

Are you by chance able to provide a remote debug session to the device in this state with the app manager enabled for certified applications?

These are the device preferences necessary to access the app manager: https://github.com/mozilla-b2g/gaia/blob/master/build/preferences.js#L256

I am essentially looking for the HTML content of the body, as well as the content within the javascript object located at: app.grid.getItems().

Comment 26

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

(In reply to Kevin Grandon :kgrandon from comment #25)
> Thanks Tapas. These log lines do match up and I will see if there is
> additional information I can pull out of the logs.
> 
> Are you by chance able to provide a remote debug session to the device in
> this state with the app manager enabled for certified applications?
> 
> These are the device preferences necessary to access the app manager:
> https://github.com/mozilla-b2g/gaia/blob/master/build/preferences.js#L256
> 
> I am essentially looking for the HTML content of the body, as well as the
> content within the javascript object located at: app.grid.getItems().

Sorry, device is rebooted but i can arrange another remote gdb session with app manager enabled.
Thanks for the ideas.

Comment 27

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

(In reply to Kevin Grandon :kgrandon from comment #25)
> Thanks Tapas. These log lines do match up and I will see if there is
> additional information I can pull out of the logs.
> 
> Are you by chance able to provide a remote debug session to the device in
> this state with the app manager enabled for certified applications?
> 
> These are the device preferences necessary to access the app manager:
> https://github.com/mozilla-b2g/gaia/blob/master/build/preferences.js#L256
> 
> I am essentially looking for the HTML content of the body, as well as the
> content within the javascript object located at: app.grid.getItems().

Please note that it is always fast if we can do analysis using logcat logs. So please give us some patch for this if possible.

Comment 28

[Kevin Grandon :kgrandon]

Tapas - here is a patch which will add a bunch of output to the vertical homescreen. It's not nearly as large as debugging IPC logging, but it should still provide significant logging. Please try this patch out if possible and send me the updated logs if you have any.

https://github.com/mozilla-b2g/gaia/pull/22805

Comment 29

[Inder]

We got device ready with gdb access with this issue reproduced. Tapas is going to get in touch with Kevin for debug.

Comment 30

[Inder]

(In reply to Inder from comment #29)
> We got device ready with gdb access with this issue reproduced. Tapas is
> going to get in touch with Kevin for debug.
Sorry, nevermind -- wrong bug :(. We are still not able to reproduce this issue with additional logs.

Comment 31

[Yeojin Chung [:YeojinC]]

New test case needs to be added. There is no existing test case.

Comment 32

[KTucker [:KTucker][Inactive 3/4/2016]]

Test case added in moztrap:

https://moztrap.mozilla.org/manage/case/14321/

Comment 33

[Inder]

Not reproducing anymore, closing. Will reopen if it comes back again.

Comment 34

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

Attachment: tapas_cr_686636.zip

it is reproduced again and It has following logs: 

1) Screenshot of affected device after pressing home key 
2) memory report from affected device after pressing home key 
3) kernel and logcat logs

We also hav gdb acess to affected device . Please make a NI on me if you want to debug it remotely using gdb or not

Comment 35

[bhavana bajaj [:bajaj]]

NI :kgrandon here to get started on the investigation.

Comment 36

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

Attachment: about-memory-1.tar.bz2

Kyle,

I attached another memory report from same device and it contains gc-cc logs for visualizing dom tree nodes

I remember that gecko memory report and dumping layer tree (using gdb trick) didn't help us here last time and you concluded that homescreen does not have layers for icon

Please let me know how I can help you to debug further

Comment 37

[Kevin Grandon :kgrandon]

In the latest screenshots there is zero chrome from the homescreen as all. Tracking as a system issue for now.

Comment 38

[Kevin Grandon :kgrandon]

Attachment: Screenshot of issue

Updating the screenshot based on latest findings. I don't think it's the same as the originally reported issues, so clearing out legacy attachments as well.

Comment 39

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I agree with Kevin that this appears to be different since the search box is not painting.  The layer tree dump might prove useful after all.

Comment 40

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Although the DOM does look pretty much the same in both the system app and the homescreen compared to last time.

Comment 41

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

Attachment: cr_686636_layertree_dump_after_homekeypress.tar.bz2

(In reply to Kyle Huey [:khuey] ([e-mail]) from comment #39)
> I agree with Kevin that this appears to be different since the search box is
> not painting.  The layer tree dump might prove useful after all.

here is the layer tree dump in logcat after pressing homekey on affected device.

Comment 42

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

The layer tree dump indicates that we're scrolled down 56 pixels on the home screen, which is probably why we don't see the search box.  Other than that this appears to be identical to the last time we looked at this.

Comment 43

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

just as an update: We have both marionette and gdb access to affected device. Please let us know if you want to try something on device for debugging.

Comment 44

[bhavana bajaj [:bajaj]]

(In reply to Tapas Kumar Kundu from comment #43)
> just as an update: We have both marionette and gdb access to affected
> device. Please let us know if you want to try something on device for
> debugging.

I did check with kyle sometime back and he had sufficient information to investigate and we discussed we'll do additional follow-up once he has some ideas from his debugging.

Comment 45

[Kevin Grandon :kgrandon]

Attachment: Pull request - add additional logging when entryPoint not found

We noticed this in the logs which seems weird: 08-20 18:31:27.137 32331 32331 E GeckoConsole: [JavaScript Error: "TypeError: manifest.entry_points is undefined" {file: "app://verticalhome.gaiamobile.org/gaia_build_defer_index.js" line: 417}]

Tapas - this patch will add a small safeguard and adds some logging when we can't find an entrypoint. Can you apply this patch and try to get into this scenario again? Thanks!

Comment 46

[Kevin Grandon :kgrandon]

I've added a second commit to the pull request with more logging.

Comment 47

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

(In reply to Kevin Grandon :kgrandon from comment #46)
> I've added a second commit to the pull request with more logging.

I am asking our internal team to reproduce with this log. I will update here asap

Comment 48

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

Attachment: cr_686636.zip

Issue is reproduced on device around "08-21 13:55:00"

Comment 49

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Kevin has this under control for now.

Comment 50

[Kevin Grandon :kgrandon]

This is a good hint: 08-21 13:55:07.076  9786  9786 E GeckoConsole: [JavaScript Error: "TypeError: cyclic object value" {file: "app://verticalhome.gaiamobile.org/gaia_build_defer_index.js" line: 330}]

Going to see if I can reproduce this on a phone, otherwise I will provide another patch which should flush this out a bit more.

Comment 51

[Kevin Grandon :kgrandon]

Tapas - I am investigating to see if I can reproduce the situation locally. In the meantime, I've refreshed the attached patch to change the logging. Would you mind applying it again and reporting the logs? Thank you.

https://github.com/mozilla-b2g/gaia/pull/23115

Comment 52

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

(In reply to Kevin Grandon :kgrandon from comment #51)
> Tapas - I am investigating to see if I can reproduce the situation locally.
> In the meantime, I've refreshed the attached patch to change the logging.
> Would you mind applying it again and reporting the logs? Thank you.
> 
> https://github.com/mozilla-b2g/gaia/pull/23115

Thanks for helping . I will give it a try.

Comment 53

[Candice Serran (:cserran)]

Is there any progress on here? There hasn't been any update for five days.

Thanks

Comment 54

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

We thought that this issue is fixed as it was not coming stability runs.
But we have seen this issue again in latest stability run today. I will update here as soon as I get logs from our test team .

Comment 55

[bhavana bajaj [:bajaj]]

(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #54)
> We thought that this issue is fixed as it was not coming stability runs.
> But we have seen this issue again in latest stability run today. I will
> update here as soon as I get logs from our test team .

Inder/Tapas, any logs on this yet ?

Comment 56

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

Attachment: mozilla_logs.tar.bz2

(In reply to bhavana bajaj [:bajaj] from comment #55)
> (In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from
> comment #54)
> > We thought that this issue is fixed as it was not coming stability runs.
> > But we have seen this issue again in latest stability run today. I will
> > update here as soon as I get logs from our test team .
> 
> Inder/Tapas, any logs on this yet ?

We tried to reproduce it in test build with logging patch from Comment 52 but that test build didn't reproduce this issue. 

But we are seeing same issue in another test build.

Comment 57

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

Hi Kevin,

Could you please take a look in logs of Comment 56 ?

Comment 58

[Kevin Grandon :kgrandon]

Attachment: Github pull request

After inspecting the patch, it appears that the issue might not reproduce after applying it because we are catching the exception. It seems absolutely impossible to get into this case, so I'm not sure how it's happening, unless you have some test app that is doing weird things, or you are installing some malformed apps. If you are not seeing additional weirdness with the homescreen it seems that we can land this guard and it may fix it. Though it seems this may be an issue that the user will never see.

Cristian - Could you review this patch? Thanks!

Comment 59

[Cristian Rodriguez (:crdlc)]

Comment on attachment 8480998 [details] [review]
Github pull request

LGTM, thanks

Comment 60

[Kevin Grandon :kgrandon]

In master: https://github.com/mozilla-b2g/gaia/commit/4419091ae760328a52606920e335331a16fcb448
v2.0: https://github.com/mozilla-b2g/gaia/commit/aa7bcde4a751aaf65dd8c57bdfa6cbd75eabaa77

Tapas - please let us know if you run into this issue again or see something similar. Thanks!

Comment 61

[Inder]

Tapas -- please monitor this fix in the stability runs. Keeping a ni on you.

Comment 62

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

(In reply to Inder from comment #61)
> Tapas -- please monitor this fix in the stability runs. Keeping a ni on you.

I will update here if this really fixes this bug :)

Comment 63

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

We are not seeing this issue for a long time. It seems to be fixed !