924912 - Provide a way to determine that B2G is ready to use

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Performance, defect, P2)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

In automation we stop and start the b2g process between tests. At the moment we wait for the mozbrowserloadend to fire for either the ftu or homescreen app, however it appears that sometimes this may be too early for us to start interacting with the device. As a result, when we kill the FTU app we sometimes leave the splash screen running, and we also sometimes see asynchronous JavaScript calls never returning.

Ideally, we would have a JavaScript event fired to indicate that B2G has finished launching and is ready for use. If we had per-app events then we could wait to receive this from either the ftu or homescreen app.

Our current wait is here: https://github.com/mozilla-b2g/gaia/blob/7b80daa4f61cd04dd8b3241024a1d42e37666742/tests/python/gaia-ui-tests/gaiatest/gaia_test.py#L408

If I've misunderstood any part of the boot sequence please feel free to suggest improvements.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

One potential solution here is to wait for either the homescreen to be visible, or the FTU app to be visible /and/ the splash screen to be removed. This will need testing.

Comment 2

[[:fabrice] Fabrice Desré]

What do you mean by having b2g "ready" exactly? If the homescreen or the ftu is launched, everything should be usable, so I'm not sure what's going wrong for you.

Comment 3

[Bob Silverberg [:bsilverberg]]

It seems that what is happening is this:

1. The FTU app launches, with the splash screen visible.
2. The fact that the FTU app has finished launching signals to our script to continue.
3. Our script kills the FTU app, but the splash screen remains visible - indefinitely.

Perhaps #3 is a bug which needs to be investigated?

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment 2 explains one side-effect (the splash screen persisting) but I have also seen issues with asynchronous scripts timing out, which I suspect is another symptom of this. When investigating this last week in 30 runs I had 4 timeouts, after adding a 5 second sleep I had none.

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

I hope it's relevant, but I just saw a b2g.hamachi.mozilla-central.master.perf.fps (#193) [1] job fail due to "adb: command not found," but the device is clearly on the FTU screen.

Still, it's not found via adb:

"Last login: Wed Oct  9 08:57:42 2013 from host-2-135.mv.mozilla.com
adwebqa@b2g-3:~$ adb devices
List of devices attached

"

:-(

[1] http://qa-selenium.mv.mozilla.com:8080/view/B2G%20Hamachi/job/b2g.hamachi.mozilla-central.master.perf.fps/193/console

Comment 6

[[:fabrice] Fabrice Desré]

Hm, could be because of the way we enable/disable adb. Did the phone screen turn off at some point?

Comment 7

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Fabrice Desré [:fabrice] from comment #6)
> Hm, could be because of the way we enable/disable adb. Did the phone screen
> turn off at some point?

Fabrice, Hub (because I know you did a patch to fix our broken engineering build before, Hub, thx!): yes, I've seen the device go "dormant" when walking by a build in our automation rig -- sometimes, it seems related to the battery-charge level, which should only be low when we take a device offline for an extended period of time for troubleshooting).

What I see, most often, are failure after failure in automation, with the device in question succesfully/happily booted into the FTU screen, and -- most times -- present in ADB.  I still have seen ADB issues aside, though, and not sure; I think we've seen problems with some Ubuntu installs, right :retornam?

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

To me this sounds like a different issue. Stephen: We've landed a temporary workaround (a 5 second sleep) which should resolve this issue, so if you still see ADB issues I would consider this separate.

Comment 9

[raymond [:retornam] (inactive)]

(In reply to Stephen Donner [:stephend] from comment #7)
> (In reply to Fabrice Desré [:fabrice] from comment #6)
> > Hm, could be because of the way we enable/disable adb. Did the phone screen
> > turn off at some point?
> 
> Fabrice, Hub (because I know you did a patch to fix our broken engineering
> build before, Hub, thx!): yes, I've seen the device go "dormant" when
> walking by a build in our automation rig -- sometimes, it seems related to
> the battery-charge level, which should only be low when we take a device
> offline for an extended period of time for troubleshooting).
> 
> What I see, most often, are failure after failure in automation, with the
> device in question succesfully/happily booted into the FTU screen, and --
> most times -- present in ADB.  I still have seen ADB issues aside, though,
> and not sure; I think we've seen problems with some Ubuntu installs, right
> :retornam?

I noticed that the splash screen kept running for an increased amount of time after flashing / running some tests sometimes. I did not notice any adb issues.

Comment 10

[[:fabrice] Fabrice Desré]

Clearing the ni since I don't have anything to add there.

Comment 11

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Bob, I believe you did some testing with waiting for the FTU splash to be not visible and for the homescreen to be visible. Remind me, what was the outcome of this?

Comment 12

[Bob Silverberg [:bsilverberg]]

Dave, yes I did that awhile ago. With code that waited either for the home page to be visible OR the FTU app to be visible and the splash screen not visible, we still saw the issue. The test would run while the FTU splash screen was still visible and then the splash screen would never go away until b2g was started again for the next test.

Comment 13

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Bob Silverberg [:bsilverberg] from comment #12)
> Dave, yes I did that awhile ago. With code that waited either for the home
> page to be visible OR the FTU app to be visible and the splash screen not
> visible, we still saw the issue. The test would run while the FTU splash
> screen was still visible and then the splash screen would never go away
> until b2g was started again for the next test.

That sounds like a Marionette bug then - if we return from waiting for the splash screen to not be visible while it's still visible. Could you confirm this and raise a new bug? I thought you had said that this worked, but that you were unable to get it to fail by negating the waits. I may be thinking of something else.

Comment 14

[Bob Silverberg [:bsilverberg]]

Unfortunately on today's build, on both Inari and Hamachi, I am unable to replicate the problem after I remove the existing sleep. I know it was somewhat intermittent and probably dependent on hardware. I will try a few more times, but if I cannot get it to happen again then I'm not sure what else to do.

Comment 15

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Thanks Bob, I'd like to see if we can revisit changing the current wait for event to waiting for the homescreen or FTU to be running (and splash screen hidden) because it's better than an arbitrary sleep, which is what we have at the moment. If someone wants to take this, I'd be happy to mentor.

What I think we need:
* wait for displayed app to be FTU or Homescreen after starting the B2G process
* if it's FTU wait for the splash screen to be hidden

Comment 16

[Stephen Donner [:stephend] Not actively reading bugmail]

So, what I discovered from my testing today is that adb is /disabled/ when we're /at least/ on the FTU screen -- and probably/possibly (:hub, feel free to chime in!) on the lockscreen, too.

I saw this with b2g-11 today on master/mozilla-central build (and possible 1.2?); the device was booted into the FTU screen when it got stuck, "[adb] waiting-for-device": http://qa-selenium.mv.mozilla.com:8080/job/b2g.hamachi.mozilla-central.unittests.wifi/725/console.  Then, it was perfectly fine the next run; no changes in-between: http://qa-selenium.mv.mozilla.com:8080/job/b2g.hamachi.mozilla-central.unittests.wifi/726/

When I ssh'd into the box, and |adb devices|, nothing came up; when I took it off and attached it, I hardware-reset the ROM, I was able to reflash it.  Before I did that, I couldn't find any actual errors in all of the |adb logcat| output.

I'll spin off the "adb is off" on the FTU screen bug, and hope others can corroborate.

Comment 17

[Dave Hylands [:dhylands]]

The only time that adb should be off on the lockscreen is on VARIANT=user builds when remote debugging is also disabled.

For builds with marionette enabled, adb should always be on, except when you're running  with a production boot.img, in which adb will be off until b2g starts.

Comment 18

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Dave Hylands [:dhylands] from comment #17)
> The only time that adb should be off on the lockscreen is on VARIANT=user
> builds when remote debugging is also disabled.
> 
> For builds with marionette enabled, adb should always be on, except when
> you're running  with a production boot.img, in which adb will be off until
> b2g starts.

Here's what we most-recently tested with Hamachi: https://pastebin.mozilla.org/3402269 -- is there anything in there that would cause what we're seeing?  We're using a repack build and then flashing with that script, atop.

Comment 19

[Hubert Figuiere [:hub]]

It should be noted that the JavaScript marionette client has a module to watch FirefoxOS boot.

https://github.com/mozilla-b2g/marionette-apps/blob/master/lib/bootwatcher.js

You might want to try this (porting it to Python /me thinks)

Comment 20

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Hubert Figuiere [:hub] from comment #19)
> It should be noted that the JavaScript marionette client has a module to
> watch FirefoxOS boot.
> 
> https://github.com/mozilla-b2g/marionette-apps/blob/master/lib/bootwatcher.js
> 
> You might want to try this (porting it to Python /me thinks)

I did look into this before, but implementing it in Python didn't help in this case.

(In reply to Stephen Donner [:stephend] from comment #18)
> (In reply to Dave Hylands [:dhylands] from comment #17)
> Here's what we most-recently tested with Hamachi:
> https://pastebin.mozilla.org/3402269 -- is there anything in there that
> would cause what we're seeing?  We're using a repack build and then flashing
> with that script, atop.

I really would like to keep this bug for knowing when a successful boot is completed, rather than troubleshooting issues that cause us not to successfully boot.

Comment 21

[Ben Kelly [:bkelly, not reviewing]]

Some historical context:

We would like to know when the pre-allocated process is completely ready to launch the next app.  Right now b2gperf uses a large delay between tests because it cannot know when the pre-allocated process is launched.

I believe this is important for our automation testing in order to reduce these delays and therefore increase our overall test throughput.

Comment 22

[Jonathan Griffin (:jgriffin)]

Is there a way to know when the pre-allocated process is completely ready?

Comment 23

[Ben Kelly [:bkelly, not reviewing]]

(In reply to Jonathan Griffin (:jgriffin) from comment #22)
> Is there a way to know when the pre-allocated process is completely ready?

I think that's what this bug is for. :-)  I was just commenting since we didn't see that already noted here.

Comment 24

[[:fabrice] Fabrice Desré]

The end of PreloadSlowThings() should be good for this purpose:
https://mxr.mozilla.org/mozilla-central/source/dom/ipc/TabChild.cpp#213

Comment 25

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I'm not familiar with this Fabrice, but I'd be happy to try it. Could you possibly provide a code snippet that would allow us to wait for this?

Comment 26

[[:fabrice] Fabrice Desré]

Where do you need to be notified? In the parent process?

Comment 27

[Dave Hunt [:davehunt] [he/him] ⌚BST]

After speaking to Fabrice on IRC we discussed creating a patch for Gecko to provide a notification that B2G is ready to use. Once this is ready we should be able to reliably wait for this using an execute_async_script call. This would need a lot of testing, and potentially opens us up to any JavaScript exceptions thrown during the async script call aborting it early. Fabrice is going to work on the Gecko patch.