1139987 - Flame: when running marionette test, some icons on the top header is missing

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[No-Jun Park [:njpark]]

Attachment: missingheader.log

Note: this is not a marionette bug, because it does not occur with older builds from March 1st.  Need to be triaged to correct component.

STR:
run python marionette on-device test with the latest marionette (0.9).  IN my case, ran functional/email/test_setup_and_send_imap_email.py

WHen the phone restarts to run the test, observe the top portion of the the phone. Clock/wifi/mobile icons are missing.

This only happens when the phone reboots to run gaiatest.  IF the phone is rebooted manually, this does not occur.

Version Info:
Gaia-Rev        eff3321ab4e65da3f906688ebb55ddf1e93d9452
Gecko-Rev       https://hg.mozilla.org/mozilla-central/rev/56492f7244a9
Build-ID        20150305010212
Version         39.0a1
Device-Name     flame
FW-Release      4.4.2
FW-Incremental  eng.cltbld.20150305.043221
FW-Date         Thu Mar  5 04:32:30 EST 2015
Bootloader      L1TC000118D0

Comment 1

[No-Jun Park [:njpark]]

logcat up to the point where gaiatest exits FTE to enter homescreen is attached

Comment 2

[No-Jun Park [:njpark]]

in 50% of the time, the email app also crashes:
https://crash-stats.mozilla.com/report/index/c046cf8a-de73-4887-979f-5590f2150305

Comment 3

[No-Jun Park [:njpark]]

(In reply to No-Jun Park [:njpark] from comment #2)
> in 50% of the time, the email app also crashes:
> https://crash-stats.mozilla.com/report/index/c046cf8a-de73-4887-979f-
> 5590f2150305

It looks like when the test takes more than a few minutes, it is more likely for the phone to crash with the failure in Comment 2

Comment 4

[No-Jun Park [:njpark]]

Comment 2 and Comment 3 discussions should be at Bug 1137653

Comment 5

[No-Jun Park [:njpark]]

Update: this seem to be caused by skipping FTU. when test_ftu_skip_tour.py is executed, the icons all appeared normally.

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Could you determine a regression range? This will help to narrow down the commits that may have caused this and help us to find out who to ask for help. A workaround might be to complete the FTU at the start of the test, however this will add a delay and fragility.

Comment 7

[No-Jun Park [:njpark]]

To QAnalysts, could you work on finding the regression range on this, if you have access to the python marionette on device setup?  If not, please let me know.

Comment 8

[No-Jun Park [:njpark]]

Bug 1141654 started to fail since March 10th, so I guess we can bisect starting from last week's build?

Comment 9

[No-Jun Park [:njpark]]

FYI, it looks like this is when the first failure appeared in jenkins. hope it narrows down the bisection range.

http://jenkins1.qa.scl3.mozilla.com/view/Mozilla%20Lab/job/flame-kk-319.mozilla-central.tinderbox.ui.functional.non-smoke.1/257/testReport/%28root%29/test_settings_change_time_format_py%20TestSettingsChangeTimeFormat_test_settings_change_time_format/test_settings_change_time_format_py_TestSettingsChangeTimeFormat_test_settings_change_time_format/

Comment 10

[Dave Hunt [:davehunt] [he/him] ⌚BST]

That gives us the following pushlog:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=44bcd21e59fe&tochange=c5b90c003be8

From that pushlog, bug 1132418 jumps out at me.

Let's see if we can narrow that down on b2g-inbound.

Comment 11

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I've just triggered b2g-inbound builds before/after bug 1132418 landed:
Before: http://jenkins1.qa.scl3.mozilla.com/view/Bitbar/job/flame-kk.ui.adhoc.bitbar/16/
After: http://jenkins1.qa.scl3.mozilla.com/view/Bitbar/job/flame-kk.ui.adhoc.bitbar/17/

Comment 12

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Doesn't look like that's the cause, but that does give us a datapoint in that the failure existed in 5a94300d91f7 on b2g-inbound.

Comment 13

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Another datapoint: The test_settings_change_time_format test fails in d4b2a0b65ea3 but because of bug 1136156 - the status bar looks fine.

Comment 14

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I've narrowed this down to http://hg.mozilla.org/integration/b2g-inbound/pushloghtml?fromchange=97c49ccbdfbf&tochange=5a94300d91f7 so far.

Comment 15

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Regression range narrowed to: http://hg.mozilla.org/integration/b2g-inbound/pushloghtml?fromchange=97c49ccbdfbf&tochange=11a4ffbd9ee9

This points to:
Bug 1098168 - [Statusbar] Instantiatable icons per modules

Alive: Any thoughts on why this change would cause the status bar to be missing items when we kill the FTU application?

Comment 16

[(inactive after 6/18) Alive Kuo [:alive]]

Could you elaborate more how do you kill FTU?
I think we don't expect FTU will be terminated anyway.

Comment 17

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We call appWindowManager.kill as shown here: https://github.com/mozilla-b2g/gaia/blob/ac82d6c5ce759c2fa47ca3cc8c261b67f4b2b49b/tests/atoms/gaia_apps.js#L229

We need to have a way to kill the FTU or prevent it from being present on reset/restart in engineering builds. Completing the FTU for every functional test is fragile and time intensive.

Comment 18

[(inactive after 6/18) Alive Kuo [:alive]]

(In reply to Dave Hunt (:davehunt) from comment #17)
> We call appWindowManager.kill as shown here:
> https://github.com/mozilla-b2g/gaia/blob/
> ac82d6c5ce759c2fa47ca3cc8c261b67f4b2b49b/tests/atoms/gaia_apps.js#L229
> 
> We need to have a way to kill the FTU or prevent it from being present on
> reset/restart in engineering builds. Completing the FTU for every functional
> test is fragile and time intensive.

Hmm...so why not specify NOFTU=1 when making the build to test, or set FTU.manifestURL to be null?

Comment 19

[(inactive after 6/18) Alive Kuo [:alive]]

(In reply to Dave Hunt (:davehunt) from comment #17)
> We call appWindowManager.kill as shown here:
> https://github.com/mozilla-b2g/gaia/blob/
> ac82d6c5ce759c2fa47ca3cc8c261b67f4b2b49b/tests/atoms/gaia_apps.js#L229
> 
> We need to have a way to kill the FTU or prevent it from being present on
> reset/restart in engineering builds. Completing the FTU for every functional
> test is fragile and time intensive.

marionette js(In reply to Alive Kuo [:alive][NEEDINFO!] from comment #18)
> (In reply to Dave Hunt (:davehunt) from comment #17)
> > We call appWindowManager.kill as shown here:
> > https://github.com/mozilla-b2g/gaia/blob/
> > ac82d6c5ce759c2fa47ca3cc8c261b67f4b2b49b/tests/atoms/gaia_apps.js#L229
> > 
> > We need to have a way to kill the FTU or prevent it from being present on
> > reset/restart in engineering builds. Completing the FTU for every functional
> > test is fragile and time intensive.
> 
> Hmm...so why not specify NOFTU=1 when making the build to test, or set
> FTU.manifestURL to be null?

marionette-js has a functionality to specify settings in test before running each test. I am not sure if marionette-python has this or not.

BTW, killing FTU is not a normal user behavior to me. We are waiting FTU to close/be skipped normally to show the icons, but there's no spec about what should happen if it's terminated abnormally. I even think we should re-open FTU immediately if it's killed while accepting a phone call when FTU is running, and the OOM killer kills the background FTU.

Comment 20

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Alive Kuo [:alive][NEEDINFO!] from comment #18)
> Hmm...so why not specify NOFTU=1 when making the build to test, or set
> FTU.manifestURL to be null?

We use the engineering builds provided by release engineering - we do not build them ourselves. Would setting the FTU.manifestURL to null still allow us to test the FTU? Can we simulate closing/skipping 'normally' without completing all FTU steps via interactions?

Comment 21

[(inactive after 6/18) Alive Kuo [:alive]]

(In reply to Alive Kuo [:alive][NEEDINFO!] from comment #19)
> BTW, killing FTU is not a normal user behavior to me. We are waiting FTU to
> close/be skipped normally to show the icons, but there's no spec about what
> should happen if it's terminated abnormally. I even think we should re-open
> FTU immediately if it's killed while accepting a phone call when FTU is
> running, and the OOM killer kills the background FTU.

That is to say, I don't think we should kill FTU actively to achieve your requirement.
If you are not able to change FTU settings while running individual test, let's figure out a better way to skip FTU programmatically.

Comment 22

[(inactive after 6/18) Alive Kuo [:alive]]

(In reply to Dave Hunt (:davehunt) from comment #20)
> (In reply to Alive Kuo [:alive][NEEDINFO!] from comment #18)
> > Hmm...so why not specify NOFTU=1 when making the build to test, or set
> > FTU.manifestURL to be null?
> 
> We use the engineering builds provided by release engineering - we do not
> build them ourselves. Would setting the FTU.manifestURL to null still allow
> us to test the FTU?

The key point is the ability to change the settings by the need of individual tests:
To test a normal app, set it to null;
To test FTU, leave it to the original value.

From what you said I guess you guys can't change the settings. Or you cant change the settings but cannot apply the change to single test?

> Can we simulate closing/skipping 'normally' without
> completing all FTU steps via interactions?

We could do something in system app to receive the request to skip FTU programmatically as my last comment.
For example: |Service.request('skipftu');|

Comment 23

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Alive Kuo [:alive][NEEDINFO!] from comment #22)
> (In reply to Dave Hunt (:davehunt) from comment #20)
> > (In reply to Alive Kuo [:alive][NEEDINFO!] from comment #18)
> > > Hmm...so why not specify NOFTU=1 when making the build to test, or set
> > > FTU.manifestURL to be null?
> > 
> > We use the engineering builds provided by release engineering - we do not
> > build them ourselves. Would setting the FTU.manifestURL to null still allow
> > us to test the FTU?
> 
> The key point is the ability to change the settings by the need of
> individual tests:
> To test a normal app, set it to null;
> To test FTU, leave it to the original value.
> 
> From what you said I guess you guys can't change the settings. Or you cant
> change the settings but cannot apply the change to single test?

We can change the settings, however I'm not sure we'd be able to change this one early enough to influence booting into the FTU. We reset as much as possible between tests. Is there something we can change in a file on the device while B2G is stopped to affect whether FTU is launched?

> > Can we simulate closing/skipping 'normally' without
> > completing all FTU steps via interactions?
> 
> We could do something in system app to receive the request to skip FTU
> programmatically as my last comment.
> For example: |Service.request('skipftu');|

That could work for us, assuming that we could call this even if the FTU is already running?

Comment 24

[(inactive after 6/18) Alive Kuo [:alive]]

(In reply to Dave Hunt (:davehunt) from comment #23)
> (In reply to Alive Kuo [:alive][NEEDINFO!] from comment #22)
> > (In reply to Dave Hunt (:davehunt) from comment #20)
> > > (In reply to Alive Kuo [:alive][NEEDINFO!] from comment #18)
> > > > Hmm...so why not specify NOFTU=1 when making the build to test, or set
> > > > FTU.manifestURL to be null?
> > > 
> > > We use the engineering builds provided by release engineering - we do not
> > > build them ourselves. Would setting the FTU.manifestURL to null still allow
> > > us to test the FTU?
> > 
> > The key point is the ability to change the settings by the need of
> > individual tests:
> > To test a normal app, set it to null;
> > To test FTU, leave it to the original value.
> > 
> > From what you said I guess you guys can't change the settings. Or you cant
> > change the settings but cannot apply the change to single test?
> 
> We can change the settings, however I'm not sure we'd be able to change this
> one early enough to influence booting into the FTU. We reset as much as

What do you mean by early enough and how do you reset?

> possible between tests. Is there something we can change in a file on the
> device while B2G is stopped to affect whether FTU is launched?

Maybe but I am not familiar with change settings at runtime..

> 
> > > Can we simulate closing/skipping 'normally' without
> > > completing all FTU steps via interactions?
> > 
> > We could do something in system app to receive the request to skip FTU
> > programmatically as my last comment.
> > For example: |Service.request('skipftu');|
> 
> That could work for us, assuming that we could call this even if the FTU is
> already running?

Yes, but this needs additional work in system app.

If there's no other ways, I will do this tomorrow.

Comment 25

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Alive Kuo [:alive][NEEDINFO!] from comment #24)
> > We can change the settings, however I'm not sure we'd be able to change this
> > one early enough to influence booting into the FTU. We reset as much as
> 
> What do you mean by early enough and how do you reset?

We stop b2g, reset the device, and start b2g again. If changing the settings to prevent the FTU from launching needs to be done via Marionette via JavaScript (for example) then we might not be able to execute it between starting b2g and the FTU launching.

When we reset we delete a bunch of directories from the device:
https://github.com/mozilla-b2g/gaia/blob/ac82d6c5ce759c2fa47ca3cc8c261b67f4b2b49b/tests/python/gaia-ui-tests/gaiatest/gaia_test.py#L811

> > Is there something we can change in a file on the
> > device while B2G is stopped to affect whether FTU is launched?
> 
> Maybe but I am not familiar with change settings at runtime..

Do you know who might be able to help answer this?

> > That could work for us, assuming that we could call this even if the FTU is
> > already running?
> 
> Yes, but this needs additional work in system app.
> 
> If there's no other ways, I will do this tomorrow.

Thanks.

Comment 26

[GitHub Autolander Bot]

Attachment: [gaia] alivedise:bugzilla/1139987/skip-ftu-service > mozilla-b2g:master

Comment 27

[(inactive after 6/18) Alive Kuo [:alive]]

Comment on attachment 8578406 [details] [review]
[gaia] alivedise:bugzilla/1139987/skip-ftu-service > mozilla-b2g:master

I finally decided not to implement a test-only service but fix the problem that we do not finish all steps when ftu is closed (by mozbrowsererror).

I still think the test framework should provide the functionality to easily change settings at each run, but it's impossible to be implemented right now; I'd talked to Al Tsai and he is investigating how marionette-js does that.

Comment 28

[GitHub Autolander Bot]

Attachment: [gaia] alivedise:bugzilla/1143577/update-mute-icon > mozilla-b2g:master

Comment 29

[Etienne Segonzac (:etienne)]

Comment on attachment 8578406 [details] [review]
[gaia] alivedise:bugzilla/1139987/skip-ftu-service > mozilla-b2g:master

looking good :)

Comment 30

[(inactive after 6/18) Alive Kuo [:alive]]

https://github.com/mozilla-b2g/gaia/commit/70bac3afc266da3aee0622d49a0c78fee7112e12

BTW Al Tsai told me he is able to change ftu.manifestURL settings before the test...

Al could you confirm?

Comment 31

[Al Tsai [:atsai]]

Yes. I use GaiaData to set ftu.manifestURL to 'null' ,restart the device, and won't see the FTU anymore. Once I set the ftu.manifestURL back to it's value "app://ftu.gaiamobile.org/manifest.webapp", I will see the FTU and won't be able to skip unless I go through every detail.

This should be a proper approach then simply kill all apps. I can provide a patch for it if you'd like to. I'd like to have your opinion, Dave.

Comment 32

[Dave Hunt [:davehunt] [he/him] ⌚BST]

When we restart devices we reset all settings, so we'd lose the change. The reset is to ensure the device is in a consistent known state. We'd need to influence the FTU startup outside of Marionette.

Comment 33

[Al Tsai [:atsai]]

A script like this one might help? https://github.com/altsai/dev-tools/blob/master/b2g/scripts/skip_ftu.py

Comment 34

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Al Tsai [:atsai] from comment #33)
> A script like this one might help?
> https://github.com/altsai/dev-tools/blob/master/b2g/scripts/skip_ftu.py

No, as I explained we reset the device between tests and restarting the device twice per test would add an unacceptable overhead. I have found a solution where we can modify the default settings on the device though, see bug 1145243 for details.