1176502 - Failure in test_homescreen_column_layout.py on the Flame device, because Settings app gets killed

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::System::Window Mgmt, defect)

Description

[Martijn Wargers (dead)]

See: http://jenkins1.qa.scl3.mozilla.com/view/Bitbar/job/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2.bitbar/116/testReport/%28root%29/test_homescreen_column_layout_py%20TestHomescreenLayout_test_homescreen_column_layout/test_homescreen_column_layout_py_TestHomescreenLayout_test_homescreen_column_layout/

Traceback (most recent call last):
  File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2.bitbar/workspace/.env/lib/python2.7/site-packages/marionette_client-0.13-py2.7.egg/marionette/marionette_test.py", line 296, in run
    testMethod()
  File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2.bitbar/workspace/tests/python/gaia-ui-tests/gaiatest/tests/functional/homescreen/test_homescreen_column_layout.py", line 42, in test_homescreen_column_layout
    homescreen_settings.select_icon_layout('Four Columns')
  File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2.bitbar/workspace/tests/python/gaia-ui-tests/gaiatest/apps/settings/regions/homescreen_settings.py", line 16, in select_icon_layout
    expected.element_present(*self._icon_layout_locator))
  File "/var/lib/jenkins/jobs/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2.bitbar/workspace/.env/lib/python2.7/site-packages/marionette_driver-0.7-py2.7.egg/marionette_driver/wait.py", line 143, in until
    cause=last_exc)

TimeoutException: TimeoutException: Timed out after 10.3 seconds


I also could reproduce it locally.

The problem is that the Settings app gets killed after getting back to the Homescreen app, which deletes the state it was in (in the Homescreen select part).

Might be a regression similar as to bug 1172167, I don't know.

Comment 1

[Martijn Wargers (dead)]

[Blocking Requested - why for this release]:

Comment 2

[Martijn Wargers (dead)]

Just to be clear, this can be also reproduced manually on a Flame device, using 319MB.

Comment 3

[Martijn Wargers (dead)]

Attachment: https://github.com/mozilla-b2g/gaia/pull/30683

Pull request to disable this test for the Flame device (until this gets fixed)

Comment 4

[Martijn Wargers (dead)]

http://mxr.mozilla.org/gaia/source/tests/python/gaia-ui-tests/gaiatest/tests/functional/homescreen/test_homescreen_column_layout.py#41
40         # No need to open homescreen setting again
41         # because it remembers where it left off

And this is what's not happening anymore with the Flame at 319MB.

Comment 5

[Martijn Wargers (dead)]

(In reply to Martijn Wargers [:mwargers] (QA) from comment #3)
> Created attachment 8625070 [details] [review]
> https://github.com/mozilla-b2g/gaia/pull/30683
> 
> Pull request to disable this test for the Flame device (until this gets
> fixed)

Merged: https://github.com/mozilla-b2g/gaia/commit/3c01b0c7e4ac8e61806bf663f00e538382d1e0c9

Comment 6

[Martijn Wargers (dead)]

Ok, I did a retest repeating this test 10 times. Out of the 11 times, it failed 6 times. It failed every time, the last 5 times. That leds me to believe something is growing in memory, while repeating the test (something in the Marionette framework?).
Perhaps this test should be adjusted to not rely on where it left off after relaunch? Or is this supposed to always work for Flame devices with 319MB?

Comment 7

[Johan Lorenzo [:jlorenzo]]

To me, this issue seems like a regression of the memory usage and then a bug needs to be filed with a memory report to highlight which app has seen its memory grow[1].

In the meantime, how much is this test intermittent? I suppose it fails starting from the 5th time because we don't reboot the phone (unlike on typical runs), so we can't say this is failing more than 50% of the time, can we?

[1] https://wiki.mozilla.org/B2G/QA/Tips_And_Tricks#Getting_About_Memory_Logs_for_a_Bug

Comment 8

[Martijn Wargers (dead)]

(In reply to Johan Lorenzo [:jlorenzo] (QA) from comment #7)
> To me, this issue seems like a regression of the memory usage and then a bug
> needs to be filed with a memory report to highlight which app has seen its
> memory grow[1].
> 
> In the meantime, how much is this test intermittent? I suppose it fails
> starting from the 5th time because we don't reboot the phone (unlike on
> typical runs), so we can't say this is failing more than 50% of the time,
> can we?
> 
> [1]
> https://wiki.mozilla.org/B2G/QA/
> Tips_And_Tricks#Getting_About_Memory_Logs_for_a_Bug

I think it's not happening often at all, but as you said, it depends on how many tests have run before. It seems that not rebooting the phone and rerunning tests is filling the phone's memory somehow. Perhaps we should just reboot the phone between tests?

Anyway, perhaps this is a regression from bug 1094759 just like bug 1172167 is?
I could do some testing with a repeat of 10 and see if there is a regression here.

Comment 9

[Martijn Wargers (dead)]

(In reply to Johan Lorenzo [:jlorenzo] (QA) from comment #7)
> To me, this issue seems like a regression of the memory usage and then a bug
> needs to be filed with a memory report to highlight which app has seen its
> memory grow[1].

Ok, I filed bug 1177732 for this.

Comment 10

[Martijn Wargers (dead)]

(In reply to Martijn Wargers [:mwargers] (QA) from comment #8)
> Anyway, perhaps this is a regression from bug 1094759 just like bug 1172167
> is?
> I could do some testing with a repeat of 10 and see if there is a regression
> here.

Ok, it seems like there is a different regression range here:
Build ID               20150618010201
Gaia Revision          b404c41c5471c31610e64defb74ec066b411e724
Gaia Date              2015-06-17 17:01:15
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/a3f280b6f8d5
Gecko Version          41.0a1

Working, but failing (4 times, but not at the end) often:
Build ID               20150619010205
Gaia Revision          a0df9c367a68764bdcf2e2e1c4d27f0d6ee165b8
Gaia Date              2015-06-18 18:49:14
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/2694ff2ace6a
Gecko Version          41.0a1

Failing (6 times, although it had passed the 7th and 8th repeat):
Build ID               20150619160206
Gaia Revision          d8bac9577a537b5e6bafc3f7ed088af5ea60c99e
Gaia Date              2015-06-19 13:21:50
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/c319f262ce3e
Gecko Version          41.0a1

Failing the last 5 times:
Build ID               20150620160206
Gaia Revision          d8bac9577a537b5e6bafc3f7ed088af5ea60c99e
Gaia Date              2015-06-19 13:21:50
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/a38c23ccca0a
Gecko Version          41.0a1

Comment 11

[Martijn Wargers (dead)]

Tentative regression range for gecko:
https://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2015-06-19+00%3A00%3A00&enddate=2015-06-19+23%3A21%3A50

Comment 12

[Gregor Wagner [:gwagner]]

Not blocking on low-mem issues.

Comment 13

[Martijn Wargers (dead)]

Marking WONFTIX, sorry for the bug spam. If somebody still wants to work on this, please open a new bug.