840929 - B2G performance tests are failing and unable to recover from failure

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Since bug 840325 has landed, we now see that at some point during the testrun a failure occurs, and the suite is unable to recover. It appears that we may be affected by Marionette becoming out of sync (see bug 839675).

Example console output from failed run (repeated lines removed for brevity)

Gallery: [21/30]
Gallery: [22/30]
Gallery: [23/30]
timed out
Gallery: [24/31]
timed out
...
Exceeded failure threshold for gathering results!
Contacts: [1/30]
{u'from': u'conn1.marionette1', u'value': {u'origin': u'app://gallery.gaiamobile.org', u'src': u'app://gallery.gaiamobile.org/index.html', u'frame': {u'ELEMENT': u'{5e9b80a3-a5be-4c30-b1a7-1a14d2d458ac}'}, u'name': u'Gallery', u'cold_load_time': 5121}}
Contacts: [2/31]
{u'from': u'conn1.marionette1', u'value': {u'origin': u'app://gallery.gaiamobile.org', u'src': u'app://gallery.gaiamobile.org/index.html', u'frame': {u'ELEMENT': u'{5e9b80a3-a5be-4c30-b1a7-1a14d2d458ac}'}, u'name': u'Gallery', u'cold_load_time': 5121}}
...
Exceeded failure threshold for gathering results!
Music: [1/30]
{u'from': u'conn1.marionette1', u'value': {u'origin': u'app://gallery.gaiamobile.org', u'src': u'app://gallery.gaiamobile.org/index.html', u'frame': {u'ELEMENT': u'{5e9b80a3-a5be-4c30-b1a7-1a14d2d458ac}'}, u'name': u'Gallery', u'cold_load_time': 5121}}
Music: [2/31]
{u'from': u'conn1.marionette1', u'value': {u'origin': u'app://gallery.gaiamobile.org', u'src': u'app://gallery.gaiamobile.org/index.html', u'frame': {u'ELEMENT': u'{5e9b80a3-a5be-4c30-b1a7-1a14d2d458ac}'}, u'name': u'Gallery', u'cold_load_time': 5121}}
...
Music: [7/36]
{u'from': u'conn1.marionette1', u'value': {u'origin': u'app://communications.gaiamobile.org/contacts/index.html', u'src': u'app://communications.gaiamobile.org/contacts/index.html', u'frame': {u'ELEMENT': u'{6c0c1553-9504-4be5-872c-33e05d099e46}'}, u'name': u'Contacts', u'cold_load_time': 5121}}
Exceeded failure threshold for gathering results!
Camera: [1/30]
{u'ok': True, u'from': u'conn1.marionette1'}
Camera: [2/31]
{u'ok': True, u'from': u'conn1.marionette1'}
...
Exceeded failure threshold for gathering results!

Comment 1

[Jonathan Griffin (:jgriffin)]

I believe this is actually due to the UI freezing.  Will attempt to confirm.

Comment 2

[Jonathan Griffin (:jgriffin)]

I've confirmed this is due to the UI freezing.  When this happens, the frozen app is completely unresponsive, and often other apps are as well, although it is possible to go to the homescreen and launch other apps.

I'm attempting to work around this for the perf tests by changing the Jenkins job to launch b2gperf once per app, and reboot the phone in between.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Looks like relaunching between apps is working well, thanks Jonathan! Do you think this is something we should consider to build into b2gperf? We've already started to look into this for the Gaia UI tests so I suspect we could share the solution easily enough.

Comment 4

[Jonathan Griffin (:jgriffin)]

Yeah I think it makes sense to have b2gperf handle this.

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This has been fixed in https://github.com/davehunt/b2gperf/commit/323e9c7853c498a7970a6ba0d2a07bf5becfae99