1068946 - Frequent B2G debug cpptest "DMError: Timeout exceeded for shell call" at the start of jsapi-tests

Status: RESOLVED FIXED

(Firefox OS Graveyard :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

These have been getting reported under Bug 908268 since late last week, with a big spike starting a couple days ago. Right now, the failure rate is ~50%. If not addressed, it's enough to consider hiding the suite again.

Here's a typical failing run:
https://tbpl.mozilla.org/php/getParsedLog.php?id=48305865&tree=Mozilla-Central&full=1#error0

Compare to a green run:
https://tbpl.mozilla.org/php/getParsedLog.php?id=48140885&tree=Mozilla-Central&full=1

What immediately stands out to me is:
12:33:25     INFO -  *** UTM:SVC TimerManager:registerTimer - id: user-agent-updates-timer

Retriggers on m-c are pointing to this inbound merge as when things went orange-crazy, but I don't see anything obvious in it:
https://tbpl.mozilla.org/?jobname=b2g_emulator_vm%20mozilla-central%20debug%20test%20cppunit&rev=02c3915bf742

We've had multiple issues in the past with update timers causing crazy things during tests in other B2G suites. Is it possible that we aren't properly disabling them for xpcshell?

Comment 1

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48313169&tree=B2g-Inbound
b2g_emulator_vm b2g-inbound debug test cppunit on 2014-09-17 13:57:01
revision: e1489f99516d
slave: tst-linux64-spot-1225

DMError: Timeout exceeded for shell call
Return code: 1

Comment 2

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48313477&tree=Fx-Team
b2g_emulator_vm fx-team debug test cppunit on 2014-09-17 13:58:42
revision: 2bebc67aa52c
slave: tst-linux64-spot-1196

Traceback (most recent call last):
DMError: Timeout exceeded for shell call
Return code: 1

Comment 3

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48317280&tree=Mozilla-Inbound
b2g_emulator_vm mozilla-inbound debug test cppunit on 2014-09-17 14:49:05
revision: 89c373278e79
slave: tst-linux64-spot-1272

DMError: Timeout exceeded for shell call
TEST-UNEXPECTED-FAIL | jsapi-tests | test failed with return code 563735
Return code: 1

Comment 4

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48321201&tree=Mozilla-Inbound
b2g_emulator_vm mozilla-inbound debug test cppunit on 2014-09-17 15:40:49
revision: f023469654dd
slave: tst-linux64-spot-702

DMError: Timeout exceeded for shell call
Return code: 1

Comment 5

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48323409&tree=Mozilla-Inbound
b2g_emulator_vm mozilla-inbound debug test cppunit on 2014-09-17 16:00:45
revision: 6af00329904f
slave: tst-linux64-spot-1115

DMError: Timeout exceeded for shell call
TEST-UNEXPECTED-FAIL | jsapi-tests | test failed with return code 720949
Return code: 1

Comment 6

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48325777&tree=Mozilla-Inbound
b2g_emulator_vm mozilla-inbound debug test cppunit on 2014-09-17 16:28:08
revision: a3adee0a1c5f
slave: tst-linux64-spot-1190

DMError: Timeout exceeded for shell call
Return code: 1

Comment 7

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48324092&tree=Mozilla-Inbound
b2g_emulator_vm mozilla-inbound debug test cppunit on 2014-09-17 16:09:35
revision: 46859d329455
slave: tst-linux64-spot-1166

DMError: Timeout exceeded for shell call
Return code: 1

Comment 8

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48327688&tree=Mozilla-Inbound
b2g_emulator_vm mozilla-inbound debug test cppunit on 2014-09-17 16:47:01
revision: 40efa924899c
slave: tst-linux64-spot-1263

DMError: Timeout exceeded for shell call
Return code: 1

Comment 9

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48332156&tree=Fx-Team
b2g_emulator_vm fx-team debug test cppunit on 2014-09-17 17:37:36
revision: ce7c078188ad
slave: tst-linux64-spot-248

DMError: Timeout exceeded for shell call
Return code: 1

Comment 10

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48336538&tree=Mozilla-Inbound
b2g_emulator_vm mozilla-inbound debug test cppunit on 2014-09-17 18:54:08
revision: 3dad11e6dcdd
slave: tst-linux64-spot-796

DMError: Timeout exceeded for shell call
Return code: 1

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Maybe we need something like this ported to remotecppunittests.py?
https://bugzilla.mozilla.org/attachment.cgi?id=8463826&action=diff

Comment 12

[Legacy TBPL/Treeherder Robot]

Tomcat
https://tbpl.mozilla.org/php/getParsedLog.php?id=48379351&tree=Mozilla-Inbound
b2g_emulator_vm mozilla-inbound debug test cppunit on 2014-09-18 06:01:25
revision: 04458f023e47
slave: tst-linux64-spot-1293

DMError: Timeout exceeded for shell call
TEST-UNEXPECTED-FAIL | jsapi-tests | test failed with return code 720949
Return code: 1

Comment 13

[Legacy TBPL/Treeherder Robot]

Tomcat
https://tbpl.mozilla.org/php/getParsedLog.php?id=48375629&tree=Mozilla-Central
b2g_emulator_vm mozilla-central debug test cppunit on 2014-09-18 05:13:37
revision: 426497473505
slave: tst-linux64-spot-778

DMError: Timeout exceeded for shell call
TEST-UNEXPECTED-FAIL | jsapi-tests | test failed with return code 604087
Return code: 1

Comment 14

[Legacy TBPL/Treeherder Robot]

Tomcat
https://tbpl.mozilla.org/php/getParsedLog.php?id=48381543&tree=Mozilla-Inbound
b2g_emulator_vm mozilla-inbound debug test cppunit on 2014-09-18 06:42:09
revision: 82461156faa4
slave: tst-linux64-spot-459

DMError: Timeout exceeded for shell call
Return code: 1

Comment 15

[Legacy TBPL/Treeherder Robot]

Tomcat
https://tbpl.mozilla.org/php/getParsedLog.php?id=48380122&tree=B2g-Inbound
b2g_emulator_vm b2g-inbound debug test cppunit on 2014-09-18 06:16:06
revision: 8b9750f96fd7
slave: tst-linux64-spot-929

DMError: Timeout exceeded for shell call
Return code: 1

Comment 16

[Dan Minor [:dminor]]

Attachment: Disable jsapi tests for b2g emulator.

Disabling the jsapi tests seems to stabilize things [1]. More work is needed to figure out the root cause of this problem.

[1] https://tbpl.mozilla.org/?tree=Try&rev=f825e077ce86

Comment 17

[Dan Minor [:dminor]]

Thanks, pushed to https://hg.mozilla.org/integration/mozilla-inbound/rev/46fc3e04fcbb

Comment 18

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48388373&tree=B2g-Inbound
b2g_emulator_vm b2g-inbound debug test cppunit on 2014-09-18 09:26:28
revision: fa8c3366cbcf
slave: tst-linux64-spot-417

DMError: Timeout exceeded for shell call
TEST-UNEXPECTED-FAIL | jsapi-tests | test failed with return code 720949
Return code: 1

Comment 19

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48388799&tree=Mozilla-Inbound
b2g_emulator_vm mozilla-inbound debug test cppunit on 2014-09-18 09:34:05
revision: 62ceb91926f0
slave: tst-linux64-spot-1153

DMError: Timeout exceeded for shell call
TEST-UNEXPECTED-FAIL | jsapi-tests | test failed with return code 720949
Return code: 1

Comment 20

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48390655&tree=Mozilla-Inbound
b2g_emulator_vm mozilla-inbound debug test cppunit on 2014-09-18 10:04:25
revision: f62b115f169f
slave: tst-linux64-spot-082

DMError: Timeout exceeded for shell call
Return code: 1

Comment 21

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48390993&tree=B2g-Inbound
b2g_emulator_vm b2g-inbound debug test cppunit on 2014-09-18 10:05:39
revision: 16e1bd2cc50f
slave: tst-linux64-spot-651

DMError: Timeout exceeded for shell call
TEST-UNEXPECTED-FAIL | jsapi-tests | test failed with return code 535703
Return code: 1

Comment 22

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48392034&tree=Mozilla-Inbound
b2g_emulator_vm mozilla-inbound debug test cppunit on 2014-09-18 10:21:39
revision: 1da7bca1c63d
slave: tst-linux64-spot-955

DMError: Timeout exceeded for shell call
Return code: 1

Comment 23

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48392998&tree=Mozilla-Inbound
b2g_emulator_vm mozilla-inbound debug test cppunit on 2014-09-18 10:30:14
revision: ecea1b6ae733
slave: tst-linux64-spot-638

DMError: Timeout exceeded for shell call
TEST-UNEXPECTED-FAIL | jsapi-tests | test failed with return code 720949
Return code: 1

Comment 24

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48393332&tree=B2g-Inbound
b2g_emulator_vm b2g-inbound debug test cppunit on 2014-09-18 10:34:40
revision: 0b26525f3ddd
slave: tst-linux64-spot-559

DMError: Timeout exceeded for shell call
Return code: 1

Comment 25

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48394751&tree=B2g-Inbound
b2g_emulator_vm b2g-inbound debug test cppunit on 2014-09-18 10:49:56
revision: ac4bfb631c1e
slave: tst-linux64-spot-337

DMError: Timeout exceeded for shell call
Return code: 1

Comment 26

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48395606&tree=B2g-Inbound
b2g_emulator_vm b2g-inbound debug test cppunit on 2014-09-18 11:02:22
revision: e8ffc02bd089
slave: tst-linux64-spot-429

DMError: Timeout exceeded for shell call
Return code: 1

Comment 27

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48397855&tree=B2g-Inbound
b2g_emulator_vm b2g-inbound debug test cppunit on 2014-09-18 11:29:37
revision: ea47a4e169c5
slave: tst-linux64-spot-714

DMError: Timeout exceeded for shell call
Return code: 1

Comment 28

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48397933&tree=Fx-Team
b2g_emulator_vm fx-team debug test cppunit on 2014-09-18 11:29:53
revision: d1ae4274cf8b
slave: tst-linux64-spot-106

DMError: Timeout exceeded for shell call
Return code: 1

Comment 29

[Legacy TBPL/Treeherder Robot]

KWierso
https://tbpl.mozilla.org/php/getParsedLog.php?id=48406260&tree=B2g-Inbound
b2g_emulator_vm b2g-inbound debug test cppunit on 2014-09-18 14:07:52
revision: 2d5de8b63f69
slave: tst-linux64-spot-300

DMError: Timeout exceeded for shell call
Return code: 1

Comment 30

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/46fc3e04fcbb

Comment 31

[Legacy TBPL/Treeherder Robot]

KWierso
https://tbpl.mozilla.org/php/getParsedLog.php?id=48417075&tree=Mozilla-Central
b2g_emulator_vm mozilla-central debug test cppunit on 2014-09-18 16:26:23
revision: d6f78faaefcc
slave: tst-linux64-spot-025

DMError: Timeout exceeded for shell call
Return code: 1

Comment 32

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48418411&tree=Mozilla-Central
b2g_emulator_vm mozilla-central debug test cppunit on 2014-09-18 16:42:58
revision: c19df62989e1
slave: tst-linux64-spot-900

DMError: Timeout exceeded for shell call
Return code: 1

Comment 33

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=48421983&tree=B2g-Inbound
b2g_emulator_vm b2g-inbound debug test cppunit on 2014-09-18 17:22:05
revision: b030bb382b31
slave: tst-linux64-spot-388

DMError: Timeout exceeded for shell call
Return code: 1

Comment 34

[Legacy TBPL/Treeherder Robot]

Tomcat
https://tbpl.mozilla.org/php/getParsedLog.php?id=48450010&tree=Mozilla-Inbound
b2g_emulator_vm mozilla-inbound debug test mochitest-debug-5 on 2014-09-19 01:53:59
revision: 3473082f0de4
slave: tst-linux64-spot-1298

DMError: Timeout exceeded for shell call
DMError: Non-zero return code for command: ['stop', 'b2g'] (output: '', retval: 'None')
0 ERROR Automation Error: Received unexpected exception while running application
Return code: 1
Return code: 1

Comment 35

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-11-28T16:53:39
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-central&job_id=704781
repository: mozilla-central
who: philringnalda[at]gmail[dot]com
machine: tst-linux64-spot-329
buildname: b2g_emulator_vm mozilla-central opt test cppunit
revision: e90536aa55dd

DMError: Timeout exceeded for shell call
Return code: 1
No tests run or test summary not found

Comment 36

[Dan Minor [:dminor]]

(In reply to TBPL Robot from comment #35)
> submit_timestamp: 2014-11-28T16:53:39
> log:
> https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-
> central&job_id=704781
> repository: mozilla-central
> who: philringnalda[at]gmail[dot]com
> machine: tst-linux64-spot-329
> buildname: b2g_emulator_vm mozilla-central opt test cppunit
> revision: e90536aa55dd
> 
> DMError: Timeout exceeded for shell call
> Return code: 1
> No tests run or test summary not found

The failure here was in test_statement_scoper not jsapi-tests. I was afraid I had missed skipping the jsapi-tests failure when I changed up the cppunit test manifest format.

Comment 37

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-12-11T00:21:50
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-inbound&job_id=4543224
repository: mozilla-inbound
who: tomcat[at]mozilla[dot]com
machine: tst-linux64-spot-074
buildname: b2g_emulator_vm mozilla-inbound debug test cppunit
revision: 99d8d92a319b

DMError: Timeout exceeded for shell call
Return code: 1
No tests run or test summary not found

Comment 38

[Dan Minor [:dminor]]

(In reply to TBPL Robot from comment #37)
> submit_timestamp: 2014-12-11T00:21:50
> log:
> https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-
> inbound&job_id=4543224
> repository: mozilla-inbound
> who: tomcat[at]mozilla[dot]com
> machine: tst-linux64-spot-074
> buildname: b2g_emulator_vm mozilla-inbound debug test cppunit
> revision: 99d8d92a319b
> 
> DMError: Timeout exceeded for shell call
> Return code: 1
> No tests run or test summary not found

The failure here was in test_transaction_helper not in jsapi-tests. I think we should either change the summary here or file a new bug for these failures so they're being starred correctly. I'm not sure offhand if this is the same problem that was affecting the now disabled jsapi-tests or if this is something new :/.

Comment 39

[Dan Minor [:dminor]]

Closing this for now. If/when someone cares enough about the jsapi-tests on the b2g emulator to notice they're no longer enabled we can follow through on identifying a root cause for this.