1050242 - Intermittent Android 4.0 debug jsreftest/reftest failures with no error summary (timeout/crash/hang?)

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Ed Morley [:emorley]]

Android 4.0 Panda mozilla-inbound debug test jsreftest-1 on 2014-08-07 00:37:05 PDT for push 1f01cae60e26

slave: panda-0587

https://tbpl.mozilla.org/php/getParsedLog.php?id=45400466&tree=Mozilla-Inbound

01:02:40     INFO -  REFTEST TEST-START | http://10.26.133.22:30587/jsreftest/tests/jsreftest.html?test=ecma/Statements/12.6.3-19.js
01:02:40     INFO -  REFTEST TEST-LOAD | http://10.26.133.22:30587/jsreftest/tests/jsreftest.html?test=ecma/Statements/12.6.3-19.js | 520 / 2114 (24%)
01:02:40     INFO -  REFTEST TEST-PASS | http://10.26.133.22:30587/jsreftest/tests/jsreftest.html?test=ecma/Statements/12.6.3-19.js | count = 0; result = ""; function f() { count++; return new Array("h","e","l","l","o"); }for ( p in f() ) { result += f()[p] }; count  item 1
01:02:40     INFO -  REFTEST TEST-PASS | http://10.26.133.22:30587/jsreftest/tests/jsreftest.html?test=ecma/Statements/12.6.3-19.js | result  item 2
01:02:40     INFO -  REFTEST INFO | Loading a blank page
01:02:40     INFO -  REFTEST TEST-END | http://10.26.133.22:30587/jsreftest/tests/jsreftest.html?test=ecma/Statements/12.6.3-19.js
01:02:40     INFO -  REFTEST TEST-START | http://10.26.133.22:30587/jsreftest/tests/jsreftest.html?test=ecma/Statements/12.6.3-2.js
01:02:40     INFO -  REFTEST TEST-LOAD | http://10.26.133.22:30587/jsreftest/tests/jsreftest.html?test=ecma/Statements/12.6.3-2.js | 521 / 2114 (24%)
01:02:40     INFO -  INFO | automation.py | Application ran for: 0:11:15.595872

It looks like 12.6.3-2.js starts, but we never get a TEST-END nor timeout or anything useful.

Comment 1

[Ed Morley [:emorley]]

Android 4.0 Panda mozilla-inbound opt test plain-reftest-2 on 2014-08-11 06:23:25 PDT for push 2e0e4d841f0f

https://tbpl.mozilla.org/php/getParsedLog.php?id=45663844&full=1&branch=mozilla-inbound

06:54:21     INFO -  REFTEST TEST-START | http://10.26.133.22:30587/tests/layout/reftests/bugs/403656-4.html
06:54:21     INFO -  REFTEST TEST-LOAD | http://10.26.133.22:30587/tests/layout/reftests/bugs/403656-4.html | 1186 / 1335 (88%)
06:54:21     INFO -  REFTEST TEST-LOAD | http://10.26.133.22:30587/tests/layout/reftests/bugs/403656-4-ref.html | 1186 / 1335 (88%)
06:54:21     INFO -  INFO | automation.py | Application ran for: 0:23:09.555395

Comment 2

[Ed Morley [:emorley]]

Slightly more useful:

https://tbpl.mozilla.org/php/getParsedLog.php?id=45706249&full=1&branch=fx-team

17:08:23     INFO -  REFTEST TEST-START | http://10.26.133.22:30584/jsreftest/tests/jsreftest.html?test=js1_5/extensions/regress-322957.js
17:08:23     INFO -  REFTEST TEST-LOAD | http://10.26.133.22:30584/jsreftest/tests/jsreftest.html?test=js1_5/extensions/regress-322957.js | 138 / 2375 (5%)
17:11:24     INFO -  mozdevice ERROR | DeviceManager: pull unsuccessful: no prompt found after file data--DeviceManager may be out of sync with agent
17:11:24     INFO -  INFO | automation.py | Application ran for: 0:04:02.888589

Comment 3

[Ed Morley [:emorley]]

Android 4.0 Panda mozilla-inbound debug test plain-reftest-7 on 2014-08-12 08:46:35 PDT for push f497c1d55fd0

slave: panda-0587

https://tbpl.mozilla.org/php/getParsedLog.php?id=45759429&full=1&branch=mozilla-inbound

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Geoff, these are pretty widespread at the moment (and due to the failure mode, we just kind of wave our hands at them and retrigger). Any ideas who can help with this?

https://tbpl.mozilla.org/php/getParsedLog.php?id=46046060&full=1&branch=mozilla-central
https://tbpl.mozilla.org/php/getParsedLog.php?id=46043820&tree=Mozilla-Inbound&full=1

Comment 5

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=46049048&full=1&branch=mozilla-central

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=46052833&tree=Mozilla-Inbound&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=46060046&tree=Mozilla-Inbound&full=1

Comment 7

[Geoff Brown [:gbrown]]

:snorp - Do you have time / people to look at this? Pandas are rebooting mid-test in several jobs, on both debug and opt. I'll try to have a look through more logs on Monday.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=46067786&tree=Mozilla-Inbound&full=1

Comment 9

[Geoff Brown [:gbrown]]

These are not specific to one particular slave, but there are several duplicates. Let's track the affected slaves.

Comment 0: panda-0587
Comment 1: panda-0587
Comment 2: panda-0584
Comment 3: panda-0587
Comment 4: panda-0587, panda-0592
Comment 5: panda-0584
Comment 6: panda-0584, panda-0584
Comment 8: panda-0501

:kmoir -- Can you think of what might be causing these Pandas to reboot during tests?

Comment 10

[Geoff Brown [:gbrown]]

There are 3 similar failures in bug 1052523 -- all are panda-0587.

Comment 11

[Kim Moir [:kmoir] ET]

Pandas 0587-592 are all on foopy 79

panda-501 is on foopy72

I can disable panda-584 and panda-587, they seem to be causing problems and their reports in slave health show a large number of retries/failures.

Comment 12

[Kim Moir [:kmoir] ET]

See bug 1055125 and bug 1055117 for panda slave tracking bugs

Comment 13

[Geoff Brown [:gbrown]]

Any new failures since those pandas were disabled? Anything more to do here?

Comment 14

[Ed Morley [:emorley]]

Not sure, since I'd virtually reached the point of generically starring any occurrences - but from memory I don't think I've seen any in the last few days. Happy to call this fixed and reopen if needs be :-)

Comment 15

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=46364097&tree=Mozilla-Inbound
panda-0592

Comment 16

[Justin Wood (:Callek)]

No new entries since 8-20, resolving again