Intermittent Android 4.0 debug jsreftest/reftest failures with no error summary (timeout/crash/hang?)

RESOLVED FIXED

Status

Release Engineering
Buildduty
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: emorley, Unassigned)

Tracking

(Blocks: 1 bug, {intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
Android 4.0 Panda mozilla-inbound debug test jsreftest-1 on 2014-08-07 00:37:05 PDT for push 1f01cae60e26

slave: panda-0587

https://tbpl.mozilla.org/php/getParsedLog.php?id=45400466&tree=Mozilla-Inbound

01:02:40     INFO -  REFTEST TEST-START | http://10.26.133.22:30587/jsreftest/tests/jsreftest.html?test=ecma/Statements/12.6.3-19.js
01:02:40     INFO -  REFTEST TEST-LOAD | http://10.26.133.22:30587/jsreftest/tests/jsreftest.html?test=ecma/Statements/12.6.3-19.js | 520 / 2114 (24%)
01:02:40     INFO -  REFTEST TEST-PASS | http://10.26.133.22:30587/jsreftest/tests/jsreftest.html?test=ecma/Statements/12.6.3-19.js | count = 0; result = ""; function f() { count++; return new Array("h","e","l","l","o"); }for ( p in f() ) { result += f()[p] }; count  item 1
01:02:40     INFO -  REFTEST TEST-PASS | http://10.26.133.22:30587/jsreftest/tests/jsreftest.html?test=ecma/Statements/12.6.3-19.js | result  item 2
01:02:40     INFO -  REFTEST INFO | Loading a blank page
01:02:40     INFO -  REFTEST TEST-END | http://10.26.133.22:30587/jsreftest/tests/jsreftest.html?test=ecma/Statements/12.6.3-19.js
01:02:40     INFO -  REFTEST TEST-START | http://10.26.133.22:30587/jsreftest/tests/jsreftest.html?test=ecma/Statements/12.6.3-2.js
01:02:40     INFO -  REFTEST TEST-LOAD | http://10.26.133.22:30587/jsreftest/tests/jsreftest.html?test=ecma/Statements/12.6.3-2.js | 521 / 2114 (24%)
01:02:40     INFO -  INFO | automation.py | Application ran for: 0:11:15.595872

It looks like 12.6.3-2.js starts, but we never get a TEST-END nor timeout or anything useful.
(Reporter)

Comment 1

3 years ago
Android 4.0 Panda mozilla-inbound opt test plain-reftest-2 on 2014-08-11 06:23:25 PDT for push 2e0e4d841f0f

https://tbpl.mozilla.org/php/getParsedLog.php?id=45663844&full=1&branch=mozilla-inbound

06:54:21     INFO -  REFTEST TEST-START | http://10.26.133.22:30587/tests/layout/reftests/bugs/403656-4.html
06:54:21     INFO -  REFTEST TEST-LOAD | http://10.26.133.22:30587/tests/layout/reftests/bugs/403656-4.html | 1186 / 1335 (88%)
06:54:21     INFO -  REFTEST TEST-LOAD | http://10.26.133.22:30587/tests/layout/reftests/bugs/403656-4-ref.html | 1186 / 1335 (88%)
06:54:21     INFO -  INFO | automation.py | Application ran for: 0:23:09.555395
(Reporter)

Comment 2

3 years ago
Slightly more useful:

https://tbpl.mozilla.org/php/getParsedLog.php?id=45706249&full=1&branch=fx-team

17:08:23     INFO -  REFTEST TEST-START | http://10.26.133.22:30584/jsreftest/tests/jsreftest.html?test=js1_5/extensions/regress-322957.js
17:08:23     INFO -  REFTEST TEST-LOAD | http://10.26.133.22:30584/jsreftest/tests/jsreftest.html?test=js1_5/extensions/regress-322957.js | 138 / 2375 (5%)
17:11:24     INFO -  mozdevice ERROR | DeviceManager: pull unsuccessful: no prompt found after file data--DeviceManager may be out of sync with agent
17:11:24     INFO -  INFO | automation.py | Application ran for: 0:04:02.888589
(Reporter)

Comment 3

3 years ago
Android 4.0 Panda mozilla-inbound debug test plain-reftest-7 on 2014-08-12 08:46:35 PDT for push f497c1d55fd0

slave: panda-0587

https://tbpl.mozilla.org/php/getParsedLog.php?id=45759429&full=1&branch=mozilla-inbound
Geoff, these are pretty widespread at the moment (and due to the failure mode, we just kind of wave our hands at them and retrigger). Any ideas who can help with this?

https://tbpl.mozilla.org/php/getParsedLog.php?id=46046060&full=1&branch=mozilla-central
https://tbpl.mozilla.org/php/getParsedLog.php?id=46043820&tree=Mozilla-Inbound&full=1
Flags: needinfo?(gbrown)
Keywords: intermittent-failure
Summary: Android 4.0 debug jsreftest-1 job with no error summary (timeout/crash/hang?) → Intermittent Android 4.0 debug jsreftest/reftest job with no error summary (timeout/crash/hang?)
https://tbpl.mozilla.org/php/getParsedLog.php?id=46049048&full=1&branch=mozilla-central
Summary: Intermittent Android 4.0 debug jsreftest/reftest job with no error summary (timeout/crash/hang?) → Intermittent Android 4.0 debug jsreftest/reftest failures with no error summary (timeout/crash/hang?)
https://tbpl.mozilla.org/php/getParsedLog.php?id=46052833&tree=Mozilla-Inbound&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=46060046&tree=Mozilla-Inbound&full=1

Updated

3 years ago
See Also: → bug 1052523
:snorp - Do you have time / people to look at this? Pandas are rebooting mid-test in several jobs, on both debug and opt. I'll try to have a look through more logs on Monday.
Flags: needinfo?(snorp)
https://tbpl.mozilla.org/php/getParsedLog.php?id=46067786&tree=Mozilla-Inbound&full=1
These are not specific to one particular slave, but there are several duplicates. Let's track the affected slaves.

Comment 0: panda-0587
Comment 1: panda-0587
Comment 2: panda-0584
Comment 3: panda-0587
Comment 4: panda-0587, panda-0592
Comment 5: panda-0584
Comment 6: panda-0584, panda-0584
Comment 8: panda-0501

:kmoir -- Can you think of what might be causing these Pandas to reboot during tests?
Flags: needinfo?(gbrown) → needinfo?(kmoir)
There are 3 similar failures in bug 1052523 -- all are panda-0587.

Comment 11

3 years ago
Pandas 0587-592 are all on foopy 79

panda-501 is on foopy72

I can disable panda-584 and panda-587, they seem to be causing problems and their reports in slave health show a large number of retries/failures.
Flags: needinfo?(kmoir)

Comment 12

3 years ago
See bug 1055125 and bug 1055117 for panda slave tracking bugs
Any new failures since those pandas were disabled? Anything more to do here?
Flags: needinfo?(snorp) → needinfo?(emorley)
(Reporter)

Comment 14

3 years ago
Not sure, since I'd virtually reached the point of generically starring any occurrences - but from memory I don't think I've seen any in the last few days. Happy to call this fixed and reopen if needs be :-)
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Flags: needinfo?(emorley)
Resolution: --- → FIXED
Component: Reftest → Buildduty
Product: Testing → Release Engineering
QA Contact: bugspam.Callek
Version: Trunk → unspecified
(Reporter)

Comment 15

3 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=46364097&tree=Mozilla-Inbound
panda-0592
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
No new entries since 8-20, resolving again
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.