957969 - Job failed but no error shown in the console log

Status: RESOLVED WORKSFORME

(Mozilla QA Graveyard :: Infrastructure, defect, P2)

Description

[Andreea Matei [:AndreeaMatei]]

There was a job on mm-osx-106-3 with the following outcome, after the restart tests results, we failed but with no error shown in the console log:

http://mm-ci-master.qa.scl3.mozilla.com:8080/job/mozilla-central_functional/18646/

04:57:55 TEST-PASS |
restartTests/testPreferences_masterPassword/test1.js | testSetMasterPassword
04:57:55 TEST-START |
restartTests/testPreferences_masterPassword/test1.js | teardownModule
04:57:55 TEST-END | restartTests/testPreferences_masterPassword/test1.js
| finished in 5692ms
04:58:56 RESULTS | Passed: 26
04:58:56 RESULTS | Failed: 0
04:58:56 RESULTS | Skipped: 6
04:58:58 Traceback (most recent call last):
04:58:58 Build step 'Invoke XShell command' marked build as failure
04:58:58 Archiving artifacts
04:58:58 Recording test results
04:58:58 Email was triggered for: Failure
04:58:58 Sending email for trigger: Failure
04:59:54 Sending email to: [e-mail]
04:59:55 Finished: FAILURE

I connected to the machine, it was running another testrun, everything looked fine. Retriggered the job but it ran on a different osx machine and passed.

Comment 1

[Andreea Matei [:AndreeaMatei]]

Happened again today with central_functional, mm-osx-108-4:
http://mm-ci-master.qa.scl3.mozilla.com:8080/job/mozilla-central_functional/18717/

04:29:57 TEST-START | restartTests/testAddons_installTheme/test1.js | teardownModule
04:29:57 TEST-END | restartTests/testAddons_installTheme/test1.js | finished in 2293ms
04:30:57 Fault in cycle collector: overflowing refcount (ptr: 0x110764aa0)
04:30:57 [92599] ###!!! ABORT: cycle collector fault: file ../../../../xpcom/base/nsCycleCollector.cpp, line 1248
04:30:57 [92599] ###!!! ABORT: cycle collector fault: file ../../../../xpcom/base/nsCycleCollector.cpp, line 1248
04:30:57 RESULTS | Passed: 7
04:30:57 RESULTS | Failed: 0
04:30:57 RESULTS | Skipped: 3
04:30:58 Traceback (most recent call last):
04:30:58 Build step 'Invoke XShell command' marked build as failure
04:30:58 Archiving artifacts
04:30:58 Recording test results
04:32:57 Email was triggered for: Failure
04:32:57 Sending email for trigger: Failure
04:33:49 Sending email to: [e-mail]
04:33:49 Finished: FAILURE

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This is critical and needs to be fully investigated. Who can do that?

Comment 3

[Andreea Matei [:AndreeaMatei]]

Giving that both times when this failed were on osx, we're going to use our local CI and all the mac machines we have to run and reproduce this, while looking through the code.

I'll update the bug if it happens again today.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Btw. how do the reports look like in the dashboard? Are those complete? It might be that due to caching the console output hasn't reached the master for the log.

Comment 5

[Andreea Matei [:AndreeaMatei]]

There was no report link in the email or the console log. :(

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Well, you have the build id and the platform. It should be easy to query for it in the dashboard.

Comment 7

[Andreea Matei [:AndreeaMatei]]

I think this is the one for the job in comment 1:
http://mozmill-daily.blargon7.com/#/functional/report/0fc916b47806e5d11cba8af7dc05ab30

Comment 8

[Cosmin Malutan, [:cosmin-malutan]]

As a duty I saw this a lot today though I'm not sure the failure from comment 1 is the same with the one from comment 2 (Cycle collection).
I couldn't reproduce this failure, I run testruns locally with the affected build, also I run over 15 testruns on the node that had the most failures (mm-osx-106-3).
What I have noticed  is that if fails in /restartTests/testAddons_installTheme/ or a couple of tests after.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Cosmin Malutan from comment #8)
> As a duty I saw this a lot today though I'm not sure the failure from
> comment 1 is the same with the one from comment 2 (Cycle collection).

Nothing anyone else can comment on because any detailed information is missing again. I really don't want to have to request that each time. But comment 2 is totally unrelated here, and needs a separate investigation.

> I couldn't reproduce this failure, I run testruns locally with the affected
> build, also I run over 15 testruns on the node that had the most failures
> (mm-osx-106-3).

15 testruns are not a lot. I would run those in a loop until a failure happens.

> What I have noticed  is that if fails in
> /restartTests/testAddons_installTheme/ or a couple of tests after.

With what result? Means, what was your action so far on those failures?

Comment 10

[Cosmin Malutan, [:cosmin-malutan]]

Attachment: jenkins log

(In reply to Henrik Skupin (:whimboo) from comment #9)
> With what result? Means, what was your action so far on those failures?
That was on cycle collector crashes.

This failed again on Vista x86 (mm-win-vista-32-1) with Firefox 28.0a2 fr.

Comment 11

[Andreea Matei [:AndreeaMatei]]

Did not failed again in the last 2 weeks, I'll decrease priority. We'll keep an eye if it happens again.

Comment 12

[Andrei Eftimie]

Lets close this with WFM.
Please reopen if this happens again.