1074929 - Test failure "Application crashed" in /restartTests/testAddons_uninstallExtension/test3.js

Status: RESOLVED INVALID

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[Tracy Walker [:tracy]]

Seeing this one frequently in Mozmill release-mozilla-beta_functional testrun for Firefox 33.0

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Usually Daniel should have gone through all those reports today, and filed bugs appropriately. But sadly I can see your confusion here. There is not a single reply on the CI mailing list in case of crashes happened! This is bad! I mentioned this a couple of times, especially that this can cause confusion for QA people. Now we have such a situation. Please really try to get this solved for the duty.

Tracy, do you have some example jenkins urls? That would be very helpful! As of now it's hard to do action on it. So what has to be done here, is to login to the machine(s) and submit the crash reports. An upcoming version of Mozmill will be able to show the stack directly.

Comment 2

[Tracy Walker [:tracy]]

Here is the report for one such crash: http://mozmill-release.blargon7.com/#/functional/report/3dac09185ffafe61214f5a99be184340

Comment 3

[Andrei Eftimie]

(In reply to Henrik Skupin (:whimboo) from comment #1)
> Usually Daniel should have gone through all those reports today, and filed
> bugs appropriately. But sadly I can see your confusion here. There is not a
> single reply on the CI mailing list in case of crashes happened! This is
> bad! I mentioned this a couple of times, especially that this can cause
> confusion for QA people. Now we have such a situation. Please really try to
> get this solved for the duty.

With the recent changes to run all (most) locales against beta, the CI machines are on full capacity most of the day. I remember this was discussed on IRC yesterday a bit. 

To properly investigate (and submit) a crash report we must take the affected machine offline, which we can not currently do without disrupting the testrun batch. With > 1000 jobs in the queue during such a run this gets difficult to handle at the time of the crash.

What we've done before (and still do right now) is to mark everything that can't be handled on spot, and do that once the CI environment finishes with the heave load.

With the current load and requirements we might have to change this to be able to handle crashes faster.
I've filed https://github.com/mozilla/mozmill-ci/issues/503 to help us in that regard.

**TLDR: These crashes will be handled and ci mails replied, as always, but we have a slower reaction time due to CI machines much higher load**

Comment 4

[Daniel Gherasim]

(In reply to Henrik Skupin (:whimboo) from comment #1)
> Usually Daniel should have gone through all those reports today, and filed
> bugs appropriately. But sadly I can see your confusion here. There is not a
> single reply on the CI mailing list in case of crashes happened! This is
> bad! I mentioned this a couple of times, especially that this can cause
> confusion for QA people. Now we have such a situation. Please really try to
> get this solved for the duty.
> 

As I already assumpted yesterday on IRC to you, most of the crashes were on mm-osx-106-3, so bug 980800 it's the cause. Machine was busy running tests in the afternoon when the actual tests started. I tried to get the information via ssh, but I hadn't succeed, so I noted in the etherpad that I'll investigate this today.

I did this and my assumption was correct, so I gave the information in bug 980800.

Thanks!

** Oh, it's still today given the different timezone, so I still have time. :)

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Andrei Eftimie from comment #3)
> To properly investigate (and submit) a crash report we must take the
> affected machine offline, which we can not currently do without disrupting
> the testrun batch. With > 1000 jobs in the queue during such a run this gets
> difficult to handle at the time of the crash.

To mention it again, there is no need to take a machine offline for submitting crash reports. At least not for OS X and Linux. The crash reporter works perfectly from the command line when logged in via SSH. And indeed, I told this on IRC yesterday.

> What we've done before (and still do right now) is to mark everything that
> can't be handled on spot, and do that once the CI environment finishes with
> the heave load.

Please take the above under consideration. Windows might still be an exception, but we might wanna figure out how to get it solved there too.

(In reply to daniel.gherasim from comment #4)
> As I already assumpted yesterday on IRC to you, most of the crashes were on
> mm-osx-106-3, so bug 980800 it's the cause. Machine was busy running tests
> in the afternoon when the actual tests started. I tried to get the
> information via ssh, but I hadn't succeed, so I noted in the etherpad that
> I'll investigate this today.

There was enough information for you to see which test is failing. So this bug could have been created way earlier, and the information posted to the CI list. There is nothing here we would have to wait for. But I have no problems when the work regarding the underlying crash issue is delayed a bit.

Comment 6

[Tracy Walker [:tracy]]

Here's some from overnight all happen on Mac 10.6.8:
http://mozmill-release.blargon7.com/#/functional/report/41b4fea479d86c7288732f3ea8142b3a
http://mozmill-release.blargon7.com/#/functional/report/41b4fea479d86c7288732f3ea810d784
http://mozmill-release.blargon7.com/#/functional/report/41b4fea479d86c7288732f3ea8083670
http://mozmill-release.blargon7.com/#/functional/report/41b4fea479d86c7288732f3ea807b4d1
http://mozmill-release.blargon7.com/#/functional/report/41b4fea479d86c7288732f3ea804ded6
http://mozmill-release.blargon7.com/#/functional/report/41b4fea479d86c7288732f3ea8030705
http://mozmill-release.blargon7.com/#/functional/report/41b4fea479d86c7288732f3ea8026e36
http://mozmill-release.blargon7.com/#/functional/report/53ec3148fca67aea1b1c3528f94a7437

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I was hoping that the current CI duty would check for those crashes. It didn't happen since Friday. So Cosmin, can you please check the affected boxes from the last comment?

Comment 8

[Cosmin Malutan, [:cosmin-malutan]]

Sorry if I missed this bug, none of those crashes was under my watch, but they've been already addressed in bug 980800 comment 22.
All creases are on the same machine mm-osx-106-3 and the same test.
Marking this as dupe of bug 980800.

Comment 9

[Cosmin Malutan, [:cosmin-malutan]]

Ups sorry I closed this this is actually for tracking the test failure and the other is for tracking the crash in FF.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mozmill tests have been superseded by Marionette tests.