Closed Bug 1326124 Opened 8 years ago Closed 1 year ago

Intermittent test_crash.py TestCrashInTearDown.test_crash_in_teardown | AssertionError: "Process crashed" does not match "Process killed because the connection to Marionette server is lost

Categories

(Testing :: Marionette Client and Harness, defect, P5)

Product:
Testing
Component:
Marionette Client and Harness
Platform:
All
Windows
Type:
defect

Tracking

(firefox-esr60 wontfix, firefox56 wontfix, firefox57 wontfix, firefox58 wontfix)

Status:
RESOLVED INCOMPLETE
Tracking Flags:
Tracking Status
firefox-esr60 --- wontfix
firefox56 --- wontfix
firefox57 --- wontfix
firefox58 --- wontfix

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell unknown]) 

     INFO -  Traceback (most recent call last):

 13:02:49     INFO -    File "C:\slave\test\build\venv\lib\site-packages\marionette_harness\marionette_test\testcases.py", line 197, in run

 13:02:49     INFO -      self.tearDown()

 13:02:49     INFO -    File "C:\slave\test\build\tests\marionette\tests\testing\marionette\harness\marionette_harness\tests\unit\test_crash.py", line 150, in tearDown

13:02:49 INFO - self.assertEqual(self.marionette.crashed, 1) 

And some lines below:

 13:02:49     INFO -  mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/Sytx6rygSHumWRib9E5HBA/artifacts/public/build/firefox-53.0a1.en-US.win64.crashreporter-symbols.zip

 13:02:55     INFO -  mozcrash Copy/paste: C:\slave\test\build\win32-minidump_stackwalk.exe c:\users\cltbld~1.t-w\appdata\local\temp\tmp61oft6.mozrunner\minidumps\b9b2b714-8f4d-43a0-a2af-effcb4660d09.dmp c:\users\cltbld~1.t-w\appdata\local\temp\tmpdjzsuy

 13:02:55     INFO -  mozcrash Saved minidump as C:\slave\test\build\blobber_upload_dir\b9b2b714-8f4d-43a0-a2af-effcb4660d09.dmp

 13:02:55    ERROR -  PROCESS-CRASH | test_crash.py TestCrashInTearDown.test_crash_in_teardown | application crashed [None]

13:02:55 INFO - Crash dump filename: c:\users\cltbld~1.t-w\appdata\local\temp\tmp61oft6.mozrunner\minidumps\b9b2b714-8f4d-43a0-a2af-effcb4660d09.dmp 

So whether we missed to monkey patch mozcrash in this case, or there was a real crash of Firefox.
(In reply to Henrik Skupin (:whimboo) from comment #1)
> So whether we missed to monkey patch mozcrash in this case, or there was a
> real crash of Firefox.

The former is the problem here and it will be fixed with my patch on bug 1223277.
Depends on: 1223277
The underlying issue here should be bug 1376795.
Depends on: 1376795
The landed workaround didn't seem to help for this test. So we have to wait until we are clear about the real underlying issue.
is there any work being done to get to the bottom of this?
Flags: needinfo?(hskupin)
So all the last days these failure seem to happen nearly exclusively for headless mode. There are only rare situations for Mn jobs. I tried to reproduce it locally but wasn't successful at all. Maybe it's specific 10.10 related? I have 10.12 installed.

I will pick one of the affected builds. Maybe it will help to get it reproduced.
status-firefox56: --- → affected
status-firefox57: --- → affected
OS: Unspecified → Mac OS X
Hardware: Unspecified → All
I cannot reproduce it locally whatever I try to do. So I'm figuring out now how to use MacOS one click loaners, if possible yet.
Flags: needinfo?(hskupin)
no traction in bug 1395504, I pinged again there
Whiteboard: [stockwell unknown]
The signature has been changed. Updating summary.
Summary: Intermittent test_crash.py TestCrashInTearDown.test_crash_in_teardown | AssertionError: 0 != 1 → Intermittent test_crash.py TestCrashInTearDown.test_crash_in_teardown | AssertionError: "Process crashed" does not match "Process killed because the connection to Marionette server is lost
Priority: -- → P3
Version: Version 3 → Trunk
There have been 31 failures in the last 7 days.

Failures occur on Windows 7 / pgo with 2 exceptions for  windows10-64 /pgo and windows7-32-nightly / opt.

Summary: test_crash.py TestCrashInTearDown.test_crash_in_teardown | AssertionError: "Process crashed" does not match "Process killed because the connection to Marionette server is lost. Check gecko.log for errors (Reason: Connection timed out after 10s)"

Here is a recent log file:
https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=175959315&lineNumber=36821

:whimboo I see you've done some work on this, could you by any chance take another look?
Flags: needinfo?(hskupin)
Whiteboard: [stockwell unknown] → [stockwell needswork]
This hang might actually be related to bug 1433873 because we hang after the restart. The OS X failures will be gone soon.
Depends on: 1433873
Flags: needinfo?(hskupin)
OS: Mac OS X → Windows
This failure didn't appear again after September 25th:

https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2018-06-17&endday=2018-10-15&tree=all&bug=1326124
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WORKSFORME
New occurrences: https://treeherder.mozilla.org/logviewer.html#?job_id=211188507&repo=mozilla-central&lineNumber=40702
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---

Looks like no more failures of this test on Windows since mid of July. As such I would say we just mark this bug as WFM given that I don't know what could have fixed it.

Status: REOPENED → RESOLVED
Closed: 6 years ago → 5 years ago
status-firefox56: affected → wontfix
status-firefox57: affected → wontfix
status-firefox58: affected → wontfix
status-firefox-esr60: --- → affected
Resolution: --- → WORKSFORME

Happened again based on bug 1584868.

Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Priority: P3 → P5

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: REOPENED → RESOLVED
Closed: 5 years ago → 3 years ago
Resolution: --- → INCOMPLETE

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=371083253&repo=try

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: REOPENED → RESOLVED
Closed: 3 years ago → 3 years ago
Resolution: --- → INCOMPLETE

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=374242175&repo=mozilla-central

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: REOPENED → RESOLVED
Closed: 3 years ago → 2 years ago
Resolution: --- → INCOMPLETE

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=391576097&repo=try

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Severity: normal → S3
Product: Testing → Remote Protocol
Moving bug to Testing::Marionette Client and Harness component per bug 1815831.
Component: Marionette → Marionette Client and Harness
Product: Remote Protocol → Testing

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Status: REOPENED → RESOLVED
Closed: 2 years ago → 1 year ago
status-firefox-esr60: affected → wontfix
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.