1229765 - Monitor Marionette browser process for start-up crashes

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Andreas Tolfsen ❲:ato❳]

The Marionette test runner (somewhat oddly named marionette-client) does not monitor the browser process it spawns.  When the browser crashes we would like to detect this and possibly collect a crash report, so that cases like bug 1229549 will not occur again.

Currently Marionette produces this when the browser crashes:

    AssertionError: Timed out waiting for port!

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Well, the crash analysis is part of mozcrash and would need symbols to actually process the crash. If those are not provided mozcrash cannot do anything.

One example of such a crash as what we see in one of the newly added dependency bugs is:
https://treeherder.allizom.org/logviewer.html#?job_id=2408059&repo=mozilla-central

Here we still fail in getting the symbols but as you can see mozcrash does not print the PROCESSCRASH lines or such. I get the feeling that this is not a problem by Marionette but mozcrash.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Crash processing works perfectly for us in firefox-ui-tests since today after the right symbol url (bug 1228644) for installer builds on Windows is used when triggering the mozharness script:

https://treeherder.allizom.org/logviewer.html#?job_id=2444766&repo=mozilla-central

 08:42:35     INFO - PROCESS-CRASH | runner.py | application crashed [None]
 08:42:35     INFO - Crash dump filename: c:\jenkins\workspace\mozilla-central_update\build\tmps7o6ku.mozrunner-1449247284\minidumps\ffc0b75d-f14f-477e-880c-bf8fdf14561f.dmp

So I wonder which jobs are affected.

Comment 3

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Looking at the media-test job pointed out in bug 1229549 (https://treeherder.mozilla.org/logviewer.html#?job_id=14306272&repo=try) - the same Marionette error manifests itself. If I understand correctly, the issue is specific to start-up crashes. (Those same media-test jobs have processed other crashes correctly in the past. As far as I can tell, MINIDUMP_STACKWALK is set correctly, as is the --symbols-url option.)

Comment 4

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

As per conversation with :ted, I will investigate adding some logic to the Marionette runner to look for a minidump file when waiting for a port times out.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Does minidump stackwalk error out? I have seen some startup crashes for our update tests which have invalid minidump files. See bug 474863 for the underlying isuse.

Comment 6

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

I've modified the Marionette class to check for crashes whenever we hit a timeout while waiting for a port. Works locally with a crashy Firefox Desktop. Let's see how that fares on try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=f0a22e5771cd

Comment 7

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Attachment: MozReview Request: Bug 1229765 - Monitor Marionette browser process for start-up crashes; r?automatedtester

Review commit: https://reviewboard.mozilla.org/r/28293/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/28293/

Comment 8

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Note that the Mnw job [1] in the try run doesn't log PROCESS-CRASH. I think that may be because the emulator doesn't even manage to start up in the first place before we hit a timeout, which is a known problem. [2] 

This leads me to our conversation about checking for a crash sooner by trying to communicate with the process right after it's started: I think it might actually be better to wait until the port timeout, as in the current patch, so that we don't check too early to catch a start-up crash in the first place. Technically a start-up crash is defined as any crash within the first 60 seconds or so, right? What do you think, David?

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=f0a22e5771cd&exclusion_profile=false&selectedJob=14708759
[2] logs.glob.uno/?c=mozilla%23ateam&s=26+Nov+2015&e=26+Nov+2015&h=emulator+is+slow#c1000252

Comment 9

[David Burns :automatedtester]

Let's leave it for 60s then.

Comment 10

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

(In reply to Maja Frydrychowicz (:maja_zf) from comment #6)
> I've modified the Marionette class to check for crashes whenever we hit a
> timeout while waiting for a port. Works locally with a crashy Firefox
> Desktop. Let's see how that fares on try:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=f0a22e5771cd

Now a try run with a non-start-up-crashy build: https://treeherder.mozilla.org/#/jobs?repo=try&revision=120ad4bb6d8d

Comment 11

[David Burns :automatedtester]

Comment on attachment 8699599 [details]
MozReview Request: Bug 1229765 - Monitor Marionette browser process for start-up crashes; r?automatedtester

https://reviewboard.mozilla.org/r/28293/#review25583

Comment 12

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/86b32f1e4aa1

Comment 13

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/86b32f1e4aa1