Open Bug 1827125 Opened 2 years ago Updated 14 hours ago

Intermittent | application crashed [None] (Error reading dump: Missing minidump header)

Categories

(Toolkit :: Crash Reporting, defect, P5)

defect

Tracking

()

REOPENED

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Keywords: intermittent-failure)

Filed by: smolnar [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=411779657&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LoiblsCvRxiRivyQ2G-luQ/runs/0/artifacts/public/logs/live_backing.log


INFO - PID 5520 | setupTelemetry/this._delayedInitTask<@resource://gre/modules/TelemetryControllerParent.sys.mjs:824:36
[task 2023-04-09T03:47:32.931Z] 03:47:32     INFO - PID 5520 | observe@resource://gre/modules/AsyncShutdown.sys.mjs:576:16
[task 2023-04-09T03:47:32.932Z] 03:47:32     INFO - PID 5520 | JavaScript error: resource://gre/modules/AsyncShutdown.sys.mjs, line 727: Error: Phase "profile-before-change" is finished, it is too late to register completion condition "OS.File: flush I/O queued before profileBeforeChange"
[task 2023-04-09T03:47:32.933Z] 03:47:32     INFO - PID 5520 | JavaScript error: resource://gre/modules/AsyncShutdown.sys.mjs, line 727: Error: Phase "profile-before-change" is finished, it is too late to register completion condition "OS.File: flush I/O queued before profileBeforeChange"
[task 2023-04-09T03:47:32.934Z] 03:47:32     INFO - PID 5520 | JavaScript error: resource://gre/modules/AsyncShutdown.sys.mjs, line 727: Error: Phase "profile-before-change" is finished, it is too late to register completion condition "OS.File: flush I/O queued before profileBeforeChange"
[task 2023-04-09T03:47:32.935Z] 03:47:32     INFO - PID 5520 | JavaScript error: resource://gre/modules/AsyncShutdown.sys.mjs, line 727: Error: Phase "profile-before-change" is finished, it is too late to register completion condition "OS.File: flush I/O queued before profileBeforeChange"
[task 2023-04-09T03:47:32.936Z] 03:47:32     INFO - PID 5520 | JavaScript error: resource://gre/modules/AsyncShutdown.sys.mjs, line 727: Error: Phase "profile-before-change" is finished, it is too late to register completion condition "OS.File: flush I/O queued before profileBeforeChange"
[task 2023-04-09T03:47:36.440Z] 03:47:36     INFO - PID 5520 | [Parent 3996, IPC I/O Parent] WARNING: Process 6664 may be hanging at shutdown; will wait for up to 80000ms: file Z:/task_168100472682872/build/src/ipc/chromium/src/chrome/common/process_watcher_win.cc:129
[task 2023-04-09T03:48:11.246Z] 03:48:11     INFO - mozcrash Downloading symbols from: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/WYfX_TbWTkqZXFHNIQP_eA/artifacts/public/build/target.crashreporter-symbols.zip
[task 2023-04-09T03:48:21.293Z] 03:48:21     INFO - PID 5520 | [Parent 3996, IPC I/O Parent] WARNING: Process 5736 may be hanging at shutdown; will wait for up to 35141ms: file Z:/task_168100472682872/build/src/ipc/chromium/src/chrome/common/process_watcher_win.cc:129
[task 2023-04-09T03:48:32.312Z] 03:48:32     INFO - PID 5520 | 1681012112313	geckodriver::browser	DEBUG	Browser process stopped: exit code: 0
[task 2023-04-09T03:48:32.324Z] 03:48:32     INFO - PID 5520 | 1681012112323	webdriver::server	DEBUG	<- 200 OK {"value":null}
[task 2023-04-09T03:48:32.797Z] 03:48:32     INFO - mozcrash Copy/paste: Z:\task_168100931436629\fetches\minidump-stackwalk\minidump-stackwalk.exe --symbols-url=https://symbols.mozilla.org/ --cyborg=C:\Users\task_168100931436629\AppData\Local\Temp\tmplce3mslr\a76be10a-139d-4b1c-a586-6b0677e12622.trace C:\Users\task_168100931436629\AppData\Local\Temp\tmpgyqnvc9x.mozrunner\minidumps\a76be10a-139d-4b1c-a586-6b0677e12622.dmp C:\Users\task_168100931436629\AppData\Local\Temp\tmprts4lxwb
[task 2023-04-09T03:48:32.906Z] 03:48:32     INFO - mozcrash Saved minidump as Z:\task_168100931436629\build\blobber_upload_dir\a76be10a-139d-4b1c-a586-6b0677e12622.dmp
[task 2023-04-09T03:48:32.916Z] 03:48:32     INFO - PROCESS-CRASH | /webdriver/tests/get_window_rect/get.py | application crashed [None]
[task 2023-04-09T03:48:32.916Z] 03:48:32     INFO - Crash dump filename: C:\Users\task_168100931436629\AppData\Local\Temp\tmpgyqnvc9x.mozrunner\minidumps\a76be10a-139d-4b1c-a586-6b0677e12622.dmp
[task 2023-04-09T03:48:32.916Z] 03:48:32     INFO - stderr from minidump-stackwalk:
[task 2023-04-09T03:48:32.916Z] 03:48:32     INFO - ERROR MissingHeader - Error reading dump: Missing minidump header (empty minidump?)
[task 2023-04-09T03:48:32.917Z] 03:48:32     INFO - mozcrash Copy/paste: Z:\task_168100931436629\fetches\minidump-stackwalk\minidump-stackwalk.exe --symbols-url=https://symbols.mozilla.org/ --cyborg=C:\Users\task_168100931436629\AppData\Local\Temp\tmpl8bolgj5\3a46d829-c6f1-4494-8da1-79446140384f.trace C:\Users\task_168100931436629\AppData\Local\Temp\tmpgyqnvc9x.mozrunner\minidumps\3a46d829-c6f1-4494-8da1-79446140384f.dmp C:\Users\task_168100931436629\AppData\Local\Temp\tmprts4lxwb
[task 2023-04-09T03:48:32.929Z] 03:48:32     INFO - mozcrash Saved minidump as Z:\task_168100931436629\build\blobber_upload_dir\3a46d829-c6f1-4494-8da1-79446140384f.dmp
[task 2023-04-09T03:48:32.932Z] 03:48:32     INFO - PROCESS-CRASH | /webdriver/tests/get_window_rect/get.py | application crashed [None]
[task 2023-04-09T03:48:32.932Z] 03:48:32     INFO - Crash dump filename: C:\Users\task_168100931436629\AppData\Local\Temp\tmpgyqnvc9x.mozrunner\minidumps\3a46d829-c6f1-4494-8da1-79446140384f.dmp
[task 2023-04-09T03:48:32.932Z] 03:48:32     INFO - stderr from minidump-stackwalk:
[task 2023-04-09T03:48:32.932Z] 03:48:32     INFO - ERROR MissingHeader - Error reading dump: Missing minidump header (empty minidump?)
[task 2023-04-09T03:48:32.933Z] 03:48:32     INFO - mozcrash Copy/paste: Z:\task_168100931436629\fetches\minidump-stackwalk\minidump-stackwalk.exe --symbols-url=https://symbols.mozilla.org/ --cyborg=C:\Users\task_168100931436629\AppData\Local\Temp\tmpkhq7hacp\062eaf55-d124-4f0d-a655-e1fd1ac44966.trace C:\Users\task_168100931436629\AppData\Local\Temp\tmpgyqnvc9x.mozrunner\minidumps\062eaf55-d124-4f0d-a655-e1fd1ac44966.dmp C:\Users\task_168100931436629\AppData\Local\Temp\tmprts4lxwb
[task 2023-04-09T03:48:32.944Z] 03:48:32     INFO - mozcrash Saved minidump as Z:\task_168100931436629\build\blobber_upload_dir\062eaf55-d124-4f0d-a655-e1fd1ac44966.dmp
[task 2023-04-09T03:48:32.946Z] 03:48:32     INFO - PROCESS-CRASH | /webdriver/tests/get_window_rect/get.py | application crashed [None]
[task 2023-04-09T03:48:32.946Z] 03:48:32     INFO - Crash dump filename: C:\Users\task_168100931436629\AppData\Local\Temp\tmpgyqnvc9x.mozrunner\minidumps\062eaf55-d124-4f0d-a655-e1fd1ac44966.dmp
[task 2023-04-09T03:48:32.946Z] 03:48:32     INFO - stderr from minidump-stackwalk:
[task 2023-04-09T03:48:32.946Z] 03:48:32     INFO - ERROR MissingHeader - Error reading dump: Missing minidump header (empty minidump?)
[task 2023-04-09T03:48:33.375Z] 03:48:33     INFO - Found a crash dump file; changing status to CRASH
[task 2023-04-09T03:48:33.387Z] 03:48:33     INFO - TEST-UNEXPECTED-CRASH | /webdriver/tests/get_window_rect/get.py | expected OK
[task 2023-04-09T03:48:33.387Z] 03:48:33     INFO - TEST-INFO took 77218ms
INFO - STDOUT: ======================== 3 passed in 75.84s (0:01:15) =========================
[task 2023-04-09T03:48:33.490Z] 03:48:33     INFO - Closing logging queue
[task 2023-04-09T03:48:33.490Z] 03:48:33     INFO - queue closed
[task 2023-04-09T03:48:34.000Z] 03:48:34     INFO - PID 8324 | 1681012113545	geckodriver	INFO	Listening on 127.0.0.1:51772
[task 2023-04-09T03:48:34.002Z] 03:48:34     INFO - Starting runner
[task 2023-04-09T03:48:35.782Z] 03:48:35     INFO - TEST-START | /webdriver/tests/get_window_rect/user_prompts.py

There is a problem with Error reading dump: Missing minidump header (empty minidump?). And yes all the created minidump files for content process crashes during a shutdown have a file size of 0.

Gabriele, are you aware of that and do we have a bug open?

Flags: needinfo?(gsvelto)

We sometimes run into issues generating minidumps but three in a row is a novelty and a bit worrying too. I need to investigate though there's not much in the log to go by.

Flags: needinfo?(gsvelto)

What I actually missed initially is that the related build is actually a CCOV one. Not sure if this might more easily get us into such a situation. Also the push to autoland was faulty and got backed out. Maybe it could be used to trigger it. Note that also two more Wd2 jobs were failing but only with timeouts (hangs) during shutdown.

Given that I can see more crashes where Firefox failed to generate the minidump headers lets make this a generic bug in the crash reporter component.

Component: Marionette → Crash Reporting
Product: Remote Protocol → Toolkit
Summary: Intermittent /webdriver/tests/get_window_rect/get.py | application crashed [None] → Intermittent | application crashed [None] (Error reading dump: Missing minidump header)

Gabriele, see also bug 1829572 which is a good case that shows that minidump generation could fail if we are close to an OOM. Hereby the first and third crash didn't cause such a file to be created but the second crash did.

Duplicate of this bug: 1830270
Duplicate of this bug: 1830295
Duplicate of this bug: 1831729
Duplicate of this bug: 1835448
Duplicate of this bug: 1837259
Duplicate of this bug: 1838303
Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

(In reply to Treeherder Bug Filer from comment #24)

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=429848672&repo=autoland

No, this is bug 1816119.

Status: REOPENED → RESOLVED
Closed: 1 year ago1 year ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

(In reply to Treeherder Bug Filer from comment #26)

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=431949988&repo=mozilla-beta

That is actually bug 1851376.

Status: REOPENED → RESOLVED
Closed: 1 year ago1 year ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 1 year ago5 months ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 5 months ago3 months ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
You need to log in before you can comment on or make changes to this bug.