Closed Bug 1536221 Opened 8 months ago Closed 5 months ago

windows/aarch64 - issues with crashreporter test in xpcshell

Categories

(Toolkit :: Crash Reporting, defect)

defect
Not set

Tracking

()

RESOLVED FIXED
mozilla69
Tracking Status
firefox69 --- fixed

People

(Reporter: egao, Assigned: gsvelto)

References

Details

Attachments

(2 files)

Try: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=retry%2Ctestfailed%2Cbusted%2Cexception%2Csuccess%2Crunning%2Cpending%2Crunnable&group_state=expanded&revision=cfd502758634e1b8bbb205e53028fff355c8fbbd&selectedJob=233750044

Log: https://queue.taskcluster.net/v1/task/YEPev-ibRuyjDxZgsRKSBg/runs/0/artifacts/public/logs/live_backing.log

Test: xpcshell-8

Platform: windows10-aarch64

Context:
01:12:24 INFO - TEST-START | toolkit/crashreporter/test/unit/test_crash_after_js_large_allocation_failure_reporting.js
01:12:26 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_after_js_large_allocation_failure_reporting.js | xpcshell return code: 0
01:12:26 INFO - TEST-INFO took 1274ms
01:12:26 INFO - >>>>>>>
01:12:26 INFO - PID 11216 | Unable to load \untrusted-startup-test-dll.dll; LoadLibraryW failed: 126
01:12:26 INFO - (xpcshell/head.js) | test MAIN run_test pending (1)
01:12:26 INFO - PID 11216 | Unable to load \untrusted-startup-test-dll.dll; LoadLibraryW failed: 126
01:12:26 INFO - No minidump found!
01:12:26 INFO - C:/tasks/task_1552525365/build/tests/xpcshell/tests/toolkit/crashreporter/test/unit/head_crashreporter.js:handleMinidump:131
01:12:26 INFO - C:/tasks/task_1552525365/build/tests/xpcshell/tests/toolkit/crashreporter/test/unit/head_crashreporter.js:do_crash:87
01:12:26 INFO - C:/tasks/task_1552525365/build/tests/xpcshell/tests/toolkit/crashreporter/test/unit/test_crash_after_js_large_allocation_failure_reporting.js:run_test:7
01:12:26 INFO - C:\tasks\task_1552525365\build\tests\xpcshell\head.js:_execute_test:520
01:12:26 INFO - -e:null:1
01:12:26 INFO - exiting test
01:12:26 INFO - <<<<<<<
01:12:26 INFO - TEST-START | toolkit/crashreporter/test/unit/test_crash_oom.js
01:12:27 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_oom.js | xpcshell return code: 0
01:12:27 INFO - TEST-INFO took 1309ms
01:12:27 INFO - >>>>>>>
01:12:27 INFO - PID 9860 | Unable to load \untrusted-startup-test-dll.dll; LoadLibraryW failed: 126
01:12:27 INFO - (xpcshell/head.js) | test MAIN run_test pending (1)
01:12:27 INFO - PID 9860 | Unable to load \untrusted-startup-test-dll.dll; LoadLibraryW failed: 126out of memory: 0xFFFFFFFFFFFFFFFF bytes requested
01:12:27 INFO - No minidump found!
01:12:27 INFO - C:/tasks/task_1552525365/build/tests/xpcshell/tests/toolkit/crashreporter/test/unit/head_crashreporter.js:handleMinidump:131
01:12:27 INFO - C:/tasks/task_1552525365/build/tests/xpcshell/tests/toolkit/crashreporter/test/unit/head_crashreporter.js:do_crash:87
01:12:27 INFO - C:/tasks/task_1552525365/build/tests/xpcshell/tests/toolkit/crashreporter/test/unit/test_crash_oom.js:run_test:7
01:12:27 INFO - C:\tasks\task_1552525365\build\tests\xpcshell\head.js:_execute_test:520
01:12:27 INFO - -e:null:1
01:12:27 INFO - exiting test
01:12:27 INFO - <<<<<<<

Similar logs outputs repeat for:

TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_after_js_large_allocation_failure_reporting.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_oom.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_moz_crash.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_after_js_oom_reported.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_uncaught_exception.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_after_js_large_allocation_failure.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_after_js_oom_recovered.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_oom_annotation_windows.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_alloc_small.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_unknown_op.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_alloc_large.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_after_js_oom_reported_2.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_save_xmm128.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crashreporter_crash.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_with_memory_report.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_event_files.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crashreporter_appmem.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_override_exception_handler.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_save_nonvol.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_save_nonvol_far.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_thread_annotation.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_epilog.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_infinite_entry_chain.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_infinite_code_chain.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_save_xmm128_far.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_invalid_exception_rva.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_not_a_pe.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_oom_annotation_windows.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_oom_annotation_windows.js | handleCrash - TypeError: getMinidump(...) is null at ../unit/head_crashreporter.js:202
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_exception_time_annotation.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_exception_time_annotation.js | handleCrash - TypeError: getMinidump(...) is null at ../unit/head_crashreporter.js:202
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_annotation.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_annotation.js | handleCrash - TypeError: getMinidump(...) is null at ../unit/head_crashreporter.js:202
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_memory_list.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_memory_list.js | handleCrash - TypeError: getMinidump(...) is null at ../unit/head_crashreporter.js:202
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_push_nonvol.js | xpcshell return code: 0

:erahm - early in the windows10-aarch64 test runs an issue with crashreporter was identified and resolved in 1526276, and :dmajor did some work there. However we're still seeing tests from crashreporter manifest files fail when run in xpcshell.

Flags: needinfo?(erahm)
See Also: → 1534858

(In reply to Edwin Gao (:egao) from comment #1)

:erahm - early in the windows10-aarch64 test runs an issue with crashreporter was identified and resolved in 1526276, and :dmajor did some work there. However we're still seeing tests from crashreporter manifest files fail when run in xpcshell.

Gabriele, can you take a look at this? Bug 1534858, comment 12 seems to imply minidumpwriter isn't working on WoA.

Flags: needinfo?(erahm) → needinfo?(gsvelto)

(In reply to Eric Rahm [:erahm] from comment #2)

Gabriele, can you take a look at this? Bug 1534858, comment 12 seems to imply minidumpwriter isn't working on WoA.

Yes, though I don't think that's the issue here. Taking this.

Assignee: nobody → gsvelto
Status: NEW → ASSIGNED
Flags: needinfo?(gsvelto)

I'm slowly setting up my ARM laptop to run tests locally. The win64cfi tests are expected to fail on ARM64 because they're specific to x86-64, we'll need to write separate CFI tests for AArch64 once we have support for it. All the other tests should run and I'll re-enable them as soon as I figure out what's wrong here.

:gsvelto - not to add to your plate when it's full - you noted in bug 1539312 that none of the crashreporter tests are working for windows10-aarch64. Is it correct to assume this is the reason behind the series of failures complaining about crash dump IDs that cropped up recently in https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=f1f517212728922eb0d7aaab8b0ee1c2e3403ca6?

mochitest-browser-chrome-3: TEST-UNEXPECTED-FAIL | browser/base/content/test/general/browser_restore_isAppTab.js | Uncaught exception - dumpID was not present despite crash reporting being enabled

mochitest-chrome-2: TEST-UNEXPECTED-FAIL | dom/ipc/tests/test_process_error.xul | dumpID is present and not an empty string

Flags: needinfo?(gsvelto)

I also see:
toolkit/content/tests/browser/browser_content_url_annotation.js
toolkit/content/tests/browser/browser_crash_previous_frameloader.js
browser/components/extensions/test/browser/test-oop-extensions/browser_ext_port_disconnect_on_crash.js
toolkit/components/thumbnails/test/browser_thumbnails_bg_crash_during_capture.js
toolkit/components/thumbnails/test/browser_thumbnails_bg_crash_while_idle.js
browser/base/content/test/tabcrashed/*
browser/base/content/test/plugins/browser_pluginCrashCommentAndURL.js
browser/base/content/test/plugins/browser_pluginCrashReportNonDeterminism.js
netwerk/test/browser/browser_child_resource.js
browser/components/sessionstore/test/browser_async_flushes.js
browser/components/sessionstore/test/browser_background_tab_crash.js
browser/components/sessionstore/test/browser_crashedTabs.js
browser/components/sessionstore/test/browser_revive_crashed_bg_tabs.js
browser/components/sessionstore/test/browser_tabicon_after_bg_tab_crash.js
browser/components/sessionstore/test/browser_unrestored_crashedTabs.js
dom/plugins/test/mochitest/test_bug1028200-*.html (possibly should be solve with bug 1538785)
dom/plugins/test/mochitest/test_crashing.html (possibly should be solve with bug 1538785)

from comment 5:
browser/base/content/test/general/browser_restore_isAppTab.js (manifest doesn't specify skip-if=!crashreporter)

I recommend we hack mozinfo to say crashreporter = False and all these tests will be skipped (except for browser_restore_isAppTab.js which should be considered dependent on crashreporter?)

Flags: needinfo?(egao)

I would also make:
browser/base/content/test/general/browser_restore_isAppTab.js

have a skip-if = !crashreporter

Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/74e9627a9c05
toggle crashreporter value in mozinfo.json to False for windows10-aarch64 r=jmaher
No longer blocks: 1539312
Status: ASSIGNED → RESOLVED
Closed: 8 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68

Still trying to debug the tests locally, not much success yet though. I've got them to run - kinda - by copying part of a local build, the sources and a bunch of tools that need to run via emulation on my ARM laptop but it's proving challenging to coerce this mishmash of stuff into something that can be properly debugged. Leaving the NI? for now.

Note that this bug hasn't been fixed yet! As such reopening it...

Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Target Milestone: mozilla68 → ---
Flags: needinfo?(egao)

There seem to be a problem with the exception handler on AArch64. I originally thought the issue was test-specific but it's not. I'm running a regression to identify when this started.

Flags: needinfo?(gsvelto)

Now that bug 1542827 is fixed I can try re-enabling those tests. I'm setting them up on my ARM laptop to check if they all work correctly.

Depends on: 1542827

For how to set up test runs on Try server, please refer to https://wiki.mozilla.org/CI_Automation/windows10_aarch64#Run_tests_on_Try.

I've tried re-enabling the tests but most still fail. The issues seems to be similar to the other crash reporter issues we've seen in Firefox itself but it's unclear why the same problems happen in xpcshell too. I'll probably manage to figure it out while in Whistler. This is the current status with all the tests re-enabled:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c1feaf9e6224c45f73c41c0053f88e229cc356a7

The issue seems to be that ffi_call_SYSV doesn't have unwinding information and since we use it to call the test crasher DLL in xpcshell we never manage to unwind out of the crashing point. Big thanks to :dmajor for figuring this out.

Depends on: 1561088
Pushed by gsvelto@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/097eedc2b14f
Re-enable mochitests and xpcshell tests that depend on the crash reporter on Windows/AArch64 r=jmaher
Status: REOPENED → RESOLVED
Closed: 8 months ago5 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla69
You need to log in before you can comment on or make changes to this bug.