1536221 - windows/aarch64 - issues with crashreporter test in xpcshell

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect)

Description

[Edwin Takahashi (:egao | infrequent contributor)]

Try: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=retry%2Ctestfailed%2Cbusted%2Cexception%2Csuccess%2Crunning%2Cpending%2Crunnable&group_state=expanded&revision=cfd502758634e1b8bbb205e53028fff355c8fbbd&selectedJob=233750044

Log: https://queue.taskcluster.net/v1/task/YEPev-ibRuyjDxZgsRKSBg/runs/0/artifacts/public/logs/live_backing.log

Test: xpcshell-8

Platform: windows10-aarch64

Context:
01:12:24 INFO - TEST-START | toolkit/crashreporter/test/unit/test_crash_after_js_large_allocation_failure_reporting.js
01:12:26 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_after_js_large_allocation_failure_reporting.js | xpcshell return code: 0
01:12:26 INFO - TEST-INFO took 1274ms
01:12:26 INFO - >>>>>>>
01:12:26 INFO - PID 11216 | Unable to load \untrusted-startup-test-dll.dll; LoadLibraryW failed: 126
01:12:26 INFO - (xpcshell/head.js) | test MAIN run_test pending (1)
01:12:26 INFO - PID 11216 | Unable to load \untrusted-startup-test-dll.dll; LoadLibraryW failed: 126
01:12:26 INFO - No minidump found!
01:12:26 INFO - C:/tasks/task_1552525365/build/tests/xpcshell/tests/toolkit/crashreporter/test/unit/head_crashreporter.js:handleMinidump:131
01:12:26 INFO - C:/tasks/task_1552525365/build/tests/xpcshell/tests/toolkit/crashreporter/test/unit/head_crashreporter.js:do_crash:87
01:12:26 INFO - C:/tasks/task_1552525365/build/tests/xpcshell/tests/toolkit/crashreporter/test/unit/test_crash_after_js_large_allocation_failure_reporting.js:run_test:7
01:12:26 INFO - C:\tasks\task_1552525365\build\tests\xpcshell\head.js:_execute_test:520
01:12:26 INFO - -e:null:1
01:12:26 INFO - exiting test
01:12:26 INFO - <<<<<<<
01:12:26 INFO - TEST-START | toolkit/crashreporter/test/unit/test_crash_oom.js
01:12:27 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_oom.js | xpcshell return code: 0
01:12:27 INFO - TEST-INFO took 1309ms
01:12:27 INFO - >>>>>>>
01:12:27 INFO - PID 9860 | Unable to load \untrusted-startup-test-dll.dll; LoadLibraryW failed: 126
01:12:27 INFO - (xpcshell/head.js) | test MAIN run_test pending (1)
01:12:27 INFO - PID 9860 | Unable to load \untrusted-startup-test-dll.dll; LoadLibraryW failed: 126out of memory: 0xFFFFFFFFFFFFFFFF bytes requested
01:12:27 INFO - No minidump found!
01:12:27 INFO - C:/tasks/task_1552525365/build/tests/xpcshell/tests/toolkit/crashreporter/test/unit/head_crashreporter.js:handleMinidump:131
01:12:27 INFO - C:/tasks/task_1552525365/build/tests/xpcshell/tests/toolkit/crashreporter/test/unit/head_crashreporter.js:do_crash:87
01:12:27 INFO - C:/tasks/task_1552525365/build/tests/xpcshell/tests/toolkit/crashreporter/test/unit/test_crash_oom.js:run_test:7
01:12:27 INFO - C:\tasks\task_1552525365\build\tests\xpcshell\head.js:_execute_test:520
01:12:27 INFO - -e:null:1
01:12:27 INFO - exiting test
01:12:27 INFO - <<<<<<<

Similar logs outputs repeat for:

TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_after_js_large_allocation_failure_reporting.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_oom.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_moz_crash.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_after_js_oom_reported.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_uncaught_exception.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_after_js_large_allocation_failure.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_after_js_oom_recovered.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_oom_annotation_windows.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_alloc_small.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_unknown_op.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_alloc_large.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_after_js_oom_reported_2.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_save_xmm128.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crashreporter_crash.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_with_memory_report.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_event_files.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crashreporter_appmem.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_override_exception_handler.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_save_nonvol.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_save_nonvol_far.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_thread_annotation.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_epilog.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_infinite_entry_chain.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_infinite_code_chain.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_save_xmm128_far.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_invalid_exception_rva.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_not_a_pe.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_oom_annotation_windows.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_oom_annotation_windows.js | handleCrash - TypeError: getMinidump(...) is null at ../unit/head_crashreporter.js:202
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_exception_time_annotation.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_exception_time_annotation.js | handleCrash - TypeError: getMinidump(...) is null at ../unit/head_crashreporter.js:202
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_annotation.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_annotation.js | handleCrash - TypeError: getMinidump(...) is null at ../unit/head_crashreporter.js:202
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_memory_list.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_memory_list.js | handleCrash - TypeError: getMinidump(...) is null at ../unit/head_crashreporter.js:202
TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_win64cfi_push_nonvol.js | xpcshell return code: 0

Comment 1

[Edwin Takahashi (:egao | infrequent contributor)]

:erahm - early in the windows10-aarch64 test runs an issue with crashreporter was identified and resolved in 1526276, and :dmajor did some work there. However we're still seeing tests from crashreporter manifest files fail when run in xpcshell.

Comment 2

[Eric Rahm [:erahm]]

(In reply to Edwin Gao (:egao) from comment #1)

:erahm - early in the windows10-aarch64 test runs an issue with crashreporter was identified and resolved in 1526276, and :dmajor did some work there. However we're still seeing tests from crashreporter manifest files fail when run in xpcshell.

Gabriele, can you take a look at this? Bug 1534858, comment 12 seems to imply minidumpwriter isn't working on WoA.

Comment 3

[Gabriele Svelto [:gsvelto]]

(In reply to Eric Rahm [:erahm] from comment #2)

Gabriele, can you take a look at this? Bug 1534858, comment 12 seems to imply minidumpwriter isn't working on WoA.

Yes, though I don't think that's the issue here. Taking this.

Comment 4

[Gabriele Svelto [:gsvelto]]

I'm slowly setting up my ARM laptop to run tests locally. The win64cfi tests are expected to fail on ARM64 because they're specific to x86-64, we'll need to write separate CFI tests for AArch64 once we have support for it. All the other tests should run and I'll re-enable them as soon as I figure out what's wrong here.

Comment 5

[Edwin Takahashi (:egao | infrequent contributor)]

:gsvelto - not to add to your plate when it's full - you noted in bug 1539312 that none of the crashreporter tests are working for windows10-aarch64. Is it correct to assume this is the reason behind the series of failures complaining about crash dump IDs that cropped up recently in https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=f1f517212728922eb0d7aaab8b0ee1c2e3403ca6?

mochitest-browser-chrome-3: TEST-UNEXPECTED-FAIL | browser/base/content/test/general/browser_restore_isAppTab.js | Uncaught exception - dumpID was not present despite crash reporting being enabled

mochitest-chrome-2: TEST-UNEXPECTED-FAIL | dom/ipc/tests/test_process_error.xul | dumpID is present and not an empty string

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I also see:
toolkit/content/tests/browser/browser_content_url_annotation.js
toolkit/content/tests/browser/browser_crash_previous_frameloader.js
browser/components/extensions/test/browser/test-oop-extensions/browser_ext_port_disconnect_on_crash.js
toolkit/components/thumbnails/test/browser_thumbnails_bg_crash_during_capture.js
toolkit/components/thumbnails/test/browser_thumbnails_bg_crash_while_idle.js
browser/base/content/test/tabcrashed/*
browser/base/content/test/plugins/browser_pluginCrashCommentAndURL.js
browser/base/content/test/plugins/browser_pluginCrashReportNonDeterminism.js
netwerk/test/browser/browser_child_resource.js
browser/components/sessionstore/test/browser_async_flushes.js
browser/components/sessionstore/test/browser_background_tab_crash.js
browser/components/sessionstore/test/browser_crashedTabs.js
browser/components/sessionstore/test/browser_revive_crashed_bg_tabs.js
browser/components/sessionstore/test/browser_tabicon_after_bg_tab_crash.js
browser/components/sessionstore/test/browser_unrestored_crashedTabs.js
dom/plugins/test/mochitest/test_bug1028200-*.html (possibly should be solve with bug 1538785)
dom/plugins/test/mochitest/test_crashing.html (possibly should be solve with bug 1538785)

from comment 5:
browser/base/content/test/general/browser_restore_isAppTab.js (manifest doesn't specify skip-if=!crashreporter)

I recommend we hack mozinfo to say crashreporter = False and all these tests will be skipped (except for browser_restore_isAppTab.js which should be considered dependent on crashreporter?)

Comment 7

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1536221 - toggle crashreporter value in mozinfo.json to False for windows10-aarch64

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

I would also make:
browser/base/content/test/general/browser_restore_isAppTab.js

have a skip-if = !crashreporter

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/74e9627a9c05
toggle crashreporter value in mozinfo.json to False for windows10-aarch64 r=jmaher

Comment 10

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/74e9627a9c05

Comment 11

[Gabriele Svelto [:gsvelto]]

Still trying to debug the tests locally, not much success yet though. I've got them to run - kinda - by copying part of a local build, the sources and a bunch of tools that need to run via emulation on my ARM laptop but it's proving challenging to coerce this mishmash of stuff into something that can be properly debugged. Leaving the NI? for now.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Note that this bug hasn't been fixed yet! As such reopening it...

Comment 13

[Gabriele Svelto [:gsvelto]]

There seem to be a problem with the exception handler on AArch64. I originally thought the issue was test-specific but it's not. I'm running a regression to identify when this started.

Comment 14

[Gabriele Svelto [:gsvelto]]

Now that bug 1542827 is fixed I can try re-enabling those tests. I'm setting them up on my ARM laptop to check if they all work correctly.

Comment 15

[Edwin Takahashi (:egao | infrequent contributor)]

For how to set up test runs on Try server, please refer to https://wiki.mozilla.org/CI_Automation/windows10_aarch64#Run_tests_on_Try.

Comment 16

[Gabriele Svelto [:gsvelto]]

I've tried re-enabling the tests but most still fail. The issues seems to be similar to the other crash reporter issues we've seen in Firefox itself but it's unclear why the same problems happen in xpcshell too. I'll probably manage to figure it out while in Whistler. This is the current status with all the tests re-enabled:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c1feaf9e6224c45f73c41c0053f88e229cc356a7

Comment 17

[Gabriele Svelto [:gsvelto]]

The issue seems to be that ffi_call_SYSV doesn't have unwinding information and since we use it to call the test crasher DLL in xpcshell we never manage to unwind out of the crashing point. Big thanks to :dmajor for figuring this out.

Comment 18

[Gabriele Svelto [:gsvelto]]

Attachment: Bug 1536221 - Re-enable mochitests and xpcshell tests that depend on the crash reporter on Windows/AArch64

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/097eedc2b14f
Re-enable mochitests and xpcshell tests that depend on the crash reporter on Windows/AArch64 r=jmaher

Comment 20

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/097eedc2b14f