1648326 - Permafailing tier 2 xpcom/string/crashtests/1113005.html (finished) | application crashed [@ mozilla::`anonymous namespace'::RunWatchdog(void*)]

Status: RESOLVED FIXED

(Core :: XPCOM, defect)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=307455781&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LZ625IG9T-KkPHq8lpjyRA/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LZ625IG9T-KkPHq8lpjyRA/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

[task 2020-06-25T00:50:23.718Z] 00:50:23 ERROR - TEST-UNEXPECTED-FAIL | xpcom/string/crashtests/1113005.html (finished) | application terminated with exit code 1
[task 2020-06-25T00:50:23.718Z] 00:50:23 INFO - REFTEST INFO | Downloading symbols from: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/CVM9M14iQ4yTAN3J64B5FQ/artifacts/public/build/target.crashreporter-symbols.zip
[task 2020-06-25T00:50:26.907Z] 00:50:26 INFO - REFTEST INFO | Copy/paste: C:/tasks/task_1593007903/fetches\minidump_stackwalk\minidump_stackwalk.exe c:\users\testdr1\appdata\local\temp\tmpkg7ta2.mozrunner\minidumps\63ea4ef5-fa64-48d9-898f-d0fc392932f2.dmp c:\users\testdr1\appdata\local\temp\tmpcfn4ce
[task 2020-06-25T00:50:35.748Z] 00:50:35 INFO - REFTEST INFO | Saved minidump as C:\tasks\task_1593007903\build\blobber_upload_dir\63ea4ef5-fa64-48d9-898f-d0fc392932f2.dmp
[task 2020-06-25T00:50:35.748Z] 00:50:35 INFO - REFTEST INFO | Saved app info as C:\tasks\task_1593007903\build\blobber_upload_dir\63ea4ef5-fa64-48d9-898f-d0fc392932f2.extra
[task 2020-06-25T00:50:35.964Z] 00:50:35 INFO - REFTEST PROCESS-CRASH | xpcom/string/crashtests/1113005.html (finished) | application crashed [@ mozilla::anonymous namespace'::RunWatchdog(void*)] [task 2020-06-25T00:50:35.964Z] 00:50:35 INFO - Mozilla crash reason: MOZ_CRASH(Shutdown too long, probably frozen, causing a crash.) [task 2020-06-25T00:50:35.964Z] 00:50:35 INFO - Crash dump filename: c:\users\testdr~1\appdata\local\temp\tmpkg7ta2.mozrunner\minidumps\63ea4ef5-fa64-48d9-898f-d0fc392932f2.dmp [task 2020-06-25T00:50:35.964Z] 00:50:35 INFO - Operating system: Windows NT [task 2020-06-25T00:50:35.964Z] 00:50:35 INFO - 10.0.17134 [task 2020-06-25T00:50:35.964Z] 00:50:35 INFO - CPU: arm64 [task 2020-06-25T00:50:35.964Z] 00:50:35 INFO - 8 CPUs [task 2020-06-25T00:50:35.964Z] 00:50:35 INFO - [task 2020-06-25T00:50:35.964Z] 00:50:35 INFO - GPU: UNKNOWN [task 2020-06-25T00:50:35.965Z] 00:50:35 INFO - [task 2020-06-25T00:50:35.965Z] 00:50:35 INFO - Crash reason: EXCEPTION_BREAKPOINT [task 2020-06-25T00:50:35.965Z] 00:50:35 INFO - Crash address: 0x32b4e720 [task 2020-06-25T00:50:35.965Z] 00:50:35 INFO - Process uptime: 784 seconds [task 2020-06-25T00:50:35.965Z] 00:50:35 INFO - [task 2020-06-25T00:50:35.965Z] 00:50:35 INFO - Thread 40 (crashed) [task 2020-06-25T00:50:35.966Z] 00:50:35 INFO - 0 xul.dll!mozilla::anonymous namespace'::RunWatchdog(void*) [nsTerminator.cpp:3add3a1747557d9b1e20c41afed20a94952ce5e8 : 0 + 0x4]
[task 2020-06-25T00:50:35.966Z] 00:50:35 INFO - x0 = 0x00000000000000d8 x1 = 0x00007ffe65207a90
[task 2020-06-25T00:50:35.966Z] 00:50:35 INFO - x2 = 0x0000000000000000 x3 = 0x00000210360f20b8
[task 2020-06-25T00:50:35.966Z] 00:50:35 INFO - x4 = 0x0000005d3f17f850 x5 = 0x0000021041970260
[task 2020-06-25T00:50:35.966Z] 00:50:35 INFO - x6 = 0x0000021041970270 x7 = 0x0000000000000001
[task 2020-06-25T00:50:35.966Z] 00:50:35 INFO - x8 = 0x00007ffe44e76090 x9 = 0x00007ffe35ad0fe0
[task 2020-06-25T00:50:35.967Z] 00:50:35 INFO - x10 = 0x0000021036326938 x11 = 0x00000000ffffffff
[task 2020-06-25T00:50:35.967Z] 00:50:35 INFO - x12 = 0x0000021035e40200 x13 = 0x0000000000000044
[task 2020-06-25T00:50:35.967Z] 00:50:35 INFO - x14 = 0x0000000000000001 x15 = 0x00007ffe62489680
[task 2020-06-25T00:50:35.967Z] 00:50:35 INFO - x16 = 0x0000799ca343895e x17 = 0x0000535cdc2d4645
[task 2020-06-25T00:50:35.967Z] 00:50:35 INFO - x18 = 0x0000000000000000 x19 = 0x00007ffe625fb240
[task 2020-06-25T00:50:35.967Z] 00:50:35 INFO - x20 = 0x000000000000003f x21 = 0x00007ffe35f4fe24
[task 2020-06-25T00:50:35.967Z] 00:50:35 INFO - x22 = 0x00007ffe394b9000 x23 = 0x00000210480b8538
[task 2020-06-25T00:50:35.967Z] 00:50:35 INFO - x24 = 0x0000021036326000 x25 = 0x00007ffe394ba000
[task 2020-06-25T00:50:35.968Z] 00:50:35 INFO - x26 = 0x00007ffe625f1370 x27 = 0x00007ffe65207b60
[task 2020-06-25T00:50:35.968Z] 00:50:35 INFO - x28 = 0x00007ffe394b81c8 fp = 0x0000005d3f17fc20
[task 2020-06-25T00:50:35.968Z] 00:50:35 INFO - lr = 0x00007ffe32b4e708 sp = 0x0000005d3f17fbf0
[task 2020-06-25T00:50:35.968Z] 00:50:35 INFO - pc = 0x00007ffe32b4e720
[task 2020-06-25T00:50:35.968Z] 00:50:35 INFO - Found by: given as instruction pointer in context
[task 2020-06-25T00:50:35.968Z] 00:50:35 INFO - 1 xul.dll!mozilla::`anonymous namespace'::RunWatchdog(void*) [nsTerminator.cpp:3add3a1747557d9b1e20c41afed20a94952ce5e8 : 214 + 0x0]
[task 2020-06-25T00:50:35.969Z] 00:50:35 INFO - fp = 0x0000005d3f17fc90 lr = 0x00007ffe39397e54
[task 2020-06-25T00:50:35.969Z] 00:50:35 INFO - sp = 0x0000005d3f17fc30 pc = 0x00007ffe32b4e708
[task 2020-06-25T00:50:35.969Z] 00:50:35 INFO - Found by: previous frame's frame pointer
[task 2020-06-25T00:50:35.969Z] 00:50:35 INFO - 2 nss3.dll!_PR_NativeRunThread(void*) [pruthr.c:3add3a1747557d9b1e20c41afed20a94952ce5e8 : 399 + 0x4]
[task 2020-06-25T00:50:35.969Z] 00:50:35 INFO - fp = 0x0000005d3f17fca0 lr = 0x00007ffe39388cb0
[task 2020-06-25T00:50:35.970Z] 00:50:35 INFO - sp = 0x0000005d3f17fca0 pc = 0x00007ffe39397e54
[task 2020-06-25T00:50:35.970Z] 00:50:35 INFO - Found by: previous frame's frame pointer
[task 2020-06-25T00:50:35.970Z] 00:50:35 INFO - 3 nss3.dll!pr_root(void*) [w95thred.c:3add3a1747557d9b1e20c41afed20a94952ce5e8 : 139 + 0x4]
[task 2020-06-25T00:50:35.970Z] 00:50:35 INFO - fp = 0x0000005d3f17fcb0 lr = 0x00007ffe617b2ee0
[task 2020-06-25T00:50:35.970Z] 00:50:35 INFO - sp = 0x0000005d3f17fcb0 pc = 0x00007ffe39388cb0
[task 2020-06-25T00:50:35.970Z] 00:50:35 INFO - Found by: previous frame's frame pointer
[task 2020-06-25T00:50:35.970Z] 00:50:35 INFO - 4 ucrtbase.dll!LdrpLangFallbackListFindNode + 0x84
[task 2020-06-25T00:50:35.970Z] 00:50:35 INFO - fp = 0x0000005d3f17fce0 lr = 0x00007ffe625e5ba4
[task 2020-06-25T00:50:35.971Z] 00:50:35 INFO - sp = 0x0000005d3f17fcc0 pc = 0x00007ffe617b2ee0
[task 2020-06-25T00:50:35.971Z] 00:50:35 INFO - Found by: previous frame's frame pointer
[task 2020-06-25T00:50:35.971Z] 00:50:35 INFO - 5 kernel32.dll + 0x15ba0
[task 2020-06-25T00:50:35.971Z] 00:50:35 INFO - fp = 0x0000005d3f17fd70 lr = 0x00007ffe44e0fc0c
[task 2020-06-25T00:50:35.971Z] 00:50:35 INFO - sp = 0x0000005d3f17fcf0 pc = 0x00007ffe625e5ba4
[task 2020-06-25T00:50:35.972Z] 00:50:35 INFO - Found by: previous frame's frame pointer
[task 2020-06-25T00:50:35.972Z] 00:50:35 INFO - 6 mozglue.dll!patched_BaseThreadInitThunk(int, void*, void*) [WindowsDllBlocklist.cpp:3add3a1747557d9b1e20c41afed20a94952ce5e8 : 592 + 0x14]
[task 2020-06-25T00:50:35.972Z] 00:50:35 INFO - fp = 0x0000005d3f17fd80 lr = 0x00007ffe65262544
[task 2020-06-25T00:50:35.972Z] 00:50:35 INFO - sp = 0x0000005d3f17fd80 pc = 0x00007ffe44e0fc0c
[task 2020-06-25T00:50:35.972Z] 00:50:35 INFO - Found by: previous frame's frame pointer
[task 2020-06-25T00:50:35.972Z] 00:50:35 INFO - 7 ntdll.dll!RtlUserThreadStart + 0x40
[task 2020-06-25T00:50:35.972Z] 00:50:35 INFO - fp = 0x0000000000000000 lr = 0x0000000000000000
[task 2020-06-25T00:50:35.973Z] 00:50:35 INFO - sp = 0x0000005d3f17fd90 pc = 0x00007ffe65262544
[task 2020-06-25T00:50:35.973Z] 00:50:35 INFO - Found by: previous frame's frame pointer
[task 2020-06-25T00:50:35.973Z] 00:50:35 INFO -

Comment 1

[Natalia Csoregi [:nataliaCs]]

:jya could you take a look at these? They're permafailing on mozilla-central, starting once https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C10%2Caarch64%2Copt%2Creftests%2Ctest-windows10-aarch64%2Fopt-crashtest-e10s%2Cr%28c%29&revision=36ea162317d744464c37bc2147b1dd7dd68b18c7&selectedTaskRun=UYVavkxZReSTQEavpokX5A.0 was merged.

Comment 2

[Intermittent Failures Robot]

49 failures in 5252 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

This is the #31 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 21
• autoland: 24
• try: 4

Platform breakdown:

• windows10-aarch64: 33
• windows10-64-qr: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1648326&startday=2020-06-22&endday=2020-06-28&tree=all

Comment 3

[Jean-Yves Avenard [:jya]]

That was fun tracking it down, especially with no ability to reproduce locally.

So we have the remoteSandboxBroker (used only on Windows for arm64) reading the current event target assuming to be on the IPC launch thread [1]
However since bug 1562762, this is now a taskqueue [2].

The remoteSandboxBroker does launch the task and dispatch a task on the TaskQueue (obtained via a call to GetCurrentSerialEventTarget()) [3]
to resolve the promise and then spin an event loop waiting for that task to run

Bug 1637500 would have exposed the issue as now GetCurrentSerialEventTarget will return the running taskqueue; rather than the TaskQueue's inner thread.
So running the event loop here won't allow the resolver task to be run while the TaskQueue is blocked; we need to dispatch the task on the inner thread.

And we do want the mIPCLaunchThread to be the running TaskQueue to make sure the tasks are run in the expected order, though on Windows the IPC launch threadpool is made of a single thread.

[1] https://searchfox.org/mozilla-central/rev/21f2b48e01f2e14a94e8d39a665b56fcc08ecbdb/security/sandbox/win/src/remotesandboxbroker/remoteSandboxBroker.cpp#41-52
[2] https://searchfox.org/mozilla-central/rev/fa23513682f77c6204adcb83c6f7e2bdd440f369/ipc/glue/GeckoChildProcessHost.cpp#152
[3] https://searchfox.org/mozilla-central/rev/21f2b48e01f2e14a94e8d39a665b56fcc08ecbdb/security/sandbox/win/src/remotesandboxbroker/remoteSandboxBroker.cpp#80-85

Comment 4

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1648326 - Don't dispatch runnable on the running taskqueue. r?bobowen,jld

The current taskqueue is blocked until the current function has finished; Running the event loop would only process events on the running thread.

Additionally, we make mIPCLaunchThread an nsISerialEventTarget to guarantee that at shutdown the tasks are run in order regardless of the IPC Launch Thread type.

Comment 5

[Jean-Yves Avenard [:jya]]

https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=U6bixNbnRMORnJEhbAOk0w.0&revision=ed16b921a47437aba53531b42b204f9a967de9ff

Comment 6

[Jean-Yves Avenard [:jya]]

Can you please review or suggest someone else to review?

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/87a66c9bce6e
Don't dispatch runnable on the running taskqueue. r=bobowen,jld

Comment 8

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/87a66c9bce6e

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Please nominate this for Beta approval when you get a chance.

Comment 10

[Jean-Yves Avenard [:jya]]

Comment on attachment 9160041 [details]
Bug 1648326 - Don't dispatch runnable on the running taskqueue. r?bobowen,jld

Beta/Release Uplift Approval Request

• User impact if declined: No ability to play encrypted content for windows on arm users (netflix, amazon prime, hulu etc)
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce: We have a test here, which caught the problem
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): We ensure we don't block a thread while waiting.
• String changes made/needed: none

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9160041 [details]
Bug 1648326 - Don't dispatch runnable on the running taskqueue. r?bobowen,jld

Fixes possible playback issues for encrypted content on Windows ARM devices. Approved for 79.0b3.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/9db3d5cb8008

Comment 13

[Intermittent Failures Robot]

23 failures in 4516 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 11
• mozilla-beta: 6
• autoland: 6

Platform breakdown:

• windows10-aarch64: 14
• windows10-64-qr: 8
• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1648326&startday=2020-06-29&endday=2020-07-05&tree=all