1603374 - Intermittent GECKO(1494) | SUMMARY: AddressSanitizer: SEGV /builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders/mozilla/dom/ServiceWorkerOpArgs.h:1178:9 in mozilla::dom::ServiceWorkerOpArgs::AssertSanity() const

Status: RESOLVED FIXED

(Core :: DOM: Service Workers, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: aciure [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=280830293&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/T4MS9rjYSDuMUzdAdXz0Dg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2019-12-12T09:28:50.984Z] 09:28:50 INFO - GECKO(1494) | ==1572==Hint: address points to the zero page.
[task 2019-12-12T09:28:51.848Z] 09:28:51 INFO - GECKO(1494) | #0 0x7fc018bef74b in mozilla::dom::ServiceWorkerOpArgs::AssertSanity() const /builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders/mozilla/dom/ServiceWorkerOpArgs.h:1178:9
[task 2019-12-12T09:28:51.889Z] 09:28:51 INFO - GECKO(1494) | #1 0x7fc01e000e5b in AssertSanity /builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders/mozilla/dom/ServiceWorkerOpArgs.h:1184:9
[task 2019-12-12T09:28:51.890Z] 09:28:51 INFO - GECKO(1494) | #2 0x7fc01e000e5b in get_ServiceWorkerUpdateStateOpArgs /builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders/mozilla/dom/ServiceWorkerOpArgs.h:1358:9
[task 2019-12-12T09:28:51.890Z] 09:28:51 INFO - GECKO(1494) | #3 0x7fc01e000e5b in mozilla::dom::UpdateServiceWorkerStateOp::Exec(JSContext*, mozilla::dom::WorkerPrivate*) /builds/worker/workspace/build/src/dom/serviceworkers/ServiceWorkerOp.cpp:543:15
[task 2019-12-12T09:28:51.890Z] 09:28:51 INFO - GECKO(1494) | #4 0x7fc01dfe0176 in mozilla::dom::UpdateServiceWorkerStateOp::UpdateStateOpRunnable::WorkerRun(JSContext*, mozilla::dom::WorkerPrivate*) /builds/worker/workspace/build/src/dom/serviceworkers/ServiceWorkerOp.cpp:507:25
[task 2019-12-12T09:28:51.906Z] 09:28:51 INFO - GECKO(1494) | #5 0x7fc01db7fc7e in mozilla::dom::WorkerRunnable::Run() /builds/worker/workspace/build/src/dom/workers/WorkerRunnable.cpp:369:12
[task 2019-12-12T09:28:51.907Z] 09:28:51 INFO - GECKO(1494) | #6 0x7fc01db810f4 in mozilla::dom::WorkerControlRunnable::Cancel() /builds/worker/workspace/build/src/dom/workers/WorkerRunnable.cpp:507:7
[task 2019-12-12T09:28:51.907Z] 09:28:51 INFO - GECKO(1494) | #7 0x7fc01dfdff28 in mozilla::dom::UpdateServiceWorkerStateOp::UpdateStateOpRunnable::Cancel() /builds/worker/workspace/build/src/dom/serviceworkers/ServiceWorkerOp.cpp:519:47
[task 2019-12-12T09:28:51.908Z] 09:28:51 INFO - GECKO(1494) | #8 0x7fc01db7f5dd in mozilla::dom::WorkerRunnable::Run() /builds/worker/workspace/build/src/dom/workers/WorkerRunnable.cpp:239:5
[task 2019-12-12T09:28:51.908Z] 09:28:51 INFO - GECKO(1494) | #9 0x7fc01db68991 in mozilla::dom::WorkerPrivate::ProcessAllControlRunnablesLocked() /builds/worker/workspace/build/src/dom/workers/WorkerPrivate.cpp:3415:9
[task 2019-12-12T09:28:51.910Z] 09:28:51 INFO - GECKO(1494) | #10 0x7fc01db6a350 in mozilla::dom::WorkerPrivate::ProcessAllControlRunnables() /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/dom/WorkerPrivate.h:953:12
[task 2019-12-12T09:28:51.910Z] 09:28:51 INFO - GECKO(1494) | #11 0x7fc01db8f933 in mozilla::dom::WorkerThread::Observer::OnProcessNextEvent(nsIThreadInternal*, bool) /builds/worker/workspace/build/src/dom/workers/WorkerThread.cpp:356:19
[task 2019-12-12T09:28:51.925Z] 09:28:51 INFO - GECKO(1494) | #12 0x7fc01757ba1c in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/workspace/build/src/xpcom/threads/nsThread.cpp:1133:3
[task 2019-12-12T09:28:51.926Z] 09:28:51 INFO - GECKO(1494) | #13 0x7fc017576f7e in NS_ProcessPendingEvents(nsIThread*, unsigned int) /builds/worker/workspace/build/src/xpcom/threads/nsThreadUtils.cpp:434:19
[task 2019-12-12T09:28:51.927Z] 09:28:51 INFO - GECKO(1494) | #14 0x7fc01db68fcd in ClearMainEventQueue /builds/worker/workspace/build/src/dom/workers/WorkerPrivate.cpp:3446:5
[task 2019-12-12T09:28:51.928Z] 09:28:51 INFO - GECKO(1494) | #15 0x7fc01db68fcd in mozilla::dom::WorkerPrivate::NotifyInternal(mozilla::dom::WorkerStatus) /builds/worker/workspace/build/src/dom/workers/WorkerPrivate.cpp:4234:7
[task 2019-12-12T09:28:51.929Z] 09:28:51 INFO - GECKO(1494) | #16 0x7fc01db7fc7e in mozilla::dom::WorkerRunnable::Run() /builds/worker/workspace/build/src/dom/workers/WorkerRunnable.cpp:369:12
[task 2019-12-12T09:28:51.930Z] 09:28:51 INFO - GECKO(1494) | #17 0x7fc01db68991 in mozilla::dom::WorkerPrivate::ProcessAllControlRunnablesLocked() /builds/worker/workspace/build/src/dom/workers/WorkerPrivate.cpp:3415:9
[task 2019-12-12T09:28:51.931Z] 09:28:51 INFO - GECKO(1494) | #18 0x7fc01db67459 in mozilla::dom::WorkerPrivate::DoRunLoop(JSContext*) /builds/worker/workspace/build/src/dom/workers/WorkerPrivate.cpp:2783:21
[task 2019-12-12T09:28:51.943Z] 09:28:51 INFO - GECKO(1494) | #19 0x7fc01db34ab3 in mozilla::dom::workerinternals::(anonymous namespace)::WorkerThreadPrimaryRunnable::Run() /builds/worker/workspace/build/src/dom/workers/RuntimeService.cpp:2323:40
[task 2019-12-12T09:28:51.945Z] 09:28:51 INFO - GECKO(1494) | #20 0x7fc01757c937 in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/workspace/build/src/xpcom/threads/nsThread.cpp:1240:14
[task 2019-12-12T09:28:51.945Z] 09:28:51 INFO - GECKO(1494) | #21 0x7fc01758538c in NS_ProcessNextEvent(nsIThread*, bool) /builds/worker/workspace/build/src/xpcom/threads/nsThreadUtils.cpp:486:10
[task 2019-12-12T09:28:51.948Z] 09:28:51 INFO - GECKO(1494) | #22 0x7fc018577cf9 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /builds/worker/workspace/build/src/ipc/glue/MessagePump.cpp:332:5
[task 2019-12-12T09:28:51.957Z] 09:28:51 INFO - GECKO(1494) | #23 0x7fc0184ac017 in RunInternal /builds/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:315:10
[task 2019-12-12T09:28:51.957Z] 09:28:51 INFO - GECKO(1494) | #24 0x7fc0184ac017 in RunHandler /builds/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:308:3
[task 2019-12-12T09:28:51.958Z] 09:28:51 INFO - GECKO(1494) | #25 0x7fc0184ac017 in MessageLoop::Run() /builds/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:290:3
[task 2019-12-12T09:28:51.960Z] 09:28:51 INFO - GECKO(1494) | #26 0x7fc017575a6a in nsThread::ThreadFunc(void*) /builds/worker/workspace/build/src/xpcom/threads/nsThread.cpp:459:11
[task 2019-12-12T09:28:51.961Z] 09:28:51 INFO - GECKO(1494) | #27 0x7fc033dd1d0e in _pt_root /builds/worker/workspace/build/src/nsprpub/pr/src/pthreads/ptthread.c:201:5
[task 2019-12-12T09:28:51.962Z] 09:28:51 INFO - GECKO(1494) | #28 0x7fc037af86b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
[task 2019-12-12T09:28:52.044Z] 09:28:52 INFO - GECKO(1494) | #29 0x7fc036b8141c in clone /build/glibc-LK5gWL/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:109
[task 2019-12-12T09:28:52.045Z] 09:28:52 INFO - GECKO(1494) | AddressSanitizer can not provide additional info.
[task 2019-12-12T09:28:52.046Z] 09:28:52 INFO - GECKO(1494) | SUMMARY: AddressSanitizer: SEGV /builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders/mozilla/dom/ServiceWorkerOpArgs.h:1178:9 in mozilla::dom::ServiceWorkerOpArgs::AssertSanity() const

Comment 1

[Intermittent Failures Robot]

3 failures in 4867 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 1

Platform breakdown:

• linux64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603374&startday=2019-12-09&endday=2019-12-15&tree=all

Comment 2

[Intermittent Failures Robot]

2 failures in 4582 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• linux64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603374&startday=2019-12-16&endday=2019-12-22&tree=all

Comment 3

[Intermittent Failures Robot]

2 failures in 1365 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1

Platform breakdown:

• linux64: 1
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603374&startday=2019-12-23&endday=2019-12-29&tree=all

Comment 4

[Intermittent Failures Robot]

2 failures in 1862 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 1

Platform breakdown:

• linux64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603374&startday=2019-12-30&endday=2020-01-05&tree=all

Comment 5

[Intermittent Failures Robot]

4 failures in 4472 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4

Platform breakdown:

• linux64: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603374&startday=2020-01-06&endday=2020-01-12&tree=all

Comment 6

[Intermittent Failures Robot]

1 failures in 3773 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603374&startday=2020-01-13&endday=2020-01-19&tree=all

Comment 7

[Jens Stutte [:jstutte]]

Even if it's rare, it might be worth looking at?

Comment 8

[Perry Jiang [:perry] [no longer employee, use ni?]]

I believe this is an IPC-related issue. The ServiceWorkerOpArgs::AssertSanity is sanity checking the private member variable ServiceWorkerOpArgs::mType, and ServiceWorkerOpArgs C++ code is generated from the IPDL definition...

Comment 9

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

The 0-argument overload of AssertSanity is a consistency check that nothing outside of IPC should be able to violate (without reinterpret casting, etc.). The 1-argument overload, however, checks that the type is equal to a specific value, and can fail if code outside of IPC has a bug and tries to access the wrong variant. I filed bug 1611338 to rename those methods to make this more obvious.

Specifically, in this case UpdateServiceWorkerStateOp::Exec assumes that mArgs holds a ServiceWorkerUpdateStateOpArgs, and apparently sometimes it doesn't; note the get_ServiceWorkerUpdateStateOpArgs frame in the crash stack.

Comment 10

[Perry Jiang [:perry] [no longer employee, use ni?]]

(In reply to Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧ from comment #9)

The 0-argument overload of AssertSanity is a consistency check that nothing outside of IPC should be able to violate (without reinterpret casting, etc.). The 1-argument overload, however, checks that the type is equal to a specific value, and can fail if code outside of IPC has a bug and tries to access the wrong variant. I filed bug 1611338 to rename those methods to make this more obvious.

Specifically, in this case UpdateServiceWorkerStateOp::Exec assumes that mArgs holds a ServiceWorkerUpdateStateOpArgs, and apparently sometimes it doesn't; note the get_ServiceWorkerUpdateStateOpArgs frame in the crash stack.

Hm, it looks like the 1-arg overload is calling the 0-arg overload, and the 0-arg overload is causing an assertion failure, and not the MOZ_RELEASE_ASSERT directly called by the 1-arg overload.

Comment 11

[Perry Jiang [:perry] [no longer employee, use ni?]]

I'm also pretty sure there's no weird casting going on - UpdateServiceWorkerStateOps are only created at https://searchfox.org/mozilla-central/rev/f98dad153b59a985efd4505912588d4651033395/dom/serviceworkers/ServiceWorkerOp.cpp#1642 which is guarded by the variant check. There's also a double check at https://searchfox.org/mozilla-central/rev/f98dad153b59a985efd4505912588d4651033395/dom/serviceworkers/ServiceWorkerOp.cpp#532 for the right variant type before creating a UpdateStateOpRunnable (which shows up in the stack)...

Comment 12

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

(In reply to Perry Jiang [:perry] from comment #10)

Hm, it looks like the 1-arg overload is calling the 0-arg overload, and the 0-arg overload is causing an assertion failure, and not the MOZ_RELEASE_ASSERT directly called by the 1-arg overload.

Good point. From the line number it looks like mType < T__None. Could this be a use-after-free, and it's reading 0xe5e5e5e5 as a negative number?

Comment 13

[Perry Jiang [:perry] [no longer employee, use ni?]]

Yeah, I think that's possible...

Comment 14

[Intermittent Failures Robot]

1 failures in 4004 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux1804-64-asan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603374&startday=2020-01-20&endday=2020-01-26&tree=all

Comment 16

[Perry Jiang [:perry] [no longer employee, use ni?]]

Attachment: Bug 1603374 - account for Cancel calling WorkerRun r?#dom-workers-and-storage

MainThreadWorkerControlRunnable::Cancel (indirectly) calls WorkerRun, which is
neither intuitive nor documented, but UpdateStateOpRunnable needs to account for
that to avoid a release assertion failure in ServiceWorkerOpArgs::AssertSanity.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/aed68d2d967d
account for Cancel calling WorkerRun r=dom-workers-and-storage-reviewers,asuth

Comment 18

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/aed68d2d967d