Closed Bug 1724168 Opened 3 years ago Closed 1 year ago

Intermittent ==3765==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x557f829e8698 bp 0x7fffeeaba670 sp 0x7fffeeaba660 T0)

Categories

(Core :: General, defect, P5)

defect

Tracking

()

RESOLVED INCOMPLETE

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Keywords: intermittent-failure)

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=347499101&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/V8uV8d80T62UnZCqw_4qbQ/runs/0/artifacts/public/logs/live_backing.log


[task 2021-08-05T07:41:58.024Z] 07:41:58     INFO - TEST-START | browser/components/urlbar/tests/browser/browser_searchMode_setURI.js
[task 2021-08-05T07:42:00.938Z] 07:42:00     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:00.939Z] 07:42:00     INFO - GECKO(1559) | Suppressions used:
[task 2021-08-05T07:42:00.939Z] 07:42:00     INFO - GECKO(1559) |   count      bytes template
[task 2021-08-05T07:42:00.939Z] 07:42:00     INFO - GECKO(1559) |      13        416 nsComponentManagerImpl
[task 2021-08-05T07:42:00.939Z] 07:42:00     INFO - GECKO(1559) |       2        288 libfontconfig.so
[task 2021-08-05T07:42:00.939Z] 07:42:00     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:12.669Z] 07:42:12     INFO - GECKO(1559) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T07:42:12.669Z] 07:42:12     INFO - GECKO(1559) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T07:42:18.002Z] 07:42:17     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:18.008Z] 07:42:17     INFO - GECKO(1559) | Suppressions used:
[task 2021-08-05T07:42:18.008Z] 07:42:17     INFO - GECKO(1559) |   count      bytes template
[task 2021-08-05T07:42:18.008Z] 07:42:17     INFO - GECKO(1559) |      13        416 nsComponentManagerImpl
[task 2021-08-05T07:42:18.008Z] 07:42:17     INFO - GECKO(1559) |       2        288 libfontconfig.so
[task 2021-08-05T07:42:18.008Z] 07:42:17     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:21.249Z] 07:42:21     INFO - GECKO(1559) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T07:42:21.249Z] 07:42:21     INFO - GECKO(1559) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T07:42:26.588Z] 07:42:26     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:26.589Z] 07:42:26     INFO - GECKO(1559) | Suppressions used:
[task 2021-08-05T07:42:26.589Z] 07:42:26     INFO - GECKO(1559) |   count      bytes template
[task 2021-08-05T07:42:26.589Z] 07:42:26     INFO - GECKO(1559) |      13        416 nsComponentManagerImpl
[task 2021-08-05T07:42:26.589Z] 07:42:26     INFO - GECKO(1559) |       2        288 libfontconfig.so
[task 2021-08-05T07:42:26.589Z] 07:42:26     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:29.848Z] 07:42:29     INFO - GECKO(1559) | out of memory: 0x0000000005400000 bytes requested
[task 2021-08-05T07:42:29.872Z] 07:42:29     INFO - GECKO(1559) | AddressSanitizer:DEADLYSIGNAL
[task 2021-08-05T07:42:29.872Z] 07:42:29     INFO - GECKO(1559) | =================================================================
[task 2021-08-05T07:42:29.872Z] 07:42:29    ERROR - GECKO(1559) | ==3765==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x557f829e8698 bp 0x7fffeeaba670 sp 0x7fffeeaba660 T0)
[task 2021-08-05T07:42:29.872Z] 07:42:29     INFO - GECKO(1559) | ==3765==The signal is caused by a WRITE memory access.
[task 2021-08-05T07:42:29.872Z] 07:42:29     INFO - GECKO(1559) | ==3765==Hint: address points to the zero page.
[task 2021-08-05T07:42:31.276Z] 07:42:31     INFO - GECKO(1559) | ==3765==WARNING: failed to fork (errno 12)
[task 2021-08-05T07:42:31.286Z] 07:42:31     INFO - GECKO(1559) | ==3765==WARNING: failed to fork (errno 12)
[task 2021-08-05T07:42:31.286Z] 07:42:31     INFO - GECKO(1559) | ==3765==WARNING: failed to fork (errno 12)
[task 2021-08-05T07:42:31.286Z] 07:42:31     INFO - GECKO(1559) | ==3765==WARNING: failed to fork (errno 12)
[task 2021-08-05T07:42:31.286Z] 07:42:31     INFO - GECKO(1559) | ==3765==WARNING: failed to fork (errno 12)
[task 2021-08-05T07:42:31.286Z] 07:42:31     INFO - GECKO(1559) | ==3765==WARNING: Failed to use and restart external symbolizer!
[task 2021-08-05T07:42:31.286Z] 07:42:31     INFO - GECKO(1559) |     #0 0x557f829e8698  (/builds/worker/workspace/build/application/firefox/firefox+0x10b698)
[task 2021-08-05T07:42:31.287Z] 07:42:31     INFO - GECKO(1559) |     #1 0x557f829e882d  (/builds/worker/workspace/build/application/firefox/firefox+0x10b82d)
[task 2021-08-05T07:42:31.287Z] 07:42:31     INFO - GECKO(1559) |     #2 0x557f829e8d5b  (/builds/worker/workspace/build/application/firefox/firefox+0x10bd5b)
[task 2021-08-05T07:42:31.287Z] 07:42:31     INFO - GECKO(1559) |     #3 0x7fe4aea743a3  (/builds/worker/workspace/build/application/firefox/libxul.so+0x34633a3)
[task 2021-08-05T07:42:31.287Z] 07:42:31     INFO - GECKO(1559) |     #4 0x7fe4af74309e  (/builds/worker/workspace/build/application/firefox/libxul.so+0x413209e)
[task 2021-08-05T07:42:31.287Z] 07:42:31     INFO - GECKO(1559) |     #5 0x7fe4af73571a  (/builds/worker/workspace/build/application/firefox/libxul.so+0x412471a)
[task 2021-08-05T07:42:31.287Z] 07:42:31     INFO - GECKO(1559) |     #6 0x7fe4af86c031  (/builds/worker/workspace/build/application/firefox/libxul.so+0x425b031)
[task 2021-08-05T07:42:31.287Z] 07:42:31     INFO - GECKO(1559) |     #7 0x7fe4af8aa41f  (/builds/worker/workspace/build/application/firefox/libxul.so+0x429941f)
[task 2021-08-05T07:42:31.287Z] 07:42:31     INFO - GECKO(1559) |     #8 0x7fe4aed0d992  (/builds/worker/workspace/build/application/firefox/libxul.so+0x36fc992)
[task 2021-08-05T07:42:31.287Z] 07:42:31     INFO - GECKO(1559) |     #9 0x7fe4aecda474  (/builds/worker/workspace/build/application/firefox/libxul.so+0x36c9474)
[task 2021-08-05T07:42:31.287Z] 07:42:31     INFO - GECKO(1559) |     #10 0x7fe4aecd7cc8  (/builds/worker/workspace/build/application/firefox/libxul.so+0x36c6cc8)
[task 2021-08-05T07:42:31.287Z] 07:42:31     INFO - GECKO(1559) |     #11 0x7fe4aecd83dd  (/builds/worker/workspace/build/application/firefox/libxul.so+0x36c73dd)
[task 2021-08-05T07:42:31.287Z] 07:42:31     INFO - GECKO(1559) |     #12 0x7fe4aed17c71  (/builds/worker/workspace/build/application/firefox/libxul.so+0x3706c71)
[task 2021-08-05T07:42:31.288Z] 07:42:31     INFO - GECKO(1559) |     #13 0x7fe4aecf4f97  (/builds/worker/workspace/build/application/firefox/libxul.so+0x36e3f97)
[task 2021-08-05T07:42:31.288Z] 07:42:31     INFO - GECKO(1559) |     #14 0x7fe4aecf06ad  (/builds/worker/workspace/build/application/firefox/libxul.so+0x36df6ad)
[task 2021-08-05T07:42:31.289Z] 07:42:31     INFO - GECKO(1559) |     #15 0x7fe4b5d7a554  (/builds/worker/workspace/build/application/firefox/libxul.so+0xa769554)
[task 2021-08-05T07:42:31.289Z] 07:42:31     INFO - GECKO(1559) |     #16 0x7fe4b9c45bdf  (/builds/worker/workspace/build/application/firefox/libxul.so+0xe634bdf)
[task 2021-08-05T07:42:31.290Z] 07:42:31     INFO - GECKO(1559) |     #17 0x7fe4afd0aae1  (/builds/worker/workspace/build/application/firefox/libxul.so+0x46f9ae1)
[task 2021-08-05T07:42:31.290Z] 07:42:31     INFO - GECKO(1559) |     #18 0x7fe4b9c455b4  (/builds/worker/workspace/build/application/firefox/libxul.so+0xe6345b4)
[task 2021-08-05T07:42:31.291Z] 07:42:31     INFO - GECKO(1559) |     #19 0x557f829e208d  (/builds/worker/workspace/build/application/firefox/firefox+0x10508d)
[task 2021-08-05T07:42:31.291Z] 07:42:31     INFO - GECKO(1559) |     #20 0x557f829e24b1  (/builds/worker/workspace/build/application/firefox/firefox+0x1054b1)
[task 2021-08-05T07:42:31.292Z] 07:42:31     INFO - GECKO(1559) |     #21 0x7fe4d2b9db96  (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
[task 2021-08-05T07:42:31.292Z] 07:42:31     INFO - GECKO(1559) |     #22 0x557f8293374c  (/builds/worker/workspace/build/application/firefox/firefox+0x5674c)
[task 2021-08-05T07:42:31.292Z] 07:42:31     INFO - GECKO(1559) | AddressSanitizer can not provide additional info.
[task 2021-08-05T07:42:31.293Z] 07:42:31     INFO - GECKO(1559) | SUMMARY: AddressSanitizer: SEGV (/builds/worker/workspace/build/application/firefox/firefox+0x10b698)
[task 2021-08-05T07:42:31.293Z] 07:42:31     INFO - GECKO(1559) | ==3765==ABORTING
[task 2021-08-05T07:42:33.108Z] 07:42:33     INFO - GECKO(1559) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T07:42:33.122Z] 07:42:33     INFO - GECKO(1559) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T07:42:34.539Z] 07:42:34     INFO - GECKO(1559) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T07:42:35.714Z] 07:42:35     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:35.715Z] 07:42:35     INFO - GECKO(1559) | Suppressions used:
[task 2021-08-05T07:42:35.716Z] 07:42:35     INFO - GECKO(1559) |   count      bytes template
[task 2021-08-05T07:42:35.717Z] 07:42:35     INFO - GECKO(1559) |      13        416 nsComponentManagerImpl
[task 2021-08-05T07:42:35.718Z] 07:42:35     INFO - GECKO(1559) |       2        288 libfontconfig.so
[task 2021-08-05T07:42:35.719Z] 07:42:35     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:35.978Z] 07:42:35     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:35.978Z] 07:42:35     INFO - GECKO(1559) | Suppressions used:
[task 2021-08-05T07:42:35.978Z] 07:42:35     INFO - GECKO(1559) |   count      bytes template
[task 2021-08-05T07:42:35.978Z] 07:42:35     INFO - GECKO(1559) |      13        416 nsComponentManagerImpl
[task 2021-08-05T07:42:35.978Z] 07:42:35     INFO - GECKO(1559) |       2        288 libfontconfig.so
[task 2021-08-05T07:42:35.978Z] 07:42:35     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:36.276Z] 07:42:36     INFO - GECKO(1559) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T07:42:37.876Z] 07:42:37     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:37.876Z] 07:42:37     INFO - GECKO(1559) | Suppressions used:
[task 2021-08-05T07:42:37.876Z] 07:42:37     INFO - GECKO(1559) |   count      bytes template
[task 2021-08-05T07:42:37.876Z] 07:42:37     INFO - GECKO(1559) |      13        416 nsComponentManagerImpl
[task 2021-08-05T07:42:37.876Z] 07:42:37     INFO - GECKO(1559) |       2        288 libfontconfig.so
[task 2021-08-05T07:42:37.876Z] 07:42:37     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:37.987Z] 07:42:37     INFO - GECKO(1559) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T07:42:37.987Z] 07:42:37     INFO - GECKO(1559) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T07:42:39.283Z] 07:42:39     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:39.283Z] 07:42:39     INFO - GECKO(1559) | Suppressions used:
[task 2021-08-05T07:42:39.283Z] 07:42:39     INFO - GECKO(1559) |   count      bytes template
[task 2021-08-05T07:42:39.283Z] 07:42:39     INFO - GECKO(1559) |      13        416 nsComponentManagerImpl
[task 2021-08-05T07:42:39.283Z] 07:42:39     INFO - GECKO(1559) |       2        288 libfontconfig.so
[task 2021-08-05T07:42:39.283Z] 07:42:39     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:40.751Z] 07:42:40     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:40.751Z] 07:42:40     INFO - GECKO(1559) | Suppressions used:
[task 2021-08-05T07:42:40.751Z] 07:42:40     INFO - GECKO(1559) |   count      bytes template
[task 2021-08-05T07:42:40.751Z] 07:42:40     INFO - GECKO(1559) |      13        416 nsComponentManagerImpl
[task 2021-08-05T07:42:40.751Z] 07:42:40     INFO - GECKO(1559) |       2        288 libfontconfig.so
[task 2021-08-05T07:42:40.751Z] 07:42:40     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:42.348Z] 07:42:42     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:42.348Z] 07:42:42     INFO - GECKO(1559) | Suppressions used:
[task 2021-08-05T07:42:42.348Z] 07:42:42     INFO - GECKO(1559) |   count      bytes template
[task 2021-08-05T07:42:42.348Z] 07:42:42     INFO - GECKO(1559) |      13        416 nsComponentManagerImpl
[task 2021-08-05T07:42:42.348Z] 07:42:42     INFO - GECKO(1559) |       2        288 libfontconfig.so
[task 2021-08-05T07:42:42.348Z] 07:42:42     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:43.868Z] 07:42:43     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:43.868Z] 07:42:43     INFO - GECKO(1559) | Suppressions used:
[task 2021-08-05T07:42:43.868Z] 07:42:43     INFO - GECKO(1559) |   count      bytes template
[task 2021-08-05T07:42:43.868Z] 07:42:43     INFO - GECKO(1559) |      13        416 nsComponentManagerImpl
[task 2021-08-05T07:42:43.868Z] 07:42:43     INFO - GECKO(1559) |       2        288 libfontconfig.so
[task 2021-08-05T07:42:43.868Z] 07:42:43     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:44.650Z] 07:42:44     INFO - GECKO(1559) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T07:42:45.684Z] 07:42:45     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:45.684Z] 07:42:45     INFO - GECKO(1559) | Suppressions used:
[task 2021-08-05T07:42:45.684Z] 07:42:45     INFO - GECKO(1559) |   count      bytes template
[task 2021-08-05T07:42:45.684Z] 07:42:45     INFO - GECKO(1559) |      13        416 nsComponentManagerImpl
[task 2021-08-05T07:42:45.684Z] 07:42:45     INFO - GECKO(1559) |       2        288 libfontconfig.so
[task 2021-08-05T07:42:45.684Z] 07:42:45     INFO - GECKO(1559) | -----------------------------------------------------
[task 2021-08-05T07:42:46.203Z] 07:42:46     INFO - GECKO(1559) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T07:42:46.206Z] 07:42:46     INFO - GECKO(1559) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-08-05T07:42:46.212Z] 07:42:46     INFO - GECKO(1559) | MEMORY STAT | vsize 20976258MB | residentFast 1829MB
[task 2021-08-05T07:42:46.219Z] 07:42:46     INFO - TEST-OK | browser/components/urlbar/tests/browser/browser_searchMode_setURI.js | took 48190ms
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → DUPLICATE

This looks unrelated to bug 1522306; it's an OOM. The WARNING: failed to fork (errno 12) messages are part of that: 12 is ENOMEM / “Out of memory”. I also symbolized the stack by hand (downloading the target.tar.bz2 and using addr2line); this is a little less pretty than fix_stacks but should be readable:

0x000000000010b698
mozalloc_abort
/builds/worker/checkouts/gecko/memory/mozalloc/mozalloc_abort.cpp:33
0x000000000010b82d
mozalloc_handle_oom(unsigned long)
/builds/worker/checkouts/gecko/memory/mozalloc/mozalloc_oom.cpp:51
0x000000000010b82d
mozalloc_handle_oom(unsigned long)
/builds/worker/checkouts/gecko/memory/mozalloc/mozalloc_oom.cpp:51
0x00000000034633a3
nsTArrayInfallibleAllocator::ResultTypeProxy nsTArray_base<nsTArrayInfallibleAllocator, nsTArray_RelocateUsingMemutils>::EnsureCapacity<nsTArrayInfallibleAllocator>(unsigned long, unsigned long)
/builds/worker/workspace/obj-build/dist/include/nsTArray.h:259
EnsureCapacity<nsTArrayInfallibleAllocator>
/builds/worker/workspace/obj-build/dist/include/nsTArray-inl.h:223
0x000000000413209e
mozilla::UniquePtr<mozilla::net::ChannelEvent>* nsTArray_Impl<mozilla::UniquePtr<mozilla::net::ChannelEvent>, nsTArrayInfallibleAllocator>::AppendElementInternal<nsTArrayInfallibleAllocator, mozilla::UniquePtr<mozilla::net::ChannelEvent>>(mozilla::UniquePtr<mozilla::net::ChannelEvent>&&)
/builds/worker/workspace/obj-build/dist/include/nsTArray.h:2664
0x000000000412471a
mozilla::net::ChannelEventQueue::RunOrEnqueue(mozilla::net::ChannelEvent*, bool)
/builds/worker/workspace/obj-build/dist/include/nsTArray.h:2807
RunOrEnqueue
/builds/worker/workspace/obj-build/dist/include/mozilla/net/ChannelEventQueue.h:218
0x000000000425b031
mozilla::net::HttpChannelChild::DoNotifyListener()
/builds/worker/checkouts/gecko/netwerk/protocol/http/HttpChannelChild.cpp:1213
mozilla::detail::RunnableMethodImpl<RefPtr<mozilla::net::HttpChannelChild>, void (mozilla::net::HttpChannelChild::*)(), true, (mozilla::RunnableKind)0>::Run()
/builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1148
apply<mozilla::net::HttpChannelChild, void (mozilla::net::HttpChannelChild::*)()>
/builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1154
Run
/builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1201
0x00000000036fc992
mozilla::RunnableTask::Run()
/builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:502

In fact, it looks like most of the crashes recently assigned to bug 1522306 have those failed to fork (errno 12) lines: 1 2 3. I haven't tried to symbolize the stacks but I bet they're also OOMs.

Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → DUPLICATE

"SEGV on unknown address" and "ThreadSanitizer: data race" are different errors so I don't think this should be duped.

Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---

My guess is that the most common cause of this is going to be RunWatchdog, as in bug 1684441. I looked at a few, and some of them had a stack for the thread that was crashing and it was the watchdog threaad.

See Also: → 1684441

I'm not sure where shutdown hangs should go, but not in Firefox.

Component: Address Bar → General
Product: Firefox → Core
See Also: → 1543737
Status: REOPENED → RESOLVED
Closed: 3 years ago2 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 2 years ago1 year ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.