Intermittent debug telemetry\marionette\tests\client\test_search_counts_across_sessions.py TestSearchCounts.test_search_counts | application crashed [@ xul.dll + 0x1abe60]
Categories
(Toolkit :: Telemetry, defect, P3)
Tracking
()
People
(Reporter: intermittent-bug-filer, Unassigned)
References
Details
(Keywords: crash, intermittent-failure, regression)
Crash Data
Filed by: malexandru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=255205588&repo=autoland
Full log: https://queue.taskcluster.net/v1/task/A7CXc8meRTKFRlgAEXI9cg/runs/0/artifacts/public/logs/live_backing.log
06:18:36 INFO - [Parent 8156, Main Thread] WARNING: Extra shutdown CC: 'i < NORMAL_SHUTDOWN_COLLECTIONS', file z:/build/build/src/xpcom/base/nsCycleCollector.cpp, line 3361
06:18:36 INFO - Hit MOZ_CRASH(NSS_Shutdown failed) at z:/build/build/src/xpcom/build/XPCOMInit.cpp:747
06:18:36 INFO - #01: workerlz4_maxCompressedSize[Z:\task_1562563271\build\application\firefox\xul.dll +0x592e6d1]
06:18:36 INFO - #02: workerlz4_maxCompressedSize[Z:\task_1562563271\build\application\firefox\xul.dll +0x592f1ce]
06:18:36 INFO - #03: Ordinal0[Z:\task_1562563271\build\application\firefox\firefox.exe +0x16f5]
06:18:36 INFO - #04: Ordinal0[Z:\task_1562563271\build\application\firefox\firefox.exe +0x1232]
06:18:36 INFO - #05: TargetNtUnmapViewOfSection[Z:\task_1562563271\build\application\firefox\firefox.exe +0x85ae8]
06:18:36 INFO - #06: BaseThreadInitThunk[C:\Windows\System32\KERNEL32.DLL +0x13034]
06:18:36 INFO - #07: RtlUserThreadStart[C:\Windows\SYSTEM32\ntdll.dll +0x71461]
06:18:39 INFO - [Parent 6000, Main Thread] WARNING: XPCOM objects created/destroyed from static ctor/dtor: file z:/build/build/src/xpcom/base/nsTraceRefcnt.cpp, line 194
06:18:39 INFO - [Parent 6000, Main Thread] WARNING: XPCOM objects created/destroyed from static ctor/dtor: file z:/build/build/src/xpcom/base/nsTraceRefcnt.cpp, line 194
06:18:39 INFO - nsStringStats
06:18:39 INFO - => mAllocCount: 96588
06:18:39 INFO - => mReallocCount: 0
06:18:39 INFO - => mFreeCount: 96588
06:18:39 INFO - => mShareCount: 90162
06:18:39 INFO - => mAdoptCount: 2522
06:18:39 INFO - => mAdoptFreeCount: 2584
06:18:39 INFO - => Process ID: 6000, Thread ID: 10288
06:19:45 INFO - mozcrash Copy/paste: Z:\task_1562563271\build\win32-minidump_stackwalk.exe Z:\task_1562563271\build\tmpfgf0p9.mozrunner\minidumps\b35ab0bf-457c-452b-9465-c15eb7335e79.dmp c:\users\task_1562563271\appdata\local\temp\tmptef830
06:19:48 INFO - mozcrash Saved minidump as Z:\task_1562563271\build\blobber_upload_dir\b35ab0bf-457c-452b-9465-c15eb7335e79.dmp
06:19:48 INFO - mozcrash Saved app info as Z:\task_1562563271\build\blobber_upload_dir\b35ab0bf-457c-452b-9465-c15eb7335e79.extra
06:19:48 INFO - PROCESS-CRASH | telemetry\marionette\tests\client\test_search_counts_across_sessions.py TestSearchCounts.test_search_counts | application crashed [@ xul.dll + 0x1abe60]
06:19:48 INFO - Crash dump filename: Z:\task_1562563271\build\tmpfgf0p9.mozrunner\minidumps\b35ab0bf-457c-452b-9465-c15eb7335e79.dmp
06:19:48 INFO - Operating system: Windows NT
06:19:48 INFO - 10.0.17134
06:19:48 INFO - CPU: amd64
06:19:48 INFO - family 6 model 85 stepping 4
06:19:48 INFO - 8 CPUs
06:19:48 INFO -
06:19:48 INFO - GPU: UNKNOWN
06:19:48 INFO -
06:19:48 INFO - Crash reason: EXCEPTION_BREAKPOINT
06:19:48 INFO - Crash address: 0x7ffa4451be60
06:19:48 INFO - Process uptime: 7 seconds
06:19:48 INFO -
06:19:48 INFO - Thread 0 (crashed)
06:19:48 INFO - 0 xul.dll + 0x1abe60
06:19:48 INFO - rax = 0x00007ffa4bd79938 rdx = 0x00007ffa8949a640
06:19:48 INFO - rcx = 0x00007ffa7f0c7aa8 rbx = 0x00007ff692c092c3
06:19:48 INFO - rsi = 0x0000026346055328 rdi = 0x000000de6cbff828
06:19:48 INFO - rbp = 0x000000de6cbff801 rsp = 0x000000de6cbff7e0
06:19:48 INFO - r8 = 0x000000de6cbf9828 r9 = 0x0000000000000000
06:19:48 INFO - r10 = 0x0000000000000000 r11 = 0x000000de6cbfae60
06:19:48 INFO - r12 = 0x000000de6cbff9c0 r13 = 0x00007ffa8ba53fc0
06:19:48 INFO - r14 = 0x000000de6cbffb70 r15 = 0x0000000000780001
06:19:48 INFO - rip = 0x00007ffa4451be60
06:19:48 INFO - Found by: given as instruction pointer in context
06:19:48 INFO - 1 ucrtbase.dll!strset + 0x392f0
Updated•7 years ago
|
Comment 1•7 years ago
|
||
Happened exactly once, only on autoland. Stack doesn't have anything actionable that I can see (the references to lz4 maybe have something to do with the compressed on-disk format of pings, but given it failed during xpcom init I'm not convinced).
ni?raphael Do you see anything in here that seems investigatable? If not, feel free to close it down.
| Comment hidden (Intermittent Failures Robot) |
Comment 4•7 years ago
|
||
If it happens again we can reopen and maybe between multiple reports there'll be enough information to work on.
Comment 5•7 years ago
|
||
Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=258172685&repo=autoland&lineNumber=2647
[task 2019-07-24T20:32:45.386Z] 20:32:45 INFO - TEST-START | telemetry/marionette/tests/client/test_search_counts_across_sessions.py TestSearchCounts.test_search_counts
[task 2019-07-24T20:34:01.990Z] 20:34:01 INFO - PROCESS-CRASH | telemetry/marionette/tests/client/test_search_counts_across_sessions.py TestSearchCounts.test_search_counts | application crashed [@ libxul.so + 0x2909859]
[task 2019-07-24T20:34:01.990Z] 20:34:01 INFO - Crash dump filename: /builds/worker/workspace/build/tmpUhJ3Qh.mozrunner/minidumps/4e601a8e-12e7-2420-0592-0c2f8877df51.dmp
[task 2019-07-24T20:34:01.991Z] 20:34:01 INFO - Operating system: Linux
[task 2019-07-24T20:34:01.991Z] 20:34:01 INFO - 0.0.0 Linux 4.4.0-1014-aws #14taskcluster1-Ubuntu SMP Tue Apr 3 10:27:00 UTC 2018 x86_64
[task 2019-07-24T20:34:01.991Z] 20:34:01 INFO - CPU: amd64
[task 2019-07-24T20:34:01.991Z] 20:34:01 INFO - family 6 model 62 stepping 4
[task 2019-07-24T20:34:01.992Z] 20:34:01 INFO - 2 CPUs
[task 2019-07-24T20:34:01.992Z] 20:34:01 INFO -
[task 2019-07-24T20:34:01.992Z] 20:34:01 INFO - GPU: UNKNOWN
[task 2019-07-24T20:34:01.993Z] 20:34:01 INFO -
[task 2019-07-24T20:34:01.993Z] 20:34:01 INFO - Crash reason: SIGSEGV /SEGV_MAPERR
[task 2019-07-24T20:34:01.993Z] 20:34:01 INFO - Crash address: 0x0
[task 2019-07-24T20:34:01.994Z] 20:34:01 INFO - Process uptime: not available
[task 2019-07-24T20:34:01.994Z] 20:34:01 INFO -
[task 2019-07-24T20:34:01.994Z] 20:34:01 INFO - Thread 26 (crashed)
[task 2019-07-24T20:34:01.995Z] 20:34:01 INFO - 0 libxul.so + 0x2909859
[task 2019-07-24T20:34:01.995Z] 20:34:01 INFO - rax = 0x00007f1c95630180 rdx = 0x0000000000000074
[task 2019-07-24T20:34:01.995Z] 20:34:01 INFO - rcx = 0x0000561f2ed90c40 rbx = 0x00007f1c955df340
[task 2019-07-24T20:34:01.996Z] 20:34:01 INFO - rsi = 0x00007f1c954917c0 rdi = 0x00007f1c956301b4
[task 2019-07-24T20:34:01.996Z] 20:34:01 INFO - rbp = 0x00007f1c944fee90 rsp = 0x00007f1c944fee10
[task 2019-07-24T20:34:01.997Z] 20:34:01 INFO - r8 = 0x0000000000000001 r9 = 0x0000000000000000
[task 2019-07-24T20:34:01.997Z] 20:34:01 INFO - r10 = 0x0000000000000024 r11 = 0x00007f1cbe4b5e10
[task 2019-07-24T20:34:01.997Z] 20:34:01 INFO - r12 = 0x00007f1c955df3d0 r13 = 0x0000000000000001
[task 2019-07-24T20:34:01.998Z] 20:34:01 INFO - r14 = 0x00007f1c955df370 r15 = 0x00007f1c955df3a0
[task 2019-07-24T20:34:01.998Z] 20:34:01 INFO - rip = 0x00007f1cae4ae859
[task 2019-07-24T20:34:01.998Z] 20:34:01 INFO - Found by: given as instruction pointer in context
[task 2019-07-24T20:34:01.999Z] 20:34:01 INFO - 1 libxul.so + 0x3fad0d1
[task 2019-07-24T20:34:01.999Z] 20:34:01 INFO - rbp = 0x00007f1c944feec0 rsp = 0x00007f1c944feea0
[task 2019-07-24T20:34:01.999Z] 20:34:01 INFO - rip = 0x00007f1cafb520d1
[task 2019-07-24T20:34:02.000Z] 20:34:02 INFO - Found by: previous frame's frame pointer
[task 2019-07-24T20:34:02.000Z] 20:34:02 INFO - 2 libnspr4.so + 0x2a5be
[task 2019-07-24T20:34:02.001Z] 20:34:02 INFO - rbp = 0x00007f1c944fef10 rsp = 0x00007f1c944feed0
[task 2019-07-24T20:34:02.001Z] 20:34:02 INFO - rip = 0x00007f1cbf7605be
[task 2019-07-24T20:34:02.001Z] 20:34:02 INFO - Found by: previous frame's frame pointer
[task 2019-07-24T20:34:02.002Z] 20:34:02 INFO - 3 libpthread-2.23.so!start_thread [pthread_create.c : 333 + 0x11]
[task 2019-07-24T20:34:02.002Z] 20:34:02 INFO - rsp = 0x00007f1c944fef20 rip = 0x00007f1cbf3ac6ba
[task 2019-07-24T20:34:02.003Z] 20:34:02 INFO - Found by: stack scanning
[task 2019-07-24T20:34:02.003Z] 20:34:02 INFO - 4 libc-2.23.so!__clone + 0x6d
[task 2019-07-24T20:34:02.003Z] 20:34:02 INFO - rsp = 0x00007f1c944fefc0 rip = 0x00007f1cbe43541d
[task 2019-07-24T20:34:02.004Z] 20:34:02 INFO - Found by: stack scanning
Updated•7 years ago
|
| Comment hidden (Intermittent Failures Robot) |
Comment 7•7 years ago
|
||
The priority flag is not set for this bug.
:chutten, could you have a look please?
For more information, please visit auto_nag documentation.
Comment 8•7 years ago
|
||
Raphael, any ideas?
The first report was Win10x64 debug, the second Linux64 opt... so there doesn't appear to be a build- or platform-specific component linking the crashes.
Comment 9•7 years ago
|
||
Looks like the crash happens when we try to restart Firefox.
Henrik, do you spot any investigatable information in the logs?
Comment 10•7 years ago
|
||
When the crash stack frames aren't helpful you can rely on the assertion if one is present. And that is the case here:
06:18:36 INFO - [Parent 8156, Main Thread] WARNING: Extra shutdown CC: 'i < NORMAL_SHUTDOWN_COLLECTIONS', file z:/build/build/src/xpcom/base/nsCycleCollector.cpp, line 3361
06:18:36 INFO - Hit MOZ_CRASH(NSS_Shutdown failed) at z:/build/build/src/xpcom/build/XPCOMInit.cpp:747
The source link points to:
Some helpful info can be found at bug 1230312 comment 0, but given the low frequency of this crash there might be nothing actionable for you at the moment. But maybe Dana has an idea.
Comment 11•7 years ago
|
||
Maybe it's a red herring, but the warning about the extra shutdown cycle collections might indicate we leaked some JS objects? (and if they're still alive, and they're still holding references to NSS objects, NSS won't shut down and we'll get that crash). There's not enough information here to be more specific than that.
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 16•5 years ago
|
||
Here some more details for the crash. As it looks like it's a long shutdown with a final crash:
https://treeherder.mozilla.org/logviewer?job_id=322733981&repo=autoland&lineNumber=3651-3655
[task 2020-11-25T10:34:35.921Z] 10:34:35 INFO - Hit MOZ_CRASH(Shutdown too long, probably frozen, causing a crash.) at /builds/worker/checkouts/gecko/toolkit/components/terminator/nsTerminator.cpp:221
[task 2020-11-25T10:34:36.040Z] 10:34:36 INFO - #01: PR_MD_UNLOCK[Z:\task_1606300105\build\application\firefox\nss3.dll +0x13ee51]
[task 2020-11-25T10:34:36.040Z] 10:34:36 INFO - #02: o_ceil[C:\Windows\System32\ucrtbase.dll +0x1c4be]
[task 2020-11-25T10:34:36.040Z] 10:34:36 INFO - #03: BaseThreadInitThunk[C:\Windows\System32\KERNEL32.DLL +0x13034]
[task 2020-11-25T10:34:36.040Z] 10:34:36 INFO - #04: RtlUserThreadStart[C:\Windows\SYSTEM32\ntdll.dll +0x71461]
Based on the stack it seems to be NSS releated, and only happening for debug builds.
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 20•5 years ago
|
||
In the last 7 days there have been 32 occurrences on windows7-32 and windows10-64 debug.
Recent failure: https://treeherder.mozilla.org/logviewer?job_id=325058607&repo=mozilla-central&lineNumber=3954
| Comment hidden (Intermittent Failures Robot) |
Comment 22•5 years ago
|
||
I wonder if we currently have problems with symbols on Windows, maybe specific to debug builds. The jobs download the symbols, but mozcrash is not able to find symbols when processing the crash. Gabriele, are you aware of something? It would be good to know which type of crash we are hitting here.
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 25•5 years ago
|
||
| unrelated | ||
Yes, we had a problem with system symbols not being scraped correctly. It was fixed in December and it slowly started picking up missing symbols again. I can try and manually repopulate symbols for that particular crash.
Comment 26•5 years ago
|
||
Sorry, I got it wrong, I was thinking of system symbols but this is an issue w/ Firefox ones. Double-checking what's wrong here.
Comment 27•5 years ago
|
||
These are MinGW builds, sadly we're not producing symbols properly for those yet. See bug 1580347 and bug 1621475.
| Comment hidden (Intermittent Failures Robot) |
Comment 29•5 years ago
|
||
MingW builds now support symbols but similar to normal builds on Windows we do not get stack traces reported for all the last test failures on Windows. Something might be still wrong with the symbols. Could you have another look at it Gabriele?
Comment 30•5 years ago
|
||
Sorry, I looked again and I was mistaken. The issue isn't MinGW specific. This crash is from a MinGW build and isn't symbolicated. But this one is also not symbolicated. In both cases we find symbols, download them and possibly unpack them somewhere but the stackwalker doesn't find them. This is probably a harness problem then, maybe we're just unpacking the symbols in the wrong place.
Comment 31•5 years ago
|
||
Hm, I don't see a specific failure message that mozcrash cannot find the downloaded symbols. Maybe we should add a specific failure message here if the path to symbols is not present.
Interesting are those lines:
https://treeherder.mozilla.org/logviewer?job_id=326333366&repo=autoland&lineNumber=127-132
The crashreporter symbols are unzipped into memory. As set by the harness symbols are always downloaded. In Mn jobs we only do that ondemand. So maybe it's a problem with --download-symbols=true.
Raphael, could you change the mozharness code for telemetry-client tests and change the download of symbols to ondemand? It would not only save us a small of amount of time, but also way lesser data to transfer. Those symbols are really only needed when a crash happens.
If that fixes the problem we should file a mozharness bug for download-symbols=true.
Comment 32•5 years ago
|
||
(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #31)
Interesting are those lines:
https://treeherder.mozilla.org/logviewer?job_id=326333366&repo=autoland&lineNumber=127-132
Ah, yes! That's clearly the issue. Symbols are being unpacked at this path: Z:\task_1610333867\build\symbols
But then we tell minidump_stackwalk to look for them here: c:\users\task_1610333867\appdata\local\temp\tmpklxvbn
https://treeherder.mozilla.org/logviewer?job_id=326333366&repo=autoland&lineNumber=4119
It seems to me that that temporary folder might have been created by this code. So mozcrash has not been given a path to look for the symbols and thinks it needs to fetch them by itself. It's likely that it's not given a URL either so in the end we symbolication doesn't work.
Comment 33•5 years ago
|
||
Hi folks!
(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #31)
Raphael, could you change the mozharness code for telemetry-client tests and change the download of symbols to ondemand? It would not only save us a small of amount of time, but also way lesser data to transfer. Those symbols are really only needed when a crash happens.
Looking into that now.
Comment 34•5 years ago
|
||
From what I can tell telemetry-tests-client is already running with --download-symbols ondemand.
See for example https://treeherder.mozilla.org/logviewer?job_id=326373441&repo=autoland
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Updated•4 years ago
|
Comment 69•4 years ago
|
||
Since the crash volume is low (less than 5 per week), the severity is downgraded to S3. Feel free to change it back if you think the bug is still critical.
For more information, please visit auto_nag documentation.
Description
•