Closed
Bug 1380200
Opened 7 years ago
Closed 7 years ago
windows 7 mochitests from taskcluster don't have symbols for xul.dll
Categories
(Firefox Build System :: General, enhancement)
Firefox Build System
General
Tracking
(firefox57 fixed)
RESOLVED
FIXED
mozilla57
Tracking | Status | |
---|---|---|
firefox57 | --- | fixed |
People
(Reporter: dbaron, Assigned: away)
Details
Attachments
(1 file)
1.09 KB,
patch
|
ted
:
review+
|
Details | Diff | Splinter Review |
The Windows 7 mochitests run under taskcluster (currently tier 2) don't have xul.dll symbols (but they do have symbols for mozglue.dll). The equivalent run under buildbot have working stacks. Compare these occurrences of bug 1203928: working (buildbot): all frames of stack look good https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=112135812&lineNumber=3961 broken (taskcluster): top 3 frames of stack are fine, but below is bad https://treeherder.mozilla.org/logviewer.html#?repo=oak&job_id=112474800&lineNumber=2800 https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=112791174&lineNumber=2958
Updated•7 years ago
|
Comment 1•7 years ago
|
||
Bug 1380381 is about OS X. I'm pretty confident the issues are not linked. From the log (since they get pruned): 18:44:22 INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/SyM4ERBIT2WdvcJ4LCkTiw/artifacts/public/build/target.crashreporter-symbols.zip 18:44:37 INFO - mozcrash Copy/paste: Z:\task_1499538270\build\win32-minidump_stackwalk.exe c:\users\genericworker\appdata\local\temp\tmp6s4uzy.mozrunner\minidumps\8632bad4-108c-4bbb-aa7a-fac8d174918d.dmp c:\users\genericworker\appdata\local\temp\tmpwc7_kr 18:44:45 INFO - mozcrash Saved minidump as Z:\task_1499538270\build\blobber_upload_dir\8632bad4-108c-4bbb-aa7a-fac8d174918d.dmp 18:44:45 INFO - mozcrash Saved app info as Z:\task_1499538270\build\blobber_upload_dir\8632bad4-108c-4bbb-aa7a-fac8d174918d.extra 18:44:45 WARNING - PROCESS-CRASH | dom/tests/browser/browser_xhr_sandbox.js | application crashed [@ mozalloc_abort(char const * const)] 18:44:45 INFO - Crash dump filename: c:\users\genericworker\appdata\local\temp\tmp6s4uzy.mozrunner\minidumps\8632bad4-108c-4bbb-aa7a-fac8d174918d.dmp 18:44:45 INFO - Operating system: Windows NT 18:44:45 INFO - 6.1.7601 Service Pack 1 18:44:45 INFO - CPU: x86 18:44:45 INFO - GenuineIntel family 6 model 63 stepping 2 18:44:45 INFO - 8 CPUs 18:44:45 INFO - GPU: UNKNOWN 18:44:45 INFO - Crash reason: EXCEPTION_BREAKPOINT 18:44:45 INFO - Crash address: 0x65752c8a 18:44:45 INFO - Process uptime: 106 seconds 18:44:45 INFO - Thread 0 (crashed) 18:44:45 INFO - 0 mozglue.dll!mozalloc_abort(char const * const) [mozalloc_abort.cpp:6ca7dfedd992 : 33 + 0xa] 18:44:45 INFO - eip = 0x65752c8a esp = 0x002fdbb4 ebp = 0x002fdbb4 ebx = 0x00000000 18:44:45 INFO - esi = 0x65768166 edi = 0x002fdbfe eax = 0x00000000 ecx = 0x674a068f 18:44:45 INFO - edx = 0x67545350 efl = 0x00000216 18:44:45 INFO - Found by: given as instruction pointer in context 18:44:45 INFO - 1 mozglue.dll!mozalloc_handle_oom(unsigned int) [mozalloc_oom.cpp:6ca7dfedd992 : 54 + 0x9] 18:44:45 INFO - eip = 0x65752cf4 esp = 0x002fdbbc ebp = 0x002fdc04 18:44:45 INFO - Found by: call frame info 18:44:45 INFO - 2 mozglue.dll!moz_xmalloc [mozalloc.cpp:6ca7dfedd992 : 85 + 0x8] 18:44:45 INFO - eip = 0x65752d5d esp = 0x002fdc0c ebp = 0x002fdc10 18:44:45 INFO - Found by: call frame info 18:44:45 INFO - 3 xul.dll + 0x6fe9f4 18:44:45 INFO - eip = 0x5d0ee9f4 esp = 0x002fdc18 ebp = 0x002fdc20 18:44:45 INFO - Found by: call frame info The symbols archive is being downloaded. And the zip does have a xul.pdb directory with a xul.sym. So I'm not sure why the xul.dll symbols aren't being reported. This is likely a needinfo for Ted. But he's on PTO until Monday and isn't accepting needinfo requests.
See Also: 1380381 →
Comment 2•7 years ago
|
||
Ted, can you help here please? I should note that our nightly builds on date seem to have symbols for xul.dll in their tests (RyanVM helped out there) but the opt builds still don't. I'm not sure what is wrong or how we fix it.
Flags: needinfo?(ted)
Comment 3•7 years ago
|
||
recent nightly: https://treeherder.mozilla.org/logviewer.html#?job_id=116190982&repo=date recent opt: https://treeherder.mozilla.org/logviewer.html#?job_id=116138426&repo=date
Comment 4•7 years ago
|
||
This appears to be a crash where the minidump was written without a debug_id for xul.dll. The symbols for libxul seem fine, but looking at the minidump with minidump_dump I see: module[44] MDRawModule base_of_image = 0x587c0000 size_of_image = 0x3ec3000 checksum = 0x3e5c2ec time_date_stamp = 0x59711920 2017-07-20 20:57:04 module_name_rva = 0x5346 version_info.signature = 0xfeef04bd version_info.struct_version = 0x10000 version_info.file_version = 0x380000:0x190a version_info.product_version = 0x380000:0x190a version_info.file_flags_mask = 0x3f version_info.file_flags = 0x2 version_info.file_os = 0x4 version_info.file_type = 0x2 version_info.file_subtype = 0x0 version_info.file_date = 0x0:0x0 cv_record.data_size = 0 cv_record.rva = 0x0 misc_record.data_size = 0 misc_record.rva = 0x0 (code_file) = "Z:\task_1500584905\build\application\firefox\xul.dll" (code_identifier) = "597119203ec3000" (cv_record) = (null) (misc_record) = (null) (debug_file) = "" (debug_identifier) = "" (version) = "56.0.0.6410" Without the debug_identifier Breakpad can't locate the symbols. :-/ We have code that tries to reserve a block of memory so that this doesn't happen, but maybe it's not working in this case?
Flags: needinfo?(ted)
> Without the debug_identifier Breakpad can't locate the symbols. :-/ We have > code that tries to reserve a block of memory so that this doesn't happen, > but maybe it's not working in this case? We've probably outgrown the reservation size. I noted this in bug 1380758 but it might make sense to fix it here separately from that content process issue.
Comment 6•7 years ago
|
||
11:59 AM <Callek> ted: soooo... what does 1380200 mean with regard to migration... is there something wrong in general with the test systems, with the symbols, with the build, all/any of the above 11:59 AM <Callek> ted: and is this likely a TC issue or is it a BB-too issue 12:02 PM <ted> Callek: i think it's just a weird crash, is all 12:02 PM <Callek> ted: so your "expert" opinion is to write it off as not a problem for migration? And is a weird issue but shouldn't block us? 12:03 PM — Callek tries to be explicit 12:05 PM <ted> Callek: well, we should try to get a good stack out of it. i don't have time for that today, but anyone who can use a windows debugger should be able to download the minidump + full-symbols.zip and do that 12:05 PM <ted> it might be a crash that's specific to the new configuration 12:06 PM <ted> once we get a stack hopefully we can fix it 12:06 PM <Callek> ted: but the symbol issue is unlikely specific to the new configuration, but the crash itself may be? 12:07 PM <ted> Callek: correct 12:07 PM <ted> Callek: the symbol issue is just an artifact of the crash 12:07 PM <ted> the symbols themselves look fine 12:07 PM <Callek> ok, thanks
Callek is this something that can be verified on try?
Attachment #8888840 -
Flags: feedback?(bugspam.Callek)
Comment 8•7 years ago
|
||
:dmajor the links in c#0 are from inbound/autoland, so yes. Much of the stuff on date will be landing within the week so you can either use the date project branch or wait for those pieces. But either way, it should be testable on try, I have little knowledge over what test(s) fail or how to cause them to exhibit this beyond what is already in this bug though...
Updated•7 years ago
|
Attachment #8888840 -
Flags: feedback?(bugspam.Callek)
Comment on attachment 8888840 [details] [diff] [review] breakpadsize We might as well just land this.
Attachment #8888840 -
Flags: review?(ted)
Updated•7 years ago
|
Attachment #8888840 -
Flags: review?(ted) → review+
Assignee: nobody → dmajor
Keywords: checkin-needed
Comment 10•7 years ago
|
||
Pushed by ryanvm@gmail.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/72ad8966cd17 Increase Breakpad's memory reserve size due to xul.dll inflation. r=ted
Keywords: checkin-needed
Comment 11•7 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/72ad8966cd17
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
Updated•6 years ago
|
Product: Core → Firefox Build System
You need to log in
before you can comment on or make changes to this bug.
Description
•