1380200 - windows 7 mochitests from taskcluster don't have symbols for xul.dll

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

The Windows 7 mochitests run under taskcluster (currently tier 2) don't have xul.dll symbols (but they do have symbols for mozglue.dll).  The equivalent run under buildbot have working stacks.  Compare these occurrences of bug 1203928:

working (buildbot): all frames of stack look good
https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=112135812&lineNumber=3961

broken (taskcluster): top 3 frames of stack are fine, but below is bad
https://treeherder.mozilla.org/logviewer.html#?repo=oak&job_id=112474800&lineNumber=2800
https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=112791174&lineNumber=2958

Comment 1

[Gregory Szorc [:gps]]

Bug 1380381 is about OS X. I'm pretty confident the issues are not linked.

From the log (since they get pruned):

18:44:22     INFO -  mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/SyM4ERBIT2WdvcJ4LCkTiw/artifacts/public/build/target.crashreporter-symbols.zip
18:44:37     INFO -  mozcrash Copy/paste: Z:\task_1499538270\build\win32-minidump_stackwalk.exe c:\users\genericworker\appdata\local\temp\tmp6s4uzy.mozrunner\minidumps\8632bad4-108c-4bbb-aa7a-fac8d174918d.dmp c:\users\genericworker\appdata\local\temp\tmpwc7_kr
18:44:45     INFO -  mozcrash Saved minidump as Z:\task_1499538270\build\blobber_upload_dir\8632bad4-108c-4bbb-aa7a-fac8d174918d.dmp
18:44:45     INFO -  mozcrash Saved app info as Z:\task_1499538270\build\blobber_upload_dir\8632bad4-108c-4bbb-aa7a-fac8d174918d.extra
18:44:45  WARNING -  PROCESS-CRASH | dom/tests/browser/browser_xhr_sandbox.js | application crashed [@ mozalloc_abort(char const * const)]
18:44:45     INFO -  Crash dump filename: c:\users\genericworker\appdata\local\temp\tmp6s4uzy.mozrunner\minidumps\8632bad4-108c-4bbb-aa7a-fac8d174918d.dmp
18:44:45     INFO -  Operating system: Windows NT
18:44:45     INFO -                    6.1.7601 Service Pack 1
18:44:45     INFO -  CPU: x86
18:44:45     INFO -       GenuineIntel family 6 model 63 stepping 2
18:44:45     INFO -       8 CPUs
18:44:45     INFO -  GPU: UNKNOWN
18:44:45     INFO -  Crash reason:  EXCEPTION_BREAKPOINT
18:44:45     INFO -  Crash address: 0x65752c8a
18:44:45     INFO -  Process uptime: 106 seconds
18:44:45     INFO -  Thread 0 (crashed)
18:44:45     INFO -   0  mozglue.dll!mozalloc_abort(char const * const) [mozalloc_abort.cpp:6ca7dfedd992 : 33 + 0xa]
18:44:45     INFO -      eip = 0x65752c8a   esp = 0x002fdbb4   ebp = 0x002fdbb4   ebx = 0x00000000
18:44:45     INFO -      esi = 0x65768166   edi = 0x002fdbfe   eax = 0x00000000   ecx = 0x674a068f
18:44:45     INFO -      edx = 0x67545350   efl = 0x00000216
18:44:45     INFO -      Found by: given as instruction pointer in context
18:44:45     INFO -   1  mozglue.dll!mozalloc_handle_oom(unsigned int) [mozalloc_oom.cpp:6ca7dfedd992 : 54 + 0x9]
18:44:45     INFO -      eip = 0x65752cf4   esp = 0x002fdbbc   ebp = 0x002fdc04
18:44:45     INFO -      Found by: call frame info
18:44:45     INFO -   2  mozglue.dll!moz_xmalloc [mozalloc.cpp:6ca7dfedd992 : 85 + 0x8]
18:44:45     INFO -      eip = 0x65752d5d   esp = 0x002fdc0c   ebp = 0x002fdc10
18:44:45     INFO -      Found by: call frame info
18:44:45     INFO -   3  xul.dll + 0x6fe9f4
18:44:45     INFO -      eip = 0x5d0ee9f4   esp = 0x002fdc18   ebp = 0x002fdc20
18:44:45     INFO -      Found by: call frame info

The symbols archive is being downloaded. And the zip does have a xul.pdb directory with a xul.sym. So I'm not sure why the xul.dll symbols aren't being reported.

This is likely a needinfo for Ted. But he's on PTO until Monday and isn't accepting needinfo requests.

Comment 2

[Justin Wood (:Callek)]

Ted,

can you help here please? I should note that our nightly builds on date seem to have symbols for xul.dll in their tests (RyanVM helped out there) but the opt builds still don't.

I'm not sure what is wrong or how we fix it.

Comment 3

[Justin Wood (:Callek)]

recent nightly: https://treeherder.mozilla.org/logviewer.html#?job_id=116190982&repo=date
recent opt: https://treeherder.mozilla.org/logviewer.html#?job_id=116138426&repo=date

Comment 4

[(not currently active) Ted Mielczarek]

This appears to be a crash where the minidump was written without a debug_id for xul.dll. The symbols for libxul seem fine, but looking at the minidump with minidump_dump I see:
module[44]
MDRawModule
  base_of_image                   = 0x587c0000
  size_of_image                   = 0x3ec3000
  checksum                        = 0x3e5c2ec
  time_date_stamp                 = 0x59711920 2017-07-20 20:57:04
  module_name_rva                 = 0x5346
  version_info.signature          = 0xfeef04bd
  version_info.struct_version     = 0x10000
  version_info.file_version       = 0x380000:0x190a
  version_info.product_version    = 0x380000:0x190a
  version_info.file_flags_mask    = 0x3f
  version_info.file_flags         = 0x2
  version_info.file_os            = 0x4
  version_info.file_type          = 0x2
  version_info.file_subtype       = 0x0
  version_info.file_date          = 0x0:0x0
  cv_record.data_size             = 0
  cv_record.rva                   = 0x0
  misc_record.data_size           = 0
  misc_record.rva                 = 0x0
  (code_file)                     = "Z:\task_1500584905\build\application\firefox\xul.dll"
  (code_identifier)               = "597119203ec3000"
  (cv_record)                     = (null)
  (misc_record)                   = (null)
  (debug_file)                    = ""
  (debug_identifier)              = ""
  (version)                       = "56.0.0.6410"

Without the debug_identifier Breakpad can't locate the symbols. :-/ We have code that tries to reserve a block of memory so that this doesn't happen, but maybe it's not working in this case?

Comment 5

[(Away)]

> Without the debug_identifier Breakpad can't locate the symbols. :-/ We have
> code that tries to reserve a block of memory so that this doesn't happen,
> but maybe it's not working in this case?

We've probably outgrown the reservation size. I noted this in bug 1380758 but it might make sense to fix it here separately from that content process issue.

Comment 6

[Justin Wood (:Callek)]

11:59 AM <Callek> ted: soooo... what does 1380200 mean with regard to migration... is there something wrong in general with the test systems, with the symbols, with the build, all/any of the above
11:59 AM <Callek> ted: and is this likely a TC issue or is it a BB-too issue
12:02 PM <ted> Callek: i think it's just a weird crash, is all
12:02 PM <Callek> ted: so your "expert" opinion is to write it off as not a problem for migration? And is a weird issue but shouldn't block us?
12:03 PM — Callek tries to be explicit
12:05 PM <ted> Callek: well, we should try to get a good stack out of it. i don't have time for that today, but anyone who can use a windows debugger should be able to download the minidump + full-symbols.zip and do that
12:05 PM <ted> it might be a crash that's specific to the new configuration
12:06 PM <ted> once we get a stack hopefully we can fix it
12:06 PM <Callek> ted: but the symbol issue is unlikely specific to the new configuration, but the crash itself may be?
12:07 PM <ted> Callek: correct
12:07 PM <ted> Callek: the symbol issue is just an artifact of the crash
12:07 PM <ted> the symbols themselves look fine
12:07 PM <Callek> ok, thanks

Comment 7

[(Away)]

Attachment: breakpadsize

Callek is this something that can be verified on try?

Comment 8

[Justin Wood (:Callek)]

:dmajor the links in c#0 are from inbound/autoland, so yes. Much of the stuff on date will be landing within the week so you can either use the date project branch or wait for those pieces.

But either way, it should be testable on try, I have little knowledge over what test(s) fail or how to cause them to exhibit this beyond what is already in this bug though...

Comment 9

[(Away)]

Comment on attachment 8888840 [details] [diff] [review]
breakpadsize

We might as well just land this.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/72ad8966cd17
Increase Breakpad's memory reserve size due to xul.dll inflation. r=ted

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/72ad8966cd17