1348101 - tc win64 tests use win32-minidump_stackwalk.exe

Status: RESOLVED INVALID

(Testing :: General, enhancement)

Description

[Geoff Brown [:gbrown]]

Several current win64 Talos bugs have timeouts or crashes that fail to produce crash reports. I noticed that we are calling win32-minidump_stackwalk.exe in those jobs. Hopefully a 64 bit variant will work better.

Comment 1

[Geoff Brown [:gbrown]]

https://dxr.mozilla.org/mozilla-central/rev/ff04d410e74b69acfab17ef7e73e7397602d5a68/testing/mozharness/configs/releng_infra_configs/win64.py#3

https://dxr.mozilla.org/mozilla-central/rev/ff04d410e74b69acfab17ef7e73e7397602d5a68/testing/config/tooltool-manifests/win32/releng.manifest#7

Comment 2

[Geoff Brown [:gbrown]]

All the win64 tests pull win32-minidump_stackwalk.exe from tooltool.

Comment 3

[Geoff Brown [:gbrown]]

:ted - Do we need a 64-bit variant of minidump_stackwalk for win64; if so, where would that come from?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

I verified in mozinfo.info that we have:

windows8:
bits: 64
processor: x86_64

windows7:
bits: 32
processor: x86


this is both for mochitest and talos, they are identical.

looking in taskcluster this is hardcoded:
https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/unittests/win_unittest.py#276

and on linux it is conditional:
https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/talos/linux_config.py#8

but I noticed we do not have a tooltool entry for win64 minidump:
https://dxr.mozilla.org/mozilla-central/search?q=path%3Atooltool-manifests&redirect=true (at the bottom is testing/../win32/releng.manifest)

so I think the next step is to get a win64 stackwalk_minidump on tooltool- possibly we wait for ted to weigh in before hacking too much in case there is a reason for this?

Comment 5

[(not currently active) Ted Mielczarek]

(In reply to Geoff Brown [:gbrown] from comment #3)
> :ted - Do we need a 64-bit variant of minidump_stackwalk for win64; if so,
> where would that come from?

There should be no real reason to need a 64-bit minidump_stackwalk unless the 32-bit binary is OOMing or something silly like that. minidump_stackwalk is written specifically to handle minidumps from any platform (we process minidumps from Firefox on every OS/architecture on 64-bit Linux on crash-stats).

FWIW, those binaries were built in taskcluster using this script:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/scripts/misc/minidump_stackwalk.sh

All the binaries are cross-compiled from a single Linux taskcluster task, since a) we didn't have any non-Linux workers at the time and b) it's probably easier to wrangle the cross-compilation in this case than to deal with the headaches of native Windows/OS X compilation. If we really did need a win64 binary we'd have to update the Docker image to have the right 64-bit mingw toolchain and tweak the configure targets appropriately.

We could create a toolchain task to run that script nowadays, that would be handy for generating new binaries.

Comment 6

[(not currently active) Ted Mielczarek]

So as a first cut--do we get minidump files out of those Talos crashes at all? Even if they fail to produce a stack? For unittest jobs we upload the minidumps as test artifacts, do we do that for Talos? (If not we should fix that.) If so, we should be able to look at the minidumps and see if they have useful data first.

Comment 7

[Geoff Brown [:gbrown]]

Thanks Ted. I expect there's nothing else to do here then.

We usually get minidump files from those Windows 8 Talos crashes, and they appear as test artifacts, but minidump_stackwalk fails. Bug 1310638 (which you already found and commented on today) has lots of examples.