824984 - minidump_stackwalk is hardcoded as a binary for talos, we should use env['MINIDUMP_STACKWALK']

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

mochitests and other automation uses minidump_stackwalk from the tools directory:
http://hg.mozilla.org/build/tools/file/tip/breakpad/

We have outdated copies in the talos repository:
http://hg.mozilla.org/build/talos/file/d5cf61c4a4ec/talos/breakpad

inside buildbotcustom/factory.py, we set the environment variable minidump_stackwalk to a local clone of build/tools for the unittests:
http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py#l4736

maybe we could add it to the talos factory here:
http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py#l5291

we would need to modify talos to pull the location of the minidump_stackwalk binary from the environment instead of deriving it from the platform information:
http://hg.mozilla.org/build/talos/file/d5cf61c4a4ec/talos/ttest.py#l130

from what I can tell we would need to add the clone_buildtools step from the unittests to the talos factory:
http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py#l452

Comment 1

[(not currently active) Ted Mielczarek]

Can we just switch Talos to use mozcrash? That should handle the "use the env var" portion, so we'd just have to make the env var change to the factory.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

we would have to make all the changes in the buildbot-configs and buildbotcustom repos as mentioned above.  Using mozcrash would simplify a bit of talos, but we would need to make mozcrash work for remote scenarios where we pull dumps from a remote device.

I like the idea of mozcrash though.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: support mozcrash in talos (1.0)

once we have support from buildbot, this will work great.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

crap, mozcrash doesn't work with python 2.4 :(

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: allow mozcrash to support python 2.4 (1.0)

Comment 6

[Justin Wood (:Callek)]

Comment on attachment 696114 [details] [diff] [review]
allow mozcrash to support python 2.4 (1.0)

nit try: finally: to avoid dangling fHandle (especially if further up the chain catches Exceptions)

Comment 7

[Jeff Hammel]

Comment on attachment 696114 [details] [diff] [review]
allow mozcrash to support python 2.4 (1.0)

You kids and your fancy "with" statements. :sigh:

Comment 8

[Jeff Hammel]

Comment on attachment 696107 [details] [diff] [review]
support mozcrash in talos (1.0)

God I love deleted code.

That said, we can't quite do this.  Because mozcrash needs the attachment we'll need to push a new release to pypi first, so the version will be 0.2, not 0.1. So r+ given the version number change.... just make sure we have 0.2 on pypi and in the version spec (both setup.py and create_talos_zip.py, as usual)

Comment 9

[(not currently active) Ted Mielczarek]

Comment on attachment 696114 [details] [diff] [review]
allow mozcrash to support python 2.4 (1.0)

Horrible. Do we have any sort of timeframe for using a non-ancient Python for Talos? (Surely the test slaves already have a non-ancient Python, because those same slaves run our unittests, and those don't support Python 2.4.)

Comment 10

[Jeff Hammel]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #9)
> Comment on attachment 696114 [details] [diff] [review]
> allow mozcrash to support python 2.4 (1.0)
> 
> Horrible. Do we have any sort of timeframe for using a non-ancient Python
> for Talos? (Surely the test slaves already have a non-ancient Python,
> because those same slaves run our unittests, and those don't support Python
> 2.4.)

AIUI, the schedule is as follows:
- get talos on mozharness; was supposed to happen this quarter (as well as resolving the python 2.7 issue, for that matter) but priorities were elsewhere
- with mozharness, we can *actually* install pywin32 (I think?) in a sensible way (or as sensible as can be done with that particular package)
- with this, afaik, we will be on python 2.7! \o/ however, there is follow up work in https://bugzilla.mozilla.org/show_bug.cgi?id=726700 that should be done right after

My general feeling is that if we make this a priority (and we == A*Team + Releng here) we can probably knock this out right quick. If we don't make it a priority it will very slowly simmer in the background. Obviously there are other ways this can be done; I'm just reporting the consensus as I understand it.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: update binary bits for breakpad in talos [no windows] (1.0)

This bug has a few parts:

1) update binaries to get stackwalk working now
2) releng will work on adding breakpad binaries to tooltool and allowing talos to use it (bug 779037)
3) update mozcrash to work on python 2.4
4) deploy mozcrash 0.2
5) have talos support mozcrash (using local hardcoded binaries) instead of MINIDUMP_STACKWALK environment variable
6) when talos uses tooltool, we can remove the code to determine platform and binary location and just use os.env for the MINIDUMP_STACKWALK location.

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: update binary bits for breakpad in talos [windows] (1.0)

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: allow mozcrash to support python 2.4 (1.1)

update mozcrash to work properly on 2.4 and incorporate feedback from Callek.

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: support mozcrash in talos, use local binaries, mozcrash v0.2 (2.0)

updated patch for the future!

Comment 15

[William Lachance (:wlach)]

Comment on attachment 696485 [details] [diff] [review]
allow mozcrash to support python 2.4 (1.1)

Kind of sad that we had to remove the with statement. Oh well. 

Let's use the variable name "f" instead of "fHandle", since that's more pythonic (not to mention fitting in with the style in the rest of the file). r+ with that change.

Comment 16

[William Lachance (:wlach)]

Comment on attachment 696486 [details] [diff] [review]
support mozcrash in talos, use local binaries, mozcrash v0.2 (2.0)

Love it!

Comment 17

[William Lachance (:wlach)]

Comment on attachment 696484 [details] [diff] [review]
update binary bits for breakpad in talos [windows] (1.0)

If this works for you, I'm cool with it.

Comment 18

[William Lachance (:wlach)]

Comment on attachment 696483 [details] [diff] [review]
update binary bits for breakpad in talos [no windows] (1.0)

If this works for you, it's cool with me.

Comment 19

[Ed Morley [:emorley]]

Presuming bug 561235 can be duped to this? :-)

Comment 20

[Joel Maher ( :jmaher ) (UTC -8)]

live on inbound, will be on m-c on the next merge