1482957 - Android hardware tests @ Bitbar no longer downloading minidump_stackwalk

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Bob Clary [:bc] (inactive)]

I noticed during a try run today

https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&filter-tier=1&filter-tier=2&filter-tier=3&group_state=expanded&fromchange=efa24a23e5d1e4af76a208f6b463604affc511d7&selectedJob=193673688

https://treeherder.mozilla.org/logviewer.html#?job_id=193673688&repo=try&lineNumber=6563

that the workers @ bitbar are no longer downloading the minidump_stackwalk binary. This was working at least sometime in July but is working no longer.

Bug 1471541 changed the tooltool url android_hw is using:

-    "tooltool_servers": ['https://api.pub.build.mozilla.org/tooltool/'],
+    "tooltool_servers": ['https://tooltool.mozilla-releng.net/'],

That appears to be public still and not requiring authentication.

https://searchfox.org/mozilla-central/source/testing/mozharness/configs/android/android_hw.py has

    "download_minidump_stackwalk": True,
    "tooltool_servers": ['https://tooltool.mozilla-releng.net/'],
    # minidump_tooltool_manifest_path is relative to workspace/build/tests/
    "minidump_tooltool_manifest_path": "config/tooltool-manifests/linux64/releng.manifest",

I also noticed the relatively ancient bug 1306662 that the Docker workers have been baking minidump_stackwalk into the image.

Rok, Ted, Geoff:

Any thoughts on what happened or how I might fix it?

Has automatic downloading of minidump_stackwalk recently broken somehow?

Should I just bake minidump_stackwalk into the Bitbar containers like the Docker workers do?

Comment 1

[Geoff Brown [:gbrown]]

From your try run, I notice

   MINIDUMP_STACKWALK binary not found: /usr/local/bin/linux64-minidump_stackwalk

That's the path to the baked-in minidump_stackwalk used in the linux test docker image.

It probably comes from

https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/testing/mozharness/configs/android/android_common.py#52

I would think that would be okay: if download_minidump_stackwalk is True, I'd expect mozharness to try to download to that path...but I don't see any sign of that happening.

I'm not sure what's going wrong.

I've had a good experience using the baked-in minidump_stackwalk for android emulator unit tests: It eliminates the chance of a tooltool/network/server failure and shaves a second or two off of run time.

Comment 2

[Rok Garbas [:garbas]]

:bc I migrated tooltool service to new domain, but everything should be working as before. I will check later today (in ~4-5h) if I can manually download the minidump_stackwalk binary and can tell more then.

Comment 3

[(not currently active) Ted Mielczarek]

I don't know about the tooltool issues specifically, I haven't heard of anything similar, but:

> I also noticed the relatively ancient bug 1306662 that the Docker workers have been baking minidump_stackwalk into the image.

I had forgotten about this, but this means that that binary is the same as the one in tooltool anyway, so it probably doesn't matter which one you use. It doesn't get updated very frequently.

Comment 4

[Rok Garbas [:garbas]]

:bc would it be possible to get `config/tooltool-manifests/linux64/releng.manifest` to try to download minidump_stackwalk manually?

Comment 5

[(not currently active) Ted Mielczarek]

That's this file, btw: https://dxr.mozilla.org/mozilla-central/source/testing/config/tooltool-manifests/linux64/releng.manifest

Comment 6

[Bob Clary [:bc] (inactive)]

It should be https://searchfox.org/mozilla-central/source/testing/config/tooltool-manifests/linux64/releng.manifest

I cloned git clone [e-mail]:mozilla/build-tooltool.git and was able to get linux64-minidump_stackwalk via

python tooltool.py --manifest ~/mozilla/builds/inbound-taskcluster/mozilla/testing/config/tooltool-manifests/linux64/releng.manifest --url https://tooltool.mozilla-releng.net/ fetch

I have trouble getting the in-tree mozharness version of tooltool.py in testing/mozharness/mozharness/mozilla to run locally from the command line.

Comment 7

[Geoff Brown [:gbrown]]

Attachment: do not set self.minidump_stackwalk_path -- enable downloads

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2e05ce231e623c2d208c3c86a0189e869cf1875a&filter-tier=1&filter-tier=2&filter-tier=3

Comment 8

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] (pto Aug 20-Aug 24) from comment #7)
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=2e05ce231e623c2d208c3c86a0189e869cf1875a&filter-
> tier=1&filter-tier=2&filter-tier=3

https://treeherder.mozilla.org/logviewer.html#?job_id=194188339&repo=try&lineNumber=1874

22:12:59     INFO -  INFO - File linux64-minidump_stackwalk not present in local cache folder /builds/worker/tooltool_cache
22:12:59     INFO -  INFO - Attempting to fetch from 'https://tooltool.mozilla-releng.net/'...
22:13:00     INFO -  INFO - File linux64-minidump_stackwalk fetched from https://tooltool.mozilla-releng.net/ as /builds/worker/workspace/build/tmp_s_lM_
22:13:00     INFO -  INFO - File integrity verified, renaming tmp_s_lM_ to linux64-minidump_stackwalk
22:13:00     INFO -  INFO - Updating local cache /builds/worker/tooltool_cache...
22:13:00     INFO -  INFO - Local cache /builds/worker/tooltool_cache updated with linux64-minidump_stackwalk
22:13:00     INFO - Return code: 0
22:13:00  WARNING - minidump stackwalk path was given but couldn't be found. Tried looking in '/usr/local/bin/linux64-minidump_stackwalk'
22:13:00  WARNING - # TBPL WARNING #
22:13:00  WARNING - setting return code to 1

That warning seems wrong. Probably I need to modify the path as well...

Comment 9

[Geoff Brown [:gbrown]]

Attachment: do not set self.minidump_stackwalk_path -- enable downloads

https://treeherder.mozilla.org/#/jobs?repo=try&revision=99301bb6688dd9cd823189350ace5048c51192c0&filter-tier=1&filter-tier=2&filter-tier=3

https://treeherder.mozilla.org/logviewer.html#?job_id=194192742&repo=try&lineNumber=1855

22:47:00     INFO -  INFO - File linux64-minidump_stackwalk not present in local cache folder /builds/worker/tooltool_cache
22:47:00     INFO -  INFO - Attempting to fetch from 'https://tooltool.mozilla-releng.net/'...
22:47:01     INFO -  INFO - File linux64-minidump_stackwalk fetched from https://tooltool.mozilla-releng.net/ as /builds/worker/workspace/build/tmpbjgOvA
22:47:01     INFO -  INFO - File integrity verified, renaming tmpbjgOvA to linux64-minidump_stackwalk
22:47:01     INFO -  INFO - Updating local cache /builds/worker/tooltool_cache...
22:47:01     INFO -  INFO - Local cache /builds/worker/tooltool_cache updated with linux64-minidump_stackwalk
22:47:01     INFO - Return code: 0
22:47:01     INFO - Chmoding /builds/worker/workspace/build/linux64-minidump_stackwalk to 0755

Comment 10

[Bob Clary [:bc] (inactive)]

Comment on attachment 9001426 [details] [diff] [review]
do not set self.minidump_stackwalk_path -- enable downloads

Review of attachment 9001426 [details] [diff] [review]:
-----------------------------------------------------------------

I did a quick try run with gv and it got the minidump_stackwalk just fine. Thank you Geoff. You are the best.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/04521101cd23
Download minidump_stackwalk in android-hw tests; r=bc

Comment 12

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/04521101cd23