Closed Bug 1482957 Opened 2 years ago Closed Last year
Android hardware tests @ Bitbar no longer downloading minidump
I noticed during a try run today https://firstname.lastname@example.org&filter-tier=1&filter-tier=2&filter-tier=3&group_state=expanded&fromchange=efa24a23e5d1e4af76a208f6b463604affc511d7&selectedJob=193673688 https://treeherder.mozilla.org/logviewer.html#?job_id=193673688&repo=try&lineNumber=6563 that the workers @ bitbar are no longer downloading the minidump_stackwalk binary. This was working at least sometime in July but is working no longer. Bug 1471541 changed the tooltool url android_hw is using: - "tooltool_servers": ['https://api.pub.build.mozilla.org/tooltool/'], + "tooltool_servers": ['https://tooltool.mozilla-releng.net/'], That appears to be public still and not requiring authentication. https://searchfox.org/mozilla-central/source/testing/mozharness/configs/android/android_hw.py has "download_minidump_stackwalk": True, "tooltool_servers": ['https://tooltool.mozilla-releng.net/'], # minidump_tooltool_manifest_path is relative to workspace/build/tests/ "minidump_tooltool_manifest_path": "config/tooltool-manifests/linux64/releng.manifest", I also noticed the relatively ancient bug 1306662 that the Docker workers have been baking minidump_stackwalk into the image. Rok, Ted, Geoff: Any thoughts on what happened or how I might fix it? Has automatic downloading of minidump_stackwalk recently broken somehow? Should I just bake minidump_stackwalk into the Bitbar containers like the Docker workers do?
From your try run, I notice MINIDUMP_STACKWALK binary not found: /usr/local/bin/linux64-minidump_stackwalk That's the path to the baked-in minidump_stackwalk used in the linux test docker image. It probably comes from https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/testing/mozharness/configs/android/android_common.py#52 I would think that would be okay: if download_minidump_stackwalk is True, I'd expect mozharness to try to download to that path...but I don't see any sign of that happening. I'm not sure what's going wrong. I've had a good experience using the baked-in minidump_stackwalk for android emulator unit tests: It eliminates the chance of a tooltool/network/server failure and shaves a second or two off of run time.
:bc I migrated tooltool service to new domain, but everything should be working as before. I will check later today (in ~4-5h) if I can manually download the minidump_stackwalk binary and can tell more then.
I don't know about the tooltool issues specifically, I haven't heard of anything similar, but: > I also noticed the relatively ancient bug 1306662 that the Docker workers have been baking minidump_stackwalk into the image. I had forgotten about this, but this means that that binary is the same as the one in tooltool anyway, so it probably doesn't matter which one you use. It doesn't get updated very frequently.
:bc would it be possible to get `config/tooltool-manifests/linux64/releng.manifest` to try to download minidump_stackwalk manually?
It should be https://searchfox.org/mozilla-central/source/testing/config/tooltool-manifests/linux64/releng.manifest I cloned git clone email@example.com:mozilla/build-tooltool.git and was able to get linux64-minidump_stackwalk via python tooltool.py --manifest ~/mozilla/builds/inbound-taskcluster/mozilla/testing/config/tooltool-manifests/linux64/releng.manifest --url https://tooltool.mozilla-releng.net/ fetch I have trouble getting the in-tree mozharness version of tooltool.py in testing/mozharness/mozharness/mozilla to run locally from the command line.
(In reply to Geoff Brown [:gbrown] (pto Aug 20-Aug 24) from comment #7) > https://treeherder.mozilla.org/#/ > jobs?repo=try&revision=2e05ce231e623c2d208c3c86a0189e869cf1875a&filter- > tier=1&filter-tier=2&filter-tier=3 https://treeherder.mozilla.org/logviewer.html#?job_id=194188339&repo=try&lineNumber=1874 22:12:59 INFO - INFO - File linux64-minidump_stackwalk not present in local cache folder /builds/worker/tooltool_cache 22:12:59 INFO - INFO - Attempting to fetch from 'https://tooltool.mozilla-releng.net/'... 22:13:00 INFO - INFO - File linux64-minidump_stackwalk fetched from https://tooltool.mozilla-releng.net/ as /builds/worker/workspace/build/tmp_s_lM_ 22:13:00 INFO - INFO - File integrity verified, renaming tmp_s_lM_ to linux64-minidump_stackwalk 22:13:00 INFO - INFO - Updating local cache /builds/worker/tooltool_cache... 22:13:00 INFO - INFO - Local cache /builds/worker/tooltool_cache updated with linux64-minidump_stackwalk 22:13:00 INFO - Return code: 0 22:13:00 WARNING - minidump stackwalk path was given but couldn't be found. Tried looking in '/usr/local/bin/linux64-minidump_stackwalk' 22:13:00 WARNING - # TBPL WARNING # 22:13:00 WARNING - setting return code to 1 That warning seems wrong. Probably I need to modify the path as well...
https://treeherder.mozilla.org/#/jobs?repo=try&revision=99301bb6688dd9cd823189350ace5048c51192c0&filter-tier=1&filter-tier=2&filter-tier=3 https://treeherder.mozilla.org/logviewer.html#?job_id=194192742&repo=try&lineNumber=1855 22:47:00 INFO - INFO - File linux64-minidump_stackwalk not present in local cache folder /builds/worker/tooltool_cache 22:47:00 INFO - INFO - Attempting to fetch from 'https://tooltool.mozilla-releng.net/'... 22:47:01 INFO - INFO - File linux64-minidump_stackwalk fetched from https://tooltool.mozilla-releng.net/ as /builds/worker/workspace/build/tmpbjgOvA 22:47:01 INFO - INFO - File integrity verified, renaming tmpbjgOvA to linux64-minidump_stackwalk 22:47:01 INFO - INFO - Updating local cache /builds/worker/tooltool_cache... 22:47:01 INFO - INFO - Local cache /builds/worker/tooltool_cache updated with linux64-minidump_stackwalk 22:47:01 INFO - Return code: 0 22:47:01 INFO - Chmoding /builds/worker/workspace/build/linux64-minidump_stackwalk to 0755
Comment on attachment 9001426 [details] [diff] [review] do not set self.minidump_stackwalk_path -- enable downloads Review of attachment 9001426 [details] [diff] [review]: ----------------------------------------------------------------- I did a quick try run with gv and it got the minidump_stackwalk just fine. Thank you Geoff. You are the best.
Attachment #9001426 - Flags: review?(bob) → review+
Pushed by firstname.lastname@example.org: https://hg.mozilla.org/integration/mozilla-inbound/rev/04521101cd23 Download minidump_stackwalk in android-hw tests; r=bc
You need to log in before you can comment on or make changes to this bug.