Closed Bug 1482957 Opened 2 years ago Closed Last year

Android hardware tests @ Bitbar no longer downloading minidump_stackwalk

Categories

(Testing :: General, defect)

defect
Not set

Tracking

(firefox63 fixed)

RESOLVED FIXED
mozilla63
Tracking Status
firefox63 --- fixed

People

(Reporter: bc, Assigned: bc)

References

Details

Attachments

(1 file, 1 obsolete file)

I noticed during a try run today

https://treeherder.mozilla.org/#/jobs?repo=try&author=bclary@mozilla.com&filter-tier=1&filter-tier=2&filter-tier=3&group_state=expanded&fromchange=efa24a23e5d1e4af76a208f6b463604affc511d7&selectedJob=193673688

https://treeherder.mozilla.org/logviewer.html#?job_id=193673688&repo=try&lineNumber=6563

that the workers @ bitbar are no longer downloading the minidump_stackwalk binary. This was working at least sometime in July but is working no longer.

Bug 1471541 changed the tooltool url android_hw is using:

-    "tooltool_servers": ['https://api.pub.build.mozilla.org/tooltool/'],
+    "tooltool_servers": ['https://tooltool.mozilla-releng.net/'],

That appears to be public still and not requiring authentication.

https://searchfox.org/mozilla-central/source/testing/mozharness/configs/android/android_hw.py has

    "download_minidump_stackwalk": True,
    "tooltool_servers": ['https://tooltool.mozilla-releng.net/'],
    # minidump_tooltool_manifest_path is relative to workspace/build/tests/
    "minidump_tooltool_manifest_path": "config/tooltool-manifests/linux64/releng.manifest",

I also noticed the relatively ancient bug 1306662 that the Docker workers have been baking minidump_stackwalk into the image.

Rok, Ted, Geoff:

Any thoughts on what happened or how I might fix it?

Has automatic downloading of minidump_stackwalk recently broken somehow?

Should I just bake minidump_stackwalk into the Bitbar containers like the Docker workers do?
Flags: needinfo?(ted)
Flags: needinfo?(rgarbas)
Flags: needinfo?(gbrown)
From your try run, I notice

   MINIDUMP_STACKWALK binary not found: /usr/local/bin/linux64-minidump_stackwalk

That's the path to the baked-in minidump_stackwalk used in the linux test docker image.

It probably comes from

https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/testing/mozharness/configs/android/android_common.py#52

I would think that would be okay: if download_minidump_stackwalk is True, I'd expect mozharness to try to download to that path...but I don't see any sign of that happening.

I'm not sure what's going wrong.

I've had a good experience using the baked-in minidump_stackwalk for android emulator unit tests: It eliminates the chance of a tooltool/network/server failure and shaves a second or two off of run time.
Flags: needinfo?(gbrown)
:bc I migrated tooltool service to new domain, but everything should be working as before. I will check later today (in ~4-5h) if I can manually download the minidump_stackwalk binary and can tell more then.
I don't know about the tooltool issues specifically, I haven't heard of anything similar, but:

> I also noticed the relatively ancient bug 1306662 that the Docker workers have been baking minidump_stackwalk into the image.

I had forgotten about this, but this means that that binary is the same as the one in tooltool anyway, so it probably doesn't matter which one you use. It doesn't get updated very frequently.
Flags: needinfo?(ted)
:bc would it be possible to get `config/tooltool-manifests/linux64/releng.manifest` to try to download minidump_stackwalk manually?
Flags: needinfo?(rgarbas)
It should be https://searchfox.org/mozilla-central/source/testing/config/tooltool-manifests/linux64/releng.manifest

I cloned git clone git@github.com:mozilla/build-tooltool.git and was able to get linux64-minidump_stackwalk via

python tooltool.py --manifest ~/mozilla/builds/inbound-taskcluster/mozilla/testing/config/tooltool-manifests/linux64/releng.manifest --url https://tooltool.mozilla-releng.net/ fetch

I have trouble getting the in-tree mozharness version of tooltool.py in testing/mozharness/mozharness/mozilla to run locally from the command line.
(In reply to Geoff Brown [:gbrown] (pto Aug 20-Aug 24) from comment #7)
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=2e05ce231e623c2d208c3c86a0189e869cf1875a&filter-
> tier=1&filter-tier=2&filter-tier=3

https://treeherder.mozilla.org/logviewer.html#?job_id=194188339&repo=try&lineNumber=1874

22:12:59     INFO -  INFO - File linux64-minidump_stackwalk not present in local cache folder /builds/worker/tooltool_cache
22:12:59     INFO -  INFO - Attempting to fetch from 'https://tooltool.mozilla-releng.net/'...
22:13:00     INFO -  INFO - File linux64-minidump_stackwalk fetched from https://tooltool.mozilla-releng.net/ as /builds/worker/workspace/build/tmp_s_lM_
22:13:00     INFO -  INFO - File integrity verified, renaming tmp_s_lM_ to linux64-minidump_stackwalk
22:13:00     INFO -  INFO - Updating local cache /builds/worker/tooltool_cache...
22:13:00     INFO -  INFO - Local cache /builds/worker/tooltool_cache updated with linux64-minidump_stackwalk
22:13:00     INFO - Return code: 0
22:13:00  WARNING - minidump stackwalk path was given but couldn't be found. Tried looking in '/usr/local/bin/linux64-minidump_stackwalk'
22:13:00  WARNING - # TBPL WARNING #
22:13:00  WARNING - setting return code to 1

That warning seems wrong. Probably I need to modify the path as well...
Attachment #9001405 - Flags: review?(bob)
https://treeherder.mozilla.org/#/jobs?repo=try&revision=99301bb6688dd9cd823189350ace5048c51192c0&filter-tier=1&filter-tier=2&filter-tier=3

https://treeherder.mozilla.org/logviewer.html#?job_id=194192742&repo=try&lineNumber=1855

22:47:00     INFO -  INFO - File linux64-minidump_stackwalk not present in local cache folder /builds/worker/tooltool_cache
22:47:00     INFO -  INFO - Attempting to fetch from 'https://tooltool.mozilla-releng.net/'...
22:47:01     INFO -  INFO - File linux64-minidump_stackwalk fetched from https://tooltool.mozilla-releng.net/ as /builds/worker/workspace/build/tmpbjgOvA
22:47:01     INFO -  INFO - File integrity verified, renaming tmpbjgOvA to linux64-minidump_stackwalk
22:47:01     INFO -  INFO - Updating local cache /builds/worker/tooltool_cache...
22:47:01     INFO -  INFO - Local cache /builds/worker/tooltool_cache updated with linux64-minidump_stackwalk
22:47:01     INFO - Return code: 0
22:47:01     INFO - Chmoding /builds/worker/workspace/build/linux64-minidump_stackwalk to 0755
Attachment #9001405 - Attachment is obsolete: true
Attachment #9001426 - Flags: review?(bob)
Comment on attachment 9001426 [details] [diff] [review]
do not set self.minidump_stackwalk_path -- enable downloads

Review of attachment 9001426 [details] [diff] [review]:
-----------------------------------------------------------------

I did a quick try run with gv and it got the minidump_stackwalk just fine. Thank you Geoff. You are the best.
Attachment #9001426 - Flags: review?(bob) → review+
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/04521101cd23
Download minidump_stackwalk in android-hw tests; r=bc
https://hg.mozilla.org/mozilla-central/rev/04521101cd23
Status: ASSIGNED → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
Blocks: 1483695
See Also: → 1609754
You need to log in before you can comment on or make changes to this bug.