Closed Bug 1374221 Opened 2 years ago Closed 2 years ago

MINIDUMP_STACKWALK binary not found: /usr/local/bin/linux64-minidump_stackwalk

Categories

(Testing :: General, defect, major)

52 Branch
Unspecified
macOS
defect
Not set
major

Tracking

(firefox-esr52 unaffected, firefox54 unaffected, firefox55 fixed, firefox56 fixed)

RESOLVED FIXED
mozilla56
Tracking Status
firefox-esr52 --- unaffected
firefox54 --- unaffected
firefox55 --- fixed
firefox56 --- fixed

People

(Reporter: whimboo, Assigned: wcosta)

References

Details

(Keywords: regression)

Attachments

(1 file)

Since the migration of tests from Buildbot to TaskCluster no crash is getting analyzed anymore at least for Marionette jobs. Reason is:

> MINIDUMP_STACKWALK binary not found: /usr/local/bin/linux64-minidump_stackwalk

And that happens because of:

https://dxr.mozilla.org/mozilla-central/rev/95543bdc59bd038a3d5d084b85a4fec493c349ee/taskcluster/ci/test/tests.yml#424-433

Specifically the inclusion of 'remove_executables.py' for the mozharness config, which is only allowed on Linux.

As it looks like for other job types there is an entry and the list of config inclusions has been reverted:

                macosx.*:
                    - remove_executables.py
                    - unittests/mac_unittest.py
Not sure what has all been done for other test suites. Costa could you help?
Flags: needinfo?(wcosta)
Flags: needinfo?(wcosta)
Duplicate of this bug: 1374073
Assignee: nobody → wcosta
Status: NEW → ASSIGNED
Comment on attachment 8879122 [details]
Bug 1374221: Remove "remove_executables.py" from mac tests.

https://reviewboard.mozilla.org/r/150450/#review155186

Checking the logs of Marionette tests the change seems to look fine, but I'm not a qualified reviewer for your code changes under taskcluster. Please find one, who can do it for all the test jobs you are modifying.
Attachment #8879122 - Flags: review?(hskupin)
Component: Marionette → General
Attachment #8879122 - Flags: review?(garndt) → review?(dustin)
We get lots of crashes which cannot be analyzed momentarily. Can we get someone else beside Dustin to review this? It's getting more important.
Severity: normal → major
OS: Unspecified → Mac OS X
Attachment #8879122 - Flags: review?(dustin) → review?(jmaher)
I checked out the patch and I do not see anything wrong with how it can break in-tree scheduling, but I do not know enough about the mozharness configs to know how it changes the environment and impacts other jobs.  Someone from releng or those that know mozharness more might be able to answer that.  I'm not sure if dustin would know how these mozharness scripts would impact the jobs either, it's a bit out of our comfort/knowledge zone.

Looking at treeherder for the try push on this patch I see a few failures but I am not sure if they are intermittents or perma-failures right now.  I retriggered them and we should look at the results.
catlee, do you know of someone that understands the implications of these mozharness configuration changes?
Flags: needinfo?(catlee)
The patch LGTM, as far as removing the 'remove_executables' config for OSX.
Flags: needinfo?(catlee)
Comment on attachment 8879122 [details]
Bug 1374221: Remove "remove_executables.py" from mac tests.

https://reviewboard.mozilla.org/r/150450/#review155656

In fact, I had to do the same thing to run Linux talos on hardware -- https://reviewboard.mozilla.org/r/142802/diff/1#index_header

If this is working for you, great, but I wonder if you'll have problems finding tooltool, since it's not in PATH.
Attachment #8879122 - Flags: review+
Comment on attachment 8879122 [details]
Bug 1374221: Remove "remove_executables.py" from mac tests.

https://reviewboard.mozilla.org/r/150450/#review155668

so many places to edit for remove_executables.py.  I did not ensure that every possibly test that we run on osx was touched, I am going to assume that was done here already.
Attachment #8879122 - Flags: review?(jmaher) → review+
Comment on attachment 8879122 [details]
Bug 1374221: Remove "remove_executables.py" from mac tests.

https://reviewboard.mozilla.org/r/150450/#review155676

::: taskcluster/ci/test/tests.yml:1370
(Diff revisions 3 - 4)
>      treeherder-symbol: tc-T(g3)
>      run-on-projects:
>          by-test-platform:
>              linux64-stylo/.*: ['mozilla-central', 'try']
>              default: ['mozilla-beta', 'mozilla-aurora', 'mozilla-central', 'mozilla-inbound', 'autoland', 'try'] 
> -    max-run-time: 7200
> +    max-run-time: 3600

Why did we lower the timeout here?
Attachment #8879122 - Flags: review?(garndt) → review+
We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again.

hg error in cmd: hg rebase -s 0cfac182143d -d 41c595eb1a0b: rebasing 402923:0cfac182143d "Bug 1374221: Remove "remove_executables.py" from mac tests. r=dustin,garndt,jmaher" (tip)
merging taskcluster/ci/test/tests.yml
warning: conflicts while merging taskcluster/ci/test/tests.yml! (edit, then use 'hg resolve --mark')
unresolved conflicts (see hg resolve, then hg rebase --continue)
(In reply to Greg Arndt [:garndt] from comment #14)
> Comment on attachment 8879122 [details]
> Bug 1374221: Remove "remove_executables.py" from mac tests.
> 
> https://reviewboard.mozilla.org/r/150450/#review155676
> 
> ::: taskcluster/ci/test/tests.yml:1370
> (Diff revisions 3 - 4)
> >      treeherder-symbol: tc-T(g3)
> >      run-on-projects:
> >          by-test-platform:
> >              linux64-stylo/.*: ['mozilla-central', 'try']
> >              default: ['mozilla-beta', 'mozilla-aurora', 'mozilla-central', 'mozilla-inbound', 'autoland', 'try'] 
> > -    max-run-time: 7200
> > +    max-run-time: 3600
> 
> Why did we lower the timeout here?

I mistakely bumped it in talos-g3 instead of talos-g2
We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again.

hg error in cmd: hg rebase -s 05044fc1f5ca -d 3d1f8cbf8836: rebasing 402938:05044fc1f5ca "Bug 1374221: Remove "remove_executables.py" from mac tests. r=dustin,garndt,jmaher" (tip)
merging taskcluster/ci/test/tests.yml
warning: conflicts while merging taskcluster/ci/test/tests.yml! (edit, then use 'hg resolve --mark')
unresolved conflicts (see hg resolve, then hg rebase --continue)
Wander, I assume my push from earlier today is causing those merge issues for you:
https://hg.mozilla.org/integration/autoland/rev/79f9200b6cc1
(In reply to Henrik Skupin (:whimboo) from comment #19)
> Wander, I assume my push from earlier today is causing those merge issues
> for you:
> https://hg.mozilla.org/integration/autoland/rev/79f9200b6cc1

In this case, I wonder how to rebase it? Wait for m-c merge?
Sadly, yes. :/ But you can do it now given that the other bug has been merged to m-c now.
We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again.

hg error in cmd: hg rebase -s e3ad20237ab9 -d f865971d8357: rebasing 403171:e3ad20237ab9 "Bug 1374221: Remove "remove_executables.py" from mac tests. r=dustin,garndt,jmaher" (tip)
merging taskcluster/ci/test/tests.yml
warning: conflicts while merging taskcluster/ci/test/tests.yml! (edit, then use 'hg resolve --mark')
unresolved conflicts (see hg resolve, then hg rebase --continue)
Pushed by wcosta@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6f795df6dc6f
Remove "remove_executables.py" from mac tests. r=dustin,garndt,jmaher
https://hg.mozilla.org/mozilla-central/rev/6f795df6dc6f
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
Wander, I assume this patch also needs an uplift?
Flags: needinfo?(wcosta)
(In reply to Henrik Skupin (:whimboo) from comment #27)
> Wander, I assume this patch also needs an uplift?

I am not sure, I guess so.
Flags: needinfo?(wcosta)
On which branches we use TC workers for OS X now? Are those central and beta? If yes, than at least one uplift is necessary.
OS X tasks are running in tc on inbound/autoland/try/central/beta (so yes, an uplift to beta)
Whiteboard: [checkin-needed-beta]
You need to log in before you can comment on or make changes to this bug.