Tegra crashes intermittently ending with "OSError: [Errno 8] Exec format error"

RESOLVED FIXED

Status

RESOLVED FIXED
6 years ago
6 months ago

People

(Reporter: philor, Assigned: Callek)

Tracking

({intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(4 attachments)

(Reporter)

Description

6 years ago
Had to get stuck with one I'd have to file eventually.

https://tbpl.mozilla.org/php/getParsedLog.php?id=15613435&tree=Firefox
Android Tegra 250 mozilla-central opt test mochitest-2 on 2012-09-27 21:05:18 PDT for push 2d96ee8d9dd4
slave: tegra-332

INFO | automation.py | Application ran for: 0:09:19.877020
INFO | automation.py | Reading PID log: /tmp/tmpq5N2oGpidlog
getting files in '/mnt/sdcard/tests/profile/minidumps/'
Downloading symbols from: http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android/1348801536/fennec-18.0a1.en-US.android-arm.crashreporter-symbols.zip
PROCESS-CRASH | automation.py | application crashed (minidump found)
Crash dump filename: /tmp/tmpixB71N/7dadd65d-5857-96cf-7152ab38-710ace92.dmp
INFO | runtests.py | Received unexpected exception while running application
Traceback (most recent call last):
  File "/builds/tegra-332/test/build/tests/mochitest/runtests.py", line 731, in runTests
    timeout = timeout)
  File "/builds/tegra-332/test/build/tests/mochitest/automation.py", line 1017, in runApp
    self.checkForCrashes(profileDir, symbolsPath)
  File "/builds/tegra-332/test/build/tests/mochitest/remoteautomation.py", line 79, in checkForCrashes
    automationutils.checkForCrashes(dumpDir, symbolsPath, self.lastTestSeen)
  File "/builds/tegra-332/test/build/tests/mochitest/automationutils.py", line 174, in checkForCrashes
    stderr=subprocess.PIPE)
  File "/tools/python27/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/tools/python27/lib/python2.7/subprocess.py", line 1228, in _execute_child
    raise child_exception
OSError: [Errno 8] Exec format error

Comment 1

6 years ago
The failure is at:
{        # run minidump stackwalk
        p = subprocess.Popen([stackwalkPath, d, symbolsPath],
                             stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE)
}

Ted, could this be due to the new binaries landed by bug 794122?
I think this is the problem that Callek mentioned offhand on IRC:
foopy_type: 'Linux'

The foopies run a mix of OS X and Linux, but the buildbot configs have them hardcoded to use the osx minidump_stackwalk binary. This error is a Linux foopy trying to run a Mac binary.
(Assignee)

Comment 3

6 years ago
Yea, this is likely my issue.

The short story is this is an error in our ability to report a real error elsewhere, so any reports of this is us trying to dump a stack of a real crash.

Moving to releng and planning to tackle this.
Assignee: nobody → bugspam.Callek
Component: General → Release Engineering: Automation (General)
Product: Testing → mozilla.org
QA Contact: catlee
Version: Trunk → other
(Reporter)

Comment 26

6 years ago
Little odd that it seems like this is always mochitest-2 crashes, and only mochitest-2 crashes.

https://tbpl.mozilla.org/php/getParsedLog.php?id=16210085&tree=Mozilla-Aurora
(Reporter)

Comment 30

6 years ago
(In reply to Phil Ringnalda (:philor) from comment #26)
> Little odd that it seems like this is always mochitest-2 crashes, and only
> mochitest-2 crashes.

Ah, because we have a fairly frequent crash in svg/content/test/test_pointer-events.xhtml on the new tegras only, and the new tegras are mostly on the new foopies, and the new foopies are Linux.
(Reporter)

Comment 32

6 years ago
I don't think Orange Factor actually, you know, *works*. I'd guess without having kept track that this is up to more like 20-30 failures per weekday, certainly not OF's 16 trunk, 17 total for the week (there was a push on m-a or m-b yesterday where I starred this twice on one *push*, so 1 off-trunk in the last week is certainly wrong).
Callek, did you get a chance to look at this today? :-)
(Assignee)

Comment 59

6 years ago
(In reply to Ed Morley [:edmorley UTC+0] from comment #58)
> Callek, did you get a chance to look at this today? :-)

I did I will post a patch when I'm done handing out candy for the neighborhood kids
(Assignee)

Comment 62

6 years ago
Created attachment 677258 [details] [diff] [review]
[buildbotcustom] Use symlinked minidump prog
Attachment #677258 - Flags: review?(coop)
(Assignee)

Comment 63

6 years ago
Created attachment 677260 [details] [diff] [review]
[puppet] Create the symlink to global-foopy-shared tools repo

The equiv of this patch needs to be manually applied to mac foopies, doable with:

mkdir /Users/cltbld/bin; ln -s /builds/tools/breakpad/osx/minidump_stackwalk /Users/cltbld/bin/minidump_stackwalk
Attachment #677260 - Flags: review?(coop)
Comment on attachment 677258 [details] [diff] [review]
[buildbotcustom] Use symlinked minidump prog

Review of attachment 677258 [details] [diff] [review]:
-----------------------------------------------------------------

r+ assuming the ~ gets expanded properly.
Attachment #677258 - Flags: review?(coop) → review+
Attachment #677260 - Flags: review?(coop) → review+
(Assignee)

Comment 76

6 years ago
Comment on attachment 677260 [details] [diff] [review]
[puppet] Create the symlink to global-foopy-shared tools repo

Ran locally:

for i in 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 22 23 24; do ssh cltbld@foopy$i.build.mozilla.org "bash -l -c 'mkdir /Users/cltbld/bin; ln -s /builds/tools/breakpad/osx/minidump_stackwalk /Users/cltbld/bin/minidump_stackwalk'"; done

So all the mac's are done, also checked in this and saw some green puppet runs:

http://hg.mozilla.org/build/puppet/rev/87b60f4b626b
Attachment #677260 - Flags: checked-in+
(Assignee)

Comment 78

6 years ago
Comment on attachment 677258 [details] [diff] [review]
[buildbotcustom] Use symlinked minidump prog

Review of attachment 677258 [details] [diff] [review]:
-----------------------------------------------------------------

http://hg.mozilla.org/build/buildbotcustom/rev/57cf5697bc47
Attachment #677258 - Flags: checked-in+
(Assignee)

Comment 81

6 years ago
Created attachment 677598 [details] [diff] [review]
[puppet] move the minidump symlink

So it turns out we don't actually do the shell expansion where I  thought we did here, so ~ wasn't expanding.

Therefore we need a loc that is the same on both mac foopies and linux foopies.
Attachment #677598 - Flags: review?(aki)

Updated

6 years ago
Attachment #677598 - Flags: review?(aki) → review+
(Assignee)

Comment 82

6 years ago
Created attachment 677600 [details] [diff] [review]
[buildbotcustom] Use symlinked minidump prog in new location

 (no need for tilde expansion)
Attachment #677600 - Flags: review?(aki)

Updated

6 years ago
Attachment #677600 - Flags: review?(aki) → review+
(Assignee)

Comment 83

6 years ago
Comment on attachment 677598 [details] [diff] [review]
[puppet] move the minidump symlink

Review of attachment 677598 [details] [diff] [review]:
-----------------------------------------------------------------

$ for i in 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 22 23 24; do ssh cltbld@foopy$i.build.mozilla.org "ba
sh -l -c 'rm -rf /Users/cltbld/bin; ln -s /builds/tools/breakpad/osx/minidump_stackwalk /builds/minidump_stackwa
lk'"; done

And this was checked in:
http://hg.mozilla.org/build/puppet/rev/c504e817eec6
Attachment #677598 - Flags: checked-in+
(Assignee)

Comment 84

6 years ago
Confirmed this is now fixed. We shouldn't see this anymore.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Thank you for sorting this :-)
Keywords: intermittent-failure
Whiteboard: [orange]
Product: mozilla.org → Release Engineering
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.