Closed
Bug 795206
Opened 12 years ago
Closed 12 years ago
Tegra crashes intermittently ending with "OSError: [Errno 8] Exec format error"
Categories
(Release Engineering :: General, defect)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: philor, Assigned: Callek)
References
Details
(Keywords: intermittent-failure)
Attachments
(4 files)
1.18 KB,
patch
|
coop
:
review+
Callek
:
checked-in+
|
Details | Diff | Splinter Review |
1.14 KB,
patch
|
coop
:
review+
Callek
:
checked-in+
|
Details | Diff | Splinter Review |
1.07 KB,
patch
|
mozilla
:
review+
Callek
:
checked-in+
|
Details | Diff | Splinter Review |
960 bytes,
patch
|
mozilla
:
review+
|
Details | Diff | Splinter Review |
Had to get stuck with one I'd have to file eventually.
https://tbpl.mozilla.org/php/getParsedLog.php?id=15613435&tree=Firefox
Android Tegra 250 mozilla-central opt test mochitest-2 on 2012-09-27 21:05:18 PDT for push 2d96ee8d9dd4
slave: tegra-332
INFO | automation.py | Application ran for: 0:09:19.877020
INFO | automation.py | Reading PID log: /tmp/tmpq5N2oGpidlog
getting files in '/mnt/sdcard/tests/profile/minidumps/'
Downloading symbols from: http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android/1348801536/fennec-18.0a1.en-US.android-arm.crashreporter-symbols.zip
PROCESS-CRASH | automation.py | application crashed (minidump found)
Crash dump filename: /tmp/tmpixB71N/7dadd65d-5857-96cf-7152ab38-710ace92.dmp
INFO | runtests.py | Received unexpected exception while running application
Traceback (most recent call last):
File "/builds/tegra-332/test/build/tests/mochitest/runtests.py", line 731, in runTests
timeout = timeout)
File "/builds/tegra-332/test/build/tests/mochitest/automation.py", line 1017, in runApp
self.checkForCrashes(profileDir, symbolsPath)
File "/builds/tegra-332/test/build/tests/mochitest/remoteautomation.py", line 79, in checkForCrashes
automationutils.checkForCrashes(dumpDir, symbolsPath, self.lastTestSeen)
File "/builds/tegra-332/test/build/tests/mochitest/automationutils.py", line 174, in checkForCrashes
stderr=subprocess.PIPE)
File "/tools/python27/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/tools/python27/lib/python2.7/subprocess.py", line 1228, in _execute_child
raise child_exception
OSError: [Errno 8] Exec format error
Comment 1•12 years ago
|
||
The failure is at:
{ # run minidump stackwalk
p = subprocess.Popen([stackwalkPath, d, symbolsPath],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
}
Ted, could this be due to the new binaries landed by bug 794122?
Comment 2•12 years ago
|
||
I think this is the problem that Callek mentioned offhand on IRC:
foopy_type: 'Linux'
The foopies run a mix of OS X and Linux, but the buildbot configs have them hardcoded to use the osx minidump_stackwalk binary. This error is a Linux foopy trying to run a Mac binary.
Assignee | ||
Comment 3•12 years ago
|
||
Yea, this is likely my issue.
The short story is this is an error in our ability to report a real error elsewhere, so any reports of this is us trying to dump a stack of a real crash.
Moving to releng and planning to tackle this.
Assignee: nobody → bugspam.Callek
Component: General → Release Engineering: Automation (General)
Product: Testing → mozilla.org
QA Contact: catlee
Version: Trunk → other
Reporter | ||
Comment 4•12 years ago
|
||
Reporter | ||
Comment 5•12 years ago
|
||
Reporter | ||
Comment 6•12 years ago
|
||
Reporter | ||
Comment 7•12 years ago
|
||
Reporter | ||
Comment 8•12 years ago
|
||
Reporter | ||
Comment 9•12 years ago
|
||
Reporter | ||
Comment 10•12 years ago
|
||
Reporter | ||
Comment 11•12 years ago
|
||
Comment 12•12 years ago
|
||
Reporter | ||
Comment 13•12 years ago
|
||
Reporter | ||
Comment 14•12 years ago
|
||
Reporter | ||
Comment 15•12 years ago
|
||
Reporter | ||
Comment 16•12 years ago
|
||
Reporter | ||
Comment 17•12 years ago
|
||
Reporter | ||
Comment 18•12 years ago
|
||
Reporter | ||
Comment 19•12 years ago
|
||
Reporter | ||
Comment 20•12 years ago
|
||
Reporter | ||
Comment 21•12 years ago
|
||
Reporter | ||
Comment 22•12 years ago
|
||
Reporter | ||
Comment 23•12 years ago
|
||
Reporter | ||
Comment 24•12 years ago
|
||
Reporter | ||
Comment 25•12 years ago
|
||
Reporter | ||
Comment 26•12 years ago
|
||
Little odd that it seems like this is always mochitest-2 crashes, and only mochitest-2 crashes.
https://tbpl.mozilla.org/php/getParsedLog.php?id=16210085&tree=Mozilla-Aurora
Reporter | ||
Comment 27•12 years ago
|
||
Reporter | ||
Comment 28•12 years ago
|
||
Reporter | ||
Comment 29•12 years ago
|
||
Reporter | ||
Comment 30•12 years ago
|
||
(In reply to Phil Ringnalda (:philor) from comment #26)
> Little odd that it seems like this is always mochitest-2 crashes, and only
> mochitest-2 crashes.
Ah, because we have a fairly frequent crash in svg/content/test/test_pointer-events.xhtml on the new tegras only, and the new tegras are mostly on the new foopies, and the new foopies are Linux.
Comment 31•12 years ago
|
||
Reporter | ||
Comment 32•12 years ago
|
||
I don't think Orange Factor actually, you know, *works*. I'd guess without having kept track that this is up to more like 20-30 failures per weekday, certainly not OF's 16 trunk, 17 total for the week (there was a push on m-a or m-b yesterday where I starred this twice on one *push*, so 1 off-trunk in the last week is certainly wrong).
Reporter | ||
Comment 33•12 years ago
|
||
Reporter | ||
Comment 34•12 years ago
|
||
Reporter | ||
Comment 35•12 years ago
|
||
Reporter | ||
Comment 36•12 years ago
|
||
Reporter | ||
Comment 37•12 years ago
|
||
Comment 38•12 years ago
|
||
Comment 39•12 years ago
|
||
Comment 40•12 years ago
|
||
Reporter | ||
Comment 41•12 years ago
|
||
Reporter | ||
Comment 42•12 years ago
|
||
Reporter | ||
Comment 43•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16622366&tree=Mozilla-Inbound (a bonus triple-crash!)
Reporter | ||
Comment 44•12 years ago
|
||
Reporter | ||
Comment 45•12 years ago
|
||
Reporter | ||
Comment 46•12 years ago
|
||
Reporter | ||
Comment 47•12 years ago
|
||
Reporter | ||
Comment 48•12 years ago
|
||
Reporter | ||
Comment 49•12 years ago
|
||
Reporter | ||
Comment 50•12 years ago
|
||
Reporter | ||
Comment 51•12 years ago
|
||
Reporter | ||
Comment 52•12 years ago
|
||
Reporter | ||
Comment 53•12 years ago
|
||
Reporter | ||
Comment 54•12 years ago
|
||
Reporter | ||
Comment 55•12 years ago
|
||
Reporter | ||
Comment 56•12 years ago
|
||
Reporter | ||
Comment 57•12 years ago
|
||
Comment 58•12 years ago
|
||
Callek, did you get a chance to look at this today? :-)
Assignee | ||
Comment 59•12 years ago
|
||
(In reply to Ed Morley [:edmorley UTC+0] from comment #58)
> Callek, did you get a chance to look at this today? :-)
I did I will post a patch when I'm done handing out candy for the neighborhood kids
Reporter | ||
Comment 60•12 years ago
|
||
Reporter | ||
Comment 61•12 years ago
|
||
Assignee | ||
Comment 62•12 years ago
|
||
Attachment #677258 -
Flags: review?(coop)
Assignee | ||
Comment 63•12 years ago
|
||
The equiv of this patch needs to be manually applied to mac foopies, doable with:
mkdir /Users/cltbld/bin; ln -s /builds/tools/breakpad/osx/minidump_stackwalk /Users/cltbld/bin/minidump_stackwalk
Attachment #677260 -
Flags: review?(coop)
Reporter | ||
Comment 64•12 years ago
|
||
Reporter | ||
Comment 65•12 years ago
|
||
Reporter | ||
Comment 66•12 years ago
|
||
Comment 67•12 years ago
|
||
Reporter | ||
Comment 68•12 years ago
|
||
Reporter | ||
Comment 69•12 years ago
|
||
Reporter | ||
Comment 70•12 years ago
|
||
Reporter | ||
Comment 71•12 years ago
|
||
Comment 72•12 years ago
|
||
Comment on attachment 677258 [details] [diff] [review]
[buildbotcustom] Use symlinked minidump prog
Review of attachment 677258 [details] [diff] [review]:
-----------------------------------------------------------------
r+ assuming the ~ gets expanded properly.
Attachment #677258 -
Flags: review?(coop) → review+
Updated•12 years ago
|
Attachment #677260 -
Flags: review?(coop) → review+
Reporter | ||
Comment 73•12 years ago
|
||
Reporter | ||
Comment 74•12 years ago
|
||
Reporter | ||
Comment 75•12 years ago
|
||
Assignee | ||
Comment 76•12 years ago
|
||
Comment on attachment 677260 [details] [diff] [review]
[puppet] Create the symlink to global-foopy-shared tools repo
Ran locally:
for i in 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 22 23 24; do ssh cltbld@foopy$i.build.mozilla.org "bash -l -c 'mkdir /Users/cltbld/bin; ln -s /builds/tools/breakpad/osx/minidump_stackwalk /Users/cltbld/bin/minidump_stackwalk'"; done
So all the mac's are done, also checked in this and saw some green puppet runs:
http://hg.mozilla.org/build/puppet/rev/87b60f4b626b
Attachment #677260 -
Flags: checked-in+
Reporter | ||
Comment 77•12 years ago
|
||
Assignee | ||
Comment 78•12 years ago
|
||
Comment on attachment 677258 [details] [diff] [review]
[buildbotcustom] Use symlinked minidump prog
Review of attachment 677258 [details] [diff] [review]:
-----------------------------------------------------------------
http://hg.mozilla.org/build/buildbotcustom/rev/57cf5697bc47
Attachment #677258 -
Flags: checked-in+
Reporter | ||
Comment 79•12 years ago
|
||
Reporter | ||
Comment 80•12 years ago
|
||
Assignee | ||
Comment 81•12 years ago
|
||
So it turns out we don't actually do the shell expansion where I thought we did here, so ~ wasn't expanding.
Therefore we need a loc that is the same on both mac foopies and linux foopies.
Attachment #677598 -
Flags: review?(aki)
Updated•12 years ago
|
Attachment #677598 -
Flags: review?(aki) → review+
Assignee | ||
Comment 82•12 years ago
|
||
(no need for tilde expansion)
Attachment #677600 -
Flags: review?(aki)
Updated•12 years ago
|
Attachment #677600 -
Flags: review?(aki) → review+
Assignee | ||
Comment 83•12 years ago
|
||
Comment on attachment 677598 [details] [diff] [review]
[puppet] move the minidump symlink
Review of attachment 677598 [details] [diff] [review]:
-----------------------------------------------------------------
$ for i in 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 22 23 24; do ssh cltbld@foopy$i.build.mozilla.org "ba
sh -l -c 'rm -rf /Users/cltbld/bin; ln -s /builds/tools/breakpad/osx/minidump_stackwalk /builds/minidump_stackwa
lk'"; done
And this was checked in:
http://hg.mozilla.org/build/puppet/rev/c504e817eec6
Attachment #677598 -
Flags: checked-in+
Assignee | ||
Comment 84•12 years ago
|
||
Confirmed this is now fixed. We shouldn't see this anymore.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Comment 85•12 years ago
|
||
Thank you for sorting this :-)
Updated•12 years ago
|
Keywords: intermittent-failure
Updated•12 years ago
|
Whiteboard: [orange]
Updated•12 years ago
|
Product: mozilla.org → Release Engineering
Updated•7 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•