Closed
Bug 795206
Opened 12 years ago
Closed 12 years ago
Tegra crashes intermittently ending with "OSError: [Errno 8] Exec format error"
Categories
(Release Engineering :: General, defect)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: philor, Assigned: Callek)
References
Details
(Keywords: intermittent-failure)
Attachments
(4 files)
1.18 KB,
patch
|
coop
:
review+
Callek
:
checked-in+
|
Details | Diff | Splinter Review |
1.14 KB,
patch
|
coop
:
review+
Callek
:
checked-in+
|
Details | Diff | Splinter Review |
1.07 KB,
patch
|
mozilla
:
review+
Callek
:
checked-in+
|
Details | Diff | Splinter Review |
960 bytes,
patch
|
mozilla
:
review+
|
Details | Diff | Splinter Review |
Had to get stuck with one I'd have to file eventually. https://tbpl.mozilla.org/php/getParsedLog.php?id=15613435&tree=Firefox Android Tegra 250 mozilla-central opt test mochitest-2 on 2012-09-27 21:05:18 PDT for push 2d96ee8d9dd4 slave: tegra-332 INFO | automation.py | Application ran for: 0:09:19.877020 INFO | automation.py | Reading PID log: /tmp/tmpq5N2oGpidlog getting files in '/mnt/sdcard/tests/profile/minidumps/' Downloading symbols from: http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android/1348801536/fennec-18.0a1.en-US.android-arm.crashreporter-symbols.zip PROCESS-CRASH | automation.py | application crashed (minidump found) Crash dump filename: /tmp/tmpixB71N/7dadd65d-5857-96cf-7152ab38-710ace92.dmp INFO | runtests.py | Received unexpected exception while running application Traceback (most recent call last): File "/builds/tegra-332/test/build/tests/mochitest/runtests.py", line 731, in runTests timeout = timeout) File "/builds/tegra-332/test/build/tests/mochitest/automation.py", line 1017, in runApp self.checkForCrashes(profileDir, symbolsPath) File "/builds/tegra-332/test/build/tests/mochitest/remoteautomation.py", line 79, in checkForCrashes automationutils.checkForCrashes(dumpDir, symbolsPath, self.lastTestSeen) File "/builds/tegra-332/test/build/tests/mochitest/automationutils.py", line 174, in checkForCrashes stderr=subprocess.PIPE) File "/tools/python27/lib/python2.7/subprocess.py", line 679, in __init__ errread, errwrite) File "/tools/python27/lib/python2.7/subprocess.py", line 1228, in _execute_child raise child_exception OSError: [Errno 8] Exec format error
Comment 1•12 years ago
|
||
The failure is at: { # run minidump stackwalk p = subprocess.Popen([stackwalkPath, d, symbolsPath], stdout=subprocess.PIPE, stderr=subprocess.PIPE) } Ted, could this be due to the new binaries landed by bug 794122?
Comment 2•12 years ago
|
||
I think this is the problem that Callek mentioned offhand on IRC: foopy_type: 'Linux' The foopies run a mix of OS X and Linux, but the buildbot configs have them hardcoded to use the osx minidump_stackwalk binary. This error is a Linux foopy trying to run a Mac binary.
Assignee | ||
Comment 3•12 years ago
|
||
Yea, this is likely my issue. The short story is this is an error in our ability to report a real error elsewhere, so any reports of this is us trying to dump a stack of a real crash. Moving to releng and planning to tackle this.
Assignee: nobody → bugspam.Callek
Component: General → Release Engineering: Automation (General)
Product: Testing → mozilla.org
QA Contact: catlee
Version: Trunk → other
Reporter | ||
Comment 4•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=15651492&tree=Mozilla-Beta
Reporter | ||
Comment 5•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=15683039&tree=Mozilla-Inbound
Reporter | ||
Comment 6•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=15683519&tree=Mozilla-Inbound
Reporter | ||
Comment 7•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=15690189&tree=Services-Central
Reporter | ||
Comment 8•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=15760658&tree=Mozilla-Inbound
Reporter | ||
Comment 9•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=15772956&tree=Mozilla-Inbound
Reporter | ||
Comment 10•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=15777733&tree=Mozilla-Inbound
Reporter | ||
Comment 11•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=15859927&tree=Mozilla-Aurora
Reporter | ||
Comment 13•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=15935775&tree=Mozilla-Inbound
Reporter | ||
Comment 14•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16157832&tree=Mozilla-Inbound
Reporter | ||
Comment 15•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16160993&tree=Mozilla-Inbound
Reporter | ||
Comment 16•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16161983&tree=Mozilla-Inbound
Reporter | ||
Comment 17•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16165125&tree=Mozilla-Inbound
Reporter | ||
Comment 18•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16168522&tree=Mozilla-Inbound
Reporter | ||
Comment 19•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16171754&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=16172719&tree=Mozilla-Inbound
Reporter | ||
Comment 20•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16173007&tree=Mozilla-Inbound
Reporter | ||
Comment 21•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16178879&tree=Mozilla-Inbound
Reporter | ||
Comment 22•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16183922&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=16183693&tree=Mozilla-Inbound
Reporter | ||
Comment 23•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16191156&tree=Mozilla-Inbound
Reporter | ||
Comment 24•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16201510&tree=Mozilla-Inbound
Reporter | ||
Comment 25•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16210272&tree=Mozilla-Inbound
Reporter | ||
Comment 26•12 years ago
|
||
Little odd that it seems like this is always mochitest-2 crashes, and only mochitest-2 crashes. https://tbpl.mozilla.org/php/getParsedLog.php?id=16210085&tree=Mozilla-Aurora
Reporter | ||
Comment 27•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16216086&tree=Mozilla-Beta
Reporter | ||
Comment 28•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16218493&tree=Mozilla-Inbound
Reporter | ||
Comment 29•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16232625&tree=Mozilla-Inbound
Reporter | ||
Comment 30•12 years ago
|
||
(In reply to Phil Ringnalda (:philor) from comment #26) > Little odd that it seems like this is always mochitest-2 crashes, and only > mochitest-2 crashes. Ah, because we have a fairly frequent crash in svg/content/test/test_pointer-events.xhtml on the new tegras only, and the new tegras are mostly on the new foopies, and the new foopies are Linux.
Comment 31•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16538616&tree=Mozilla-Inbound
Reporter | ||
Comment 32•12 years ago
|
||
I don't think Orange Factor actually, you know, *works*. I'd guess without having kept track that this is up to more like 20-30 failures per weekday, certainly not OF's 16 trunk, 17 total for the week (there was a push on m-a or m-b yesterday where I starred this twice on one *push*, so 1 off-trunk in the last week is certainly wrong).
Reporter | ||
Comment 33•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16608796&tree=Mozilla-Inbound
Reporter | ||
Comment 34•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16609017&tree=Mozilla-Inbound
Reporter | ||
Comment 35•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16609976&tree=Mozilla-Inbound
Reporter | ||
Comment 36•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16610769&tree=Mozilla-Inbound
Reporter | ||
Comment 37•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16612144&tree=Mozilla-Inbound
Comment 38•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16613205&tree=Mozilla-Inbound
Comment 39•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16614121&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=16614358&tree=Mozilla-Inbound
Comment 40•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16614790&tree=Mozilla-Inbound
Reporter | ||
Comment 41•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16623192&tree=Mozilla-Inbound
Reporter | ||
Comment 42•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16622533&tree=Mozilla-Inbound
Reporter | ||
Comment 43•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16622366&tree=Mozilla-Inbound (a bonus triple-crash!)
Reporter | ||
Comment 44•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16621791&tree=Mozilla-Inbound
Reporter | ||
Comment 45•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16621782&tree=Mozilla-Inbound
Reporter | ||
Comment 46•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16623095&tree=Mozilla-Inbound
Reporter | ||
Comment 47•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16626470&tree=Mozilla-Inbound
Reporter | ||
Comment 48•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16626660&tree=Mozilla-Inbound
Reporter | ||
Comment 49•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16626745&tree=Mozilla-Inbound
Reporter | ||
Comment 50•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16627492&tree=Mozilla-Inbound
Reporter | ||
Comment 51•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16628249&tree=Mozilla-Inbound
Reporter | ||
Comment 52•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16628246&tree=Mozilla-Inbound
Reporter | ||
Comment 53•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16624289&tree=Mozilla-Inbound
Reporter | ||
Comment 54•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16632712&tree=Mozilla-Inbound
Reporter | ||
Comment 55•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16633585&tree=Mozilla-Inbound
Reporter | ||
Comment 56•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16628627&tree=Mozilla-Aurora
Reporter | ||
Comment 57•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16633933&tree=Mozilla-Inbound
Comment 58•12 years ago
|
||
Callek, did you get a chance to look at this today? :-)
Assignee | ||
Comment 59•12 years ago
|
||
(In reply to Ed Morley [:edmorley UTC+0] from comment #58) > Callek, did you get a chance to look at this today? :-) I did I will post a patch when I'm done handing out candy for the neighborhood kids
Reporter | ||
Comment 60•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16634650&tree=Mozilla-Inbound
Reporter | ||
Comment 61•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16641926&tree=Mozilla-Inbound
Assignee | ||
Comment 62•12 years ago
|
||
Attachment #677258 -
Flags: review?(coop)
Assignee | ||
Comment 63•12 years ago
|
||
The equiv of this patch needs to be manually applied to mac foopies, doable with: mkdir /Users/cltbld/bin; ln -s /builds/tools/breakpad/osx/minidump_stackwalk /Users/cltbld/bin/minidump_stackwalk
Attachment #677260 -
Flags: review?(coop)
Reporter | ||
Comment 64•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16642122&tree=Mozilla-Inbound
Reporter | ||
Comment 65•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16644756&tree=Mozilla-Inbound
Reporter | ||
Comment 66•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16646816&tree=Mozilla-Inbound
Comment 67•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16656048&tree=Mozilla-Inbound
Reporter | ||
Comment 68•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16659989&tree=Mozilla-Inbound
Reporter | ||
Comment 69•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16661853&tree=Mozilla-Inbound
Reporter | ||
Comment 70•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16663006&tree=Mozilla-Inbound
Reporter | ||
Comment 71•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16663659&tree=Mozilla-Inbound
Comment 72•12 years ago
|
||
Comment on attachment 677258 [details] [diff] [review] [buildbotcustom] Use symlinked minidump prog Review of attachment 677258 [details] [diff] [review]: ----------------------------------------------------------------- r+ assuming the ~ gets expanded properly.
Attachment #677258 -
Flags: review?(coop) → review+
Updated•12 years ago
|
Attachment #677260 -
Flags: review?(coop) → review+
Reporter | ||
Comment 73•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16666845&tree=Mozilla-Inbound
Reporter | ||
Comment 74•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16663665&tree=Profiling
Reporter | ||
Comment 75•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16667149&tree=Services-Central
Assignee | ||
Comment 76•12 years ago
|
||
Comment on attachment 677260 [details] [diff] [review] [puppet] Create the symlink to global-foopy-shared tools repo Ran locally: for i in 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 22 23 24; do ssh cltbld@foopy$i.build.mozilla.org "bash -l -c 'mkdir /Users/cltbld/bin; ln -s /builds/tools/breakpad/osx/minidump_stackwalk /Users/cltbld/bin/minidump_stackwalk'"; done So all the mac's are done, also checked in this and saw some green puppet runs: http://hg.mozilla.org/build/puppet/rev/87b60f4b626b
Attachment #677260 -
Flags: checked-in+
Reporter | ||
Comment 77•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16651920&tree=Fx-Team
Assignee | ||
Comment 78•12 years ago
|
||
Comment on attachment 677258 [details] [diff] [review] [buildbotcustom] Use symlinked minidump prog Review of attachment 677258 [details] [diff] [review]: ----------------------------------------------------------------- http://hg.mozilla.org/build/buildbotcustom/rev/57cf5697bc47
Attachment #677258 -
Flags: checked-in+
Reporter | ||
Comment 79•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16668969&tree=Mozilla-Inbound
Reporter | ||
Comment 80•12 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=16669199&tree=Mozilla-Inbound
Assignee | ||
Comment 81•12 years ago
|
||
So it turns out we don't actually do the shell expansion where I thought we did here, so ~ wasn't expanding. Therefore we need a loc that is the same on both mac foopies and linux foopies.
Attachment #677598 -
Flags: review?(aki)
Updated•12 years ago
|
Attachment #677598 -
Flags: review?(aki) → review+
Assignee | ||
Comment 82•12 years ago
|
||
(no need for tilde expansion)
Attachment #677600 -
Flags: review?(aki)
Updated•12 years ago
|
Attachment #677600 -
Flags: review?(aki) → review+
Assignee | ||
Comment 83•12 years ago
|
||
Comment on attachment 677598 [details] [diff] [review] [puppet] move the minidump symlink Review of attachment 677598 [details] [diff] [review]: ----------------------------------------------------------------- $ for i in 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 22 23 24; do ssh cltbld@foopy$i.build.mozilla.org "ba sh -l -c 'rm -rf /Users/cltbld/bin; ln -s /builds/tools/breakpad/osx/minidump_stackwalk /builds/minidump_stackwa lk'"; done And this was checked in: http://hg.mozilla.org/build/puppet/rev/c504e817eec6
Attachment #677598 -
Flags: checked-in+
Assignee | ||
Comment 84•12 years ago
|
||
Confirmed this is now fixed. We shouldn't see this anymore.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Comment 85•12 years ago
|
||
Thank you for sorting this :-)
Updated•12 years ago
|
Keywords: intermittent-failure
Updated•12 years ago
|
Whiteboard: [orange]
Updated•11 years ago
|
Product: mozilla.org → Release Engineering
Updated•6 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•