910994 - need stack traces when aborting on timeouts in b2g emulator tests

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Comment 1

[Chris AtLee [:catlee]]

Any ideas how to turn this on? Is it a matter of having symbols available and pointing the test harnesses at them?

Comment 2

[Karl Tomlinson (:karlt)]

I don't know exactly what is needed here, possibly more than one thing.

Logs in bug 820739 show the application killed with signal 9, and we are not going to get a stack with that symbol.

https://tbpl.mozilla.org/php/getParsedLog.php?id=27398230&tree=Mozilla-Inbound#error2
b2g_emulator_vm mozilla-inbound opt test crashtest-1 on 2013-09-04 15:41:47 PDT for push 4a5e5f94a4f7
is more promising:

16:11:44     INFO -  mozcrash INFO | Downloading symbols from: http://pvtbuilds.pvt.build.mozilla.org//pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-generic/20130904134814/emulator.crashreporter-symbols.zip
16:11:45     INFO -  Traceback (most recent call last):
16:11:45     INFO -    File "/builds/slave/test/build/tests/reftest/b2gautomation.py", line 115, in checkForCrashes
16:11:45     INFO -      crashed = mozcrash.check_for_crashes(local_dump_dir, symbolsPath, test_name=self.lastTestSeen)
16:11:45     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 152, in check_for_crashes
16:11:45     INFO -      shutil.rmtree(symbols_path)
16:11:45     INFO -    File "/usr/lib/python2.7/shutil.py", line 237, in rmtree
16:11:45     INFO -      onerror(os.listdir, path, sys.exc_info())
16:11:45     INFO -    File "/usr/lib/python2.7/shutil.py", line 235, in rmtree
16:11:45     INFO -      names = os.listdir(path)
16:11:45     INFO -  OSError: [Errno 2] No such file or directory: 'http://pvtbuilds.pvt.build.mozilla.org//pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-generic/20130904134814/emulator.crashreporter-symbols.zip'
16:11:45     INFO -  checking for crashes in '/data/local/tests/profile/minidumps'
16:11:45     INFO -  WARNING | leakcheck | refcount logging is off, so leaks can't be detected!
16:11:45     INFO -  REFTEST INFO | runreftest.py | Running tests: end.
16:11:48    ERROR - Return code: 256

Looks like symbols might help, but I don't know whether or not there was a minidump.

Comment 3

[Chris AtLee [:catlee]]

Interesting. The crash symbols are named "b2g-26.0a1.en-US.android-arm.crashreporter-symbols.zip" there and not "emulator.crashreporter-symbols.zip"

It looks like mozharness is generating this URL from something (b2g_emulator_unittest.py)

Comment 4

[Chris AtLee [:catlee]]

The problem is that the emulator package is called emulator.tar.gz, and b2g_emulator_unittest.py via mozharness.mozilla.testing.testbase calculates the symbols url to be emulator.crashreporter-symbols.zip based on this.

We can rename the crash symbols package to match the expectation, or perhaps include the crash symbols url in the sendchange and modify testbase to pick up the symbols url from the buildbot properties.

Comment 5

[Aki Sasaki (not active)]

Attachment: debug-emu-symbols -- sendchange symbols

Untested as of yet.

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 803242 [details] [diff] [review]
debug-emu-symbols -- sendchange symbols

https://tbpl.mozilla.org/?tree=Ash&rev=82db508f2304
https://tbpl.mozilla.org/php/getParsedLog.php?id=27797676&tree=Ash&full=1

16:28:19     INFO - Found symbols url http://pvtbuilds.pvt.build.mozilla.org//pub/mozilla.org/b2g/tinderbox-builds/ash-generic/20130912113857/b2g-26.0a1.en-US.android-arm.crashreporter-symbols.zip.

Comment 7

[Aki Sasaki (not active)]

Comment on attachment 803242 [details] [diff] [review]
debug-emu-symbols -- sendchange symbols

https://hg.mozilla.org/build/mozharness/rev/a660ae1a633f

Comment 9

[Aki Sasaki (not active)]

https://tbpl.mozilla.org/?tree=Cedar&showall=1&rev=e6f8b77a8824&jobname=emulator

Instead of perma-red debug tests, we're more autumn.
But we are finding the symbols now...

Comment 10

[Karl Tomlinson (:karlt)]

Thanks.  That's helpful.  Crashtests are now generating stacks on timeouts:
https://tbpl.mozilla.org/php/getParsedLog.php?id=27849305&tree=Cedar#error4

Filed bug 919353 re signal 9 in mochitests.