Closed Bug 938706 Opened 7 years ago Closed 7 years ago

Android xpcshell might not report minidump stack for crashes

Categories

(Testing :: XPCShell Harness, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla28

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

Attachments

(2 files, 1 obsolete file)

I think we might be passing the wrong directory to check_for_crashes for Android xpcshell tests, explaining some observations in bug 937316.
I verified that minidump stacks are not generated for xpcshell crashes on Android and the primary reason is that check_for_crashes is supplied with a remote directory.

https://tbpl.mozilla.org/php/getParsedLog.php?id=30584419&tree=Try&full=1#error0

I have a solution under test now.
Close, but try run shows that if 2 or more crashes occur in one job, we run into trouble...looks like I need to clean up the tmp directory: https://tbpl.mozilla.org/?tree=Try&rev=93f207bb77d7
Here's a good try run (intentional crash, produces minidump stack): https://tbpl.mozilla.org/?tree=Try&rev=4b3b3cc88735
On device, .dmp files are written to the tmp directory. This patch pulls them back to a temp directory on the test slave, applies mozcrash.check_for_crashes to the local temp directory, and deletes the remote tmp directory.
Attachment #833204 - Flags: review?(ted)
Comment on attachment 833204 [details] [diff] [review]
fix minidumps for Android xpcshell

Review of attachment 833204 [details] [diff] [review]:
-----------------------------------------------------------------

jmaher and I once talked about adding a check_for_remote_crashes somewhere which would encapsulate this.

::: testing/xpcshell/remotexpcshelltests.py
@@ +137,5 @@
> +                        dump_directory,
> +                        symbols_path,
> +                        test_name=None):
> +        try:
> +            dumpDir = tempfile.mkdtemp()

Can you use mozfile.TemporaryDirectory here?

::: testing/xpcshell/runxpcshelltests.py
@@ +224,5 @@
>                      env=env, cwd=cwd)
>          return proc
>  
> +    def checkForCrashes(self, 
> +                        dump_directory, 

nit: trailing whitespace
Attachment #833204 - Flags: review?(ted) → review+
Updated for review comments. That's cleaner -- thanks!
Attachment #833204 - Attachment is obsolete: true
Attachment #833219 - Flags: review+
Unfortunately, this doesn't quite work:

https://tbpl.mozilla.org/?tree=Try&rev=722f9a804f46

The xpcshell .dmp/.extra files are written to the tmp directory. Some tests - test_safeoutputstream.js - write to the same location. It seems that filenames with certain characters cause a failure in devicemanager.pullFile (called from getDirectory):

18:14:03     INFO -  TEST-INFO | /builds/panda-0526/test/build/tests/xpcshell/tests/netwerk/test/unit/test_safeoutputstream.js | running test ...
18:14:03     INFO -  TEST-PASS | /builds/panda-0526/test/build/tests/xpcshell/tests/netwerk/test/unit/test_safeoutputstream.js | test passed (time: 635.871ms)
18:14:03     INFO -  INFO | Following exceptions were raised:
18:14:03     INFO -  Traceback (most recent call last):
18:14:03     INFO -    File "/builds/panda-0526/test/build/tests/xpcshell/runxpcshelltests.py", line 145, in run
18:14:03     INFO -      self.run_test()
18:14:03     INFO -    File "/builds/panda-0526/test/build/tests/xpcshell/runxpcshelltests.py", line 671, in run_test
18:14:03     INFO -      if self.checkForCrashes(self.tempDir, self.symbolsPath, test_name=name):
18:14:03     INFO -    File "/builds/panda-0526/test/build/tests/xpcshell/remotexpcshelltests.py", line 142, in checkForCrashes
18:14:03     INFO -      self.device.getDirectory(self.remoteTmpDir, dumpDir)
18:14:03     INFO -    File "/builds/panda-0526/test/build/venv/lib/python2.7/site-packages/mozdevice/devicemanagerSUT.py", line 673, in getDirectory
18:14:03     INFO -      self.getFile(remotePath, localPath)
18:14:03     INFO -    File "/builds/panda-0526/test/build/venv/lib/python2.7/site-packages/mozdevice/devicemanagerSUT.py", line 645, in getFile
18:14:03     INFO -      data = self.pullFile(remoteFile)
18:14:03     INFO -    File "/builds/panda-0526/test/build/venv/lib/python2.7/site-packages/mozdevice/devicemanagerSUT.py", line 634, in pullFile
18:14:03     INFO -      raise DMError("DeviceManager: pulling file '%s' unsuccessful: %s" % (remoteFile, error_str))
18:14:03     INFO -  DMError: DeviceManager: pulling file '/mnt/sdcard/tests/xpcshell/tmp/ओ' unsuccessful: No such file or directory
18:14:03     INFO -  Traceback (most recent call last):
18:14:03     INFO -    File "/builds/panda-0526/test/build/tests/xpcshell/remotexpcshelltests.py", line 577, in <module>
18:14:03     INFO -      main()
18:14:03     INFO -    File "/builds/panda-0526/test/build/tests/xpcshell/remotexpcshelltests.py", line 572, in main
18:14:03     INFO -      **options.__dict__):
18:14:03     INFO -    File "/builds/panda-0526/test/build/tests/xpcshell/runxpcshelltests.py", line 1462, in runTests
18:14:03     INFO -      raise exceptions[0]
18:14:03     INFO -  mozdevice.devicemanager.DMError: DeviceManager: pulling file '/mnt/sdcard/tests/xpcshell/tmp/ओ' unsuccessful: No such file or directory
18:14:03    ERROR - Return code: 1

I'm not sure what to do here:
 -- fix pullFile to work with all character sets
 -- retrieve only .dmp/.extra files
 -- place .dmp/.extra files in their own directory.
Putting the .dmp files in their own directory is probably fine, we already pass that down as an env var. That's likely to be the simplest fix too. (You should at least file the pullFile bug though, since that sounds like something that'd be useful to get fixed.)
See Also: → 940504
This updates head.js to use a different minidump directory if an environment variable is set, and updates remotexpcshelltests.py to set that environment variable to a new directory, effectively avoiding the problem I reported in Comment 7.
Attachment #8336084 - Flags: review?(ted)
Attachment #833219 - Attachment description: fix minidumps for Android xpcshell → (1) fix minidumps for Android xpcshell
Attachment #8336084 - Flags: review?(ted) → review+
https://hg.mozilla.org/mozilla-central/rev/5b1d906be501
https://hg.mozilla.org/mozilla-central/rev/4d63aa417ac3
Status: NEW → RESOLVED
Closed: 7 years ago
Flags: in-testsuite-
Resolution: --- → FIXED
Target Milestone: --- → mozilla28
You need to log in before you can comment on or make changes to this bug.