Closed Bug 1556286 Opened 6 years ago Closed 6 years ago

Intermittent DistutilsFileError: cannot copy tree '/tmp/tmpL11gnB/minidumps': not a directory after "cannot copy tree '%s': not a directory" % src

Categories

(Firefox for Android Graveyard :: Testing, defect, P5)

defect

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Keywords: intermittent-failure, regression)

Filed by: rgurzau [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=249637149&repo=mozilla-central
Full log: https://queue.taskcluster.net/v1/task/WKR1EVt4QJeNxOvlWTwD9g/runs/0/artifacts/public/logs/live_backing.log


[task 2019-06-02T23:17:30.131Z] 23:17:30 INFO - TEST-START | /referrer-policy/no-referrer-when-downgrade/http-rp/same-origin/http-https/xhr-request/swap-origin-redirect/upgrade-protocol.http.html
[task 2019-06-02T23:17:30.131Z] 23:17:30 INFO - Closing window 36
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - .
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - Traceback (most recent call last):
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 371, in run
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - new_state = self.wait_event()
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 446, in wait_event
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - return f(*data)
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 588, in test_ended
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - if self.browser.check_crash(test.id) and status != "CRASH":
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 245, in check_crash
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - return self.browser.check_crash(process=self.browser_pid, test=test_id)
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/browsers/fennec.py", line 229, in check_crash
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - return self.runner.check_for_crashes()
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozrunner/base/device.py", line 160, in check_for_crashes
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - dump_dir = self.device.pull_minidumps()
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozrunner/devices/base.py", line 63, in pull_minidumps
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - self.device.pull(remote_dump_dir, local_dump_dir)
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 2247, in pull
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - dir_util.copy_tree(local, original_local)
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - File "/usr/lib/python2.7/distutils/dir_util.py", line 128, in copy_tree
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - "cannot copy tree '%s': not a directory" % src
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR - DistutilsFileError: cannot copy tree '/tmp/tmpL11gnB/minidumps': not a directory
[task 2019-06-02T23:17:30.720Z] 23:17:30 ERROR -
[task 2019-06-02T23:17:31.493Z] 23:17:31 INFO - Closing logging queue
[task 2019-06-02T23:17:31.493Z] 23:17:31 INFO - queue closed

Basically it appears Android killed fennec. snorp: Just memory pressure?

06-03 00:17:15.860 11721 11721 D AndroidRuntime: >>>>>> START com.android.internal.os.RuntimeInit uid 0 <<<<<<
06-03 00:17:15.860 11721 11721 D AndroidRuntime: CheckJNI is ON
06-03 00:17:15.870 11721 11721 W art     : Unexpected CPU variant for X86 using defaults: x86_64
06-03 00:17:15.880 11721 11721 D ICU     : No timezone override file found: /data/misc/zoneinfo/current/icu/icu_tzdata.dat
06-03 00:17:15.880 11721 11721 E memtrack: Couldn't load memtrack module (No such file or directory)
06-03 00:17:15.880 11721 11721 E android.os.Debug: failed to load memtrack module: -2
06-03 00:17:15.880 11721 11721 I Radio-JNI: register_android_hardware_Radio DONE
06-03 00:17:15.890 11721 11721 D AndroidRuntime: Calling main entry com.android.commands.am.Am
06-03 00:17:15.890  1308  1402 I ActivityManager: Force stopping org.mozilla.fennec_aurora appid=10062 user=0: from pid 11721
Flags: needinfo?(snorp)

(In reply to Bob Clary [:bc:] from comment #2)

Basically it appears Android killed fennec. snorp: Just memory pressure?

I think that might just be the harness intentionally killing it, but not entirely sure. It looks like the exception from comment #0 is just some missing error checking from detecting crashes. We're trying to adb pull the minidump directory and that's failing (maybe because there's no crashes?). Can we find the stderr from those commands somewhere?

Flags: needinfo?(snorp) → needinfo?(bob)

The best we can do is the logcat linked from the log viewer: https://taskcluster-artifacts.net/WKR1EVt4QJeNxOvlWTwD9g/0/public/test_info//logcat-emulator-5554.log which is were the data in comment 2 came from. The lack of the minidump directory could just be our failure to create it. It does appear fennec was successfully installed and ran prior to it being killed by the ActivityManager. Would fennec being killed cause the minidump directory to not be created or to be deleted?

Flags: needinfo?(bob)
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → INCOMPLETE

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=277086602&repo=autoland&lineNumber=2854

[task 2019-11-20T02:36:14.207Z] 02:36:14 INFO - TEST-START | /html/semantics/embedded-content/media-elements/track/track-element/track-cue-negative-duration.html
[task 2019-11-20T02:36:14.209Z] 02:36:14 INFO - Closing window 66
[task 2019-11-20T02:36:15.989Z] 02:36:15 ERROR - .
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - Traceback (most recent call last):
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 386, in run
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - new_state = self.wait_event()
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 461, in wait_event
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - return f(*data)
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 606, in test_ended
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - if self.browser.check_crash(test.id) and status != "CRASH":
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 260, in check_crash
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - return self.browser.check_crash(process=self.browser_pid, test=test_id)
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox_android.py", line 202, in check_crash
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - return bool(self.runner.check_for_crashes(test_name=test))
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozrunner/base/device.py", line 161, in check_for_crashes
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - dump_dir = self.device.pull_minidumps()
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozrunner/devices/base.py", line 67, in pull_minidumps
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - self.device.pull(remote_dump_dir, local_dump_dir)
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 2421, in pull
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - dir_util.copy_tree(local, original_local)
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - File "/usr/lib/python2.7/distutils/dir_util.py", line 128, in copy_tree
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - "cannot copy tree '%s': not a directory" % src
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR - DistutilsFileError: cannot copy tree '/tmp/tmpSL3ZhD/minidumps': not a directory
[task 2019-11-20T02:36:15.990Z] 02:36:15 ERROR -
[task 2019-11-20T02:36:16.738Z] 02:36:16 INFO - Closing logging queue
[task 2019-11-20T02:36:16.738Z] 02:36:16 INFO - queue closed
[task 2019-11-20T02:36:17.177Z] 02:36:17 INFO - STDERR: Exception in thread TestRunnerManager-1:
[task 2019-11-20T02:36:17.177Z] 02:36:17 INFO - Traceback (most recent call last):
[task 2019-11-20T02:36:17.177Z] 02:36:17 INFO - File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
[task 2019-11-20T02:36:17.177Z] 02:36:17 INFO - self.run()
[task 2019-11-20T02:36:17.177Z] 02:36:17 INFO - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 386, in run
[task 2019-11-20T02:36:17.177Z] 02:36:17 INFO - new_state = self.wait_event()
[task 2019-11-20T02:36:17.177Z] 02:36:17 INFO - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 461, in wait_event
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - return f(*data)
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 606, in test_ended
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - if self.browser.check_crash(test.id) and status != "CRASH":
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 260, in check_crash
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - return self.browser.check_crash(process=self.browser_pid, test=test_id)
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox_android.py", line 202, in check_crash
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - return bool(self.runner.check_for_crashes(test_name=test))
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozrunner/base/device.py", line 161, in check_for_crashes
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - dump_dir = self.device.pull_minidumps()
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozrunner/devices/base.py", line 67, in pull_minidumps
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - self.device.pull(remote_dump_dir, local_dump_dir)
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 2421, in pull
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - dir_util.copy_tree(local, original_local)
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - File "/usr/lib/python2.7/distutils/dir_util.py", line 128, in copy_tree
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - "cannot copy tree '%s': not a directory" % src
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - DistutilsFileError: cannot copy tree '/tmp/tmpSL3ZhD/minidumps': not a directory
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - Got 0 unexpected results
[task 2019-11-20T02:36:17.178Z] 02:36:17 INFO - SUITE-END | took 582s
[task 2019-11-20T02:36:17.274Z] 02:36:17 INFO - Closing logging queue
[task 2019-11-20T02:36:17.274Z] 02:36:17 INFO - queue closed
[task 2019-11-20T02:36:17.274Z] 02:36:17 CRITICAL - Test harness output was not a valid structured log message:
[task 2019-11-20T02:36:17.274Z] 02:36:17 CRITICAL - Traceback (most recent call last):
[task 2019-11-20T02:36:17.275Z] 02:36:17 CRITICAL - File "/usr/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
[task 2019-11-20T02:36:17.275Z] 02:36:17 CRITICAL - send(obj)
[task 2019-11-20T02:36:17.275Z] 02:36:17 CRITICAL - IOError: [Errno 32] Broken pipe
[task 2019-11-20T02:36:17.513Z] 02:36:17 INFO - Return code: 0
[task 2019-11-20T02:36:17.513Z] 02:36:17 CRITICAL - # TBPL FAILURE #

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 6 years ago6 years ago
Resolution: --- → INCOMPLETE
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.