Closed Bug 1750145 Opened 3 years ago Closed 8 months ago

Intermittent Error copying results /builds/task_"id#"/workspace/build/raptor.json to upload dir /builds/task_"id#"/workspace/build/blobber_upload_dir/perfherder-data.json | [Errno 2] No such file or directory: '/builds/task_"id#"/<...>/raptor.json'

Categories

(Testing :: Raptor, defect, P5)

defect

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: intermittent-bug-filer, Unassigned)

Details

(Keywords: intermittent-failure)

Filed by: imoraru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=364070866&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/WmrIjbX3RZOx1hXQMLI3fw/runs/0/artifacts/public/logs/live_backing.log


[task 2022-01-14T03:55:52.538Z] 03:55:48     INFO -  raptor-perftest Info: starting test: imdb
[task 2022-01-14T03:55:52.538Z] 03:55:48     INFO -  raptor-perftest Info: test uses playback tool: mitmproxy-android
[task 2022-01-14T03:55:52.538Z] 03:55:48     INFO -  raptor-mitmproxy Info: mozproxy_dir used for mitmproxy downloads and exe files: /builds/task_164213225211120/workspace/testing/mozproxy
[task 2022-01-14T03:55:52.538Z] 03:55:48     INFO -  raptor-mitmproxy Info: Playback tool: mitmproxy-android
[task 2022-01-14T03:55:52.538Z] 03:55:48     INFO -  raptor-mitmproxy Info: Playback tool version: 6.0.2
[task 2022-01-14T03:55:52.538Z] 03:55:48     INFO -  raptor-mitmproxy Info: create mitmproxy 6.0.2 dir
[task 2022-01-14T03:55:52.538Z] 03:55:48     INFO -  raptor-mitmproxy Info: downloading mitmproxy binary
[task 2022-01-14T03:55:52.538Z] 03:55:48     INFO -  raptor-mitmproxy Info: b'INFO - File mitmproxy-6.0.2-linux.tar.gz not present in local cache folder /builds/tooltool_cache'
[task 2022-01-14T03:55:52.538Z] 03:55:48     INFO -  raptor-mitmproxy Info: b"INFO - Attempting to fetch from 'http://localhost:8099/tooltool.mozilla-releng.net/'..."
[task 2022-01-14T03:55:52.538Z] 03:55:50     INFO -  raptor-mitmproxy Info: b'INFO - File mitmproxy-6.0.2-linux.tar.gz fetched from http://localhost:8099/tooltool.mozilla-releng.net/ as /builds/task_164213225211120/workspace/testing/mozproxy/mitmdump-6.0.2/tmpx8n3ibtj'
[task 2022-01-14T03:55:52.538Z] 03:55:51     INFO -  raptor-mitmproxy Info: b'INFO - File integrity verified, renaming tmpx8n3ibtj to mitmproxy-6.0.2-linux.tar.gz'
[task 2022-01-14T03:55:52.538Z] 03:55:51     INFO -  raptor-mitmproxy Info: b'INFO - Updating local cache /builds/tooltool_cache...'
[task 2022-01-14T03:55:52.538Z] 03:55:51     INFO -  raptor-mitmproxy Info: b'INFO - Creating cache in /builds/tooltool_cache...'
[task 2022-01-14T03:55:52.538Z] 03:55:51     INFO -  raptor-mitmproxy Info: b'INFO - Local cache /builds/tooltool_cache updated with mitmproxy-6.0.2-linux.tar.gz'
[task 2022-01-14T03:55:52.538Z] 03:55:51     INFO -  raptor-mitmproxy Info: b'INFO - untarring "mitmproxy-6.0.2-linux.tar.gz"'
[task 2022-01-14T03:55:52.538Z] 03:55:51     INFO -  raptor-mitmproxy Info: downloading mitmproxy pageset
[task 2022-01-14T03:55:52.538Z] 03:55:51     INFO -  raptor-mitmproxy Info: b'INFO - File mitm6-android-fenix-imdb.zip not present in local cache folder /builds/tooltool_cache'
[task 2022-01-14T03:55:52.538Z] 03:55:51     INFO -  raptor-mitmproxy Info: b"INFO - Attempting to fetch from 'http://localhost:8099/tooltool.mozilla-releng.net/'..."
[task 2022-01-14T03:55:52.538Z] 03:55:52     INFO -  raptor-mitmproxy Info: b'INFO - File mitm6-android-fenix-imdb.zip fetched from http://localhost:8099/tooltool.mozilla-releng.net/ as /builds/task_164213225211120/workspace/testing/mozproxy/tmppib6d76y'
[task 2022-01-14T03:55:52.538Z] 03:55:52     INFO -  raptor-mitmproxy Info: b'INFO - File integrity verified, renaming tmppib6d76y to mitm6-android-fenix-imdb.zip'
[task 2022-01-14T03:55:52.538Z] 03:55:52     INFO -  raptor-mitmproxy Info: b'INFO - Updating local cache /builds/tooltool_cache...'
[task 2022-01-14T03:55:52.538Z] 03:55:52     INFO -  raptor-mitmproxy Info: b'INFO - Local cache /builds/tooltool_cache updated with mitm6-android-fenix-imdb.zip'
[task 2022-01-14T03:55:52.538Z] 03:55:52     INFO -  raptor-mitmproxy Info: Adding /builds/task_164213225211120/workspace/testing/mozproxy/mitm6-android-fenix-imdb.zip to recording list
[task 2022-01-14T03:55:52.538Z] 03:55:52     INFO -  raptor-mitmproxy Info: Loaded recoording generated on 2021-04-19 10:32:42.317642
[task 2022-01-14T03:55:52.538Z] 03:55:52     INFO -  raptor-mitmproxy Info: mitmdump path: /builds/task_164213225211120/workspace/testing/mozproxy/mitmdump-6.0.2/mitmdump
[task 2022-01-14T03:55:52.538Z] 03:55:52     INFO -  raptor-mitmproxy Info: browser path: org.mozilla.geckoview_example
[task 2022-01-14T03:55:52.538Z] 03:55:52     INFO -  raptor-mitmproxy Info: Starting mitmproxy playback using env path: /builds/task_164213225211120/workspace/build/venv/bin:/builds/task_164213225211120/workspace/build/venv/bin:/builds/task_164213225211120/workspace/build/venv/bin:/builds/worker/.pyenv/shims:/builds/worker/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/builds/worker/bin:/usr/local/bin:/builds/worker/android-sdk-linux/platform-tools:/builds/worker/android-sdk-linux/platform-tools/bin:/builds/worker/android-sdk-linux/tools:/builds/worker/android-sdk-linux/tools/bin:/builds/worker/android-sdk-linux/build-tools/27.0.3:/builds/worker/android-sdk-linux/tools/bin:/builds/worker/android-sdk-linux/platform-tools
[task 2022-01-14T04:00:13.097Z] 03:55:52     INFO -  raptor-mitmproxy Info: Starting mitmproxy playback using command: /builds/task_164213225211120/workspace/testing/mozproxy/mitmdump-6.0.2/mitmdump --listen-host 127.0.0.1 --listen-port 53989 --set upstream_cert=false --set upload_dir=/builds/task_164213225211120/workspace/build/blobber_upload_dir --set websocket=false --set server_replay_files=/builds/task_164213225211120/workspace/testing/mozproxy/mitm6-android-fenix-imdb/dump.mp --scripts /builds/task_164213225211120/workspace/build/venv/lib/python3.6/site-packages/mozproxy/backends/mitm/scripts/alternate-server-replay.py
[task 2022-01-14T04:00:13.143Z] 03:55:53     INFO -  raptor-mitmproxy Info: Mitmproxy playback successfully started on 127.0.0.1:53989 as pid 1495
[task 2022-01-14T04:00:13.151Z] 03:55:53     INFO -  raptor-mitmproxy Info: downloading certutil binary (hostutils)
[task 2022-01-14T04:00:13.178Z] 03:55:53     INFO -  raptor-mitmproxy Info: downloading: https://hg.mozilla.org/integration/autoland/raw-file/344f14b85f84dabbc7709b5fc5d23e9471060d75/testing/config/tooltool-manifests/linux64/hostutils.manifest to /builds/task_164213225211120/workspace/testing/mozproxy/hostutils.manifest
[task 2022-01-14T04:00:13.194Z] 03:55:54     INFO -  raptor-mitmproxy Info: b'INFO - File host-utils-93.0a1.en-US.linux-x86_64.tar.gz not present in local cache folder /builds/tooltool_cache'
[task 2022-01-14T04:00:13.194Z] 03:55:54     INFO -  raptor-mitmproxy Info: b"INFO - Attempting to fetch from 'http://localhost:8099/tooltool.mozilla-releng.net/'..."
[task 2022-01-14T04:00:13.202Z] 03:56:00     INFO -  raptor-mitmproxy Info: b'INFO - File host-utils-93.0a1.en-US.linux-x86_64.tar.gz fetched from http://localhost:8099/tooltool.mozilla-releng.net/ as /builds/task_164213225211120/workspace/testing/mozproxy/tmpfawcz38v'
[task 2022-01-14T04:00:13.202Z] 03:56:02     INFO -  raptor-mitmproxy Info: b'INFO - File integrity verified, renaming tmpfawcz38v to host-utils-93.0a1.en-US.linux-x86_64.tar.gz'
[task 2022-01-14T04:00:13.202Z] 03:56:02     INFO -  raptor-mitmproxy Info: b'INFO - Updating local cache /builds/tooltool_cache...'
[task 2022-01-14T04:00:13.202Z] 03:56:02     INFO -  raptor-mitmproxy Info: b'INFO - Local cache /builds/tooltool_cache updated with host-utils-93.0a1.en-US.linux-x86_64.tar.gz'
[task 2022-01-14T04:00:13.202Z] 03:56:02     INFO -  raptor-mitmproxy Info: b'INFO - untarring "host-utils-93.0a1.en-US.linux-x86_64.tar.gz"'
[task 2022-01-14T04:00:13.202Z] 03:56:04     INFO -  raptor-mitmproxy Info: certutil is found at: /builds/task_164213225211120/workspace/testing/mozproxy/host-utils-93.0a1.en-US.linux-x86_64/certutil
[task 2022-01-14T04:00:13.202Z] 03:56:04     INFO -  raptor-mitmproxy Info: checking if the nss cert db already exists in the android browser profile
[task 2022-01-14T04:00:13.203Z] 03:56:04     INFO -  raptor-mitmproxy Info: Certutil: Running command: /builds/task_164213225211120/workspace/testing/mozproxy/host-utils-93.0a1.en-US.linux-x86_64/certutil -d sql:/tmp/tmpu5e9ty7e/profile/ -L
[task 2022-01-14T04:00:13.208Z] 03:56:04     INFO -  raptor-mitmproxy Info: nss cert db doesn't exist yet.
[task 2022-01-14T04:00:13.208Z] 03:56:04     INFO -  raptor-mitmproxy Info: creating nss cert database
[task 2022-01-14T04:00:13.208Z] 03:56:04     INFO -  raptor-mitmproxy Info: Certutil: Running command: /builds/task_164213225211120/workspace/testing/mozproxy/host-utils-93.0a1.en-US.linux-x86_64/certutil -d sql:/tmp/tmpu5e9ty7e/profile/ -N --empty-password
[task 2022-01-14T04:00:13.209Z] 03:59:27    ERROR - Return code: -9
[task 2022-01-14T04:00:13.209Z] 03:59:27  WARNING - setting return code to -9
[task 2022-01-14T04:00:13.209Z] 03:59:27     INFO - Killing logcat pid 934.
[task 2022-01-14T04:00:13.209Z] 03:59:27     INFO - Copying Raptor results to upload dir:
[task 2022-01-14T04:00:13.209Z] 03:59:27     INFO - /builds/task_164213225211120/workspace/build/blobber_upload_dir/perfherder-data.json
[task 2022-01-14T04:00:13.209Z] 03:59:27     INFO - Copying raptor results from /builds/task_164213225211120/workspace/build/raptor.json to /builds/task_164213225211120/workspace/build/blobber_upload_dir/perfherder-data.json
[task 2022-01-14T04:00:13.209Z] 03:59:28 CRITICAL - Error copying results /builds/task_164213225211120/workspace/build/raptor.json to upload dir /builds/task_164213225211120/workspace/build/blobber_upload_dir/perfherder-data.json
[task 2022-01-14T04:00:13.209Z] 03:59:28     INFO - [Errno 2] No such file or directory: '/builds/task_164213225211120/workspace/build/raptor.json'
[task 2022-01-14T04:00:13.209Z] 03:59:28     INFO - Running post-action listener: _package_coverage_data
[task 2022-01-14T04:00:13.209Z] 03:59:28     INFO - Running post-action listener: _resource_record_post_action
[task 2022-01-14T04:00:13.209Z] 03:59:28     INFO - Running post-action listener: process_java_coverage_data
[task 2022-01-14T04:00:13.209Z] 03:59:28     INFO - Running post-action listener: stop_device
[task 2022-01-14T04:00:13.209Z] 03:59:43     INFO - /data/anr/traces.txt deleted
[task 2022-01-14T04:00:13.209Z] 03:59:48     INFO - /data/anr/traces_SystemServer_WDT.txt deleted
[task 2022-01-14T04:00:13.209Z] 04:00:01     INFO - Killing logcat pid 934.
[task 2022-01-14T04:00:13.209Z] 04:00:01     INFO - [mozharness: 2022-01-14 04:00:01.054166Z] Finished run-tests step (success)
[task 2022-01-14T04:00:13.209Z] 04:00:01     INFO - Running post-run listener: _resource_record_post_run
[task 2022-01-14T04:00:13.209Z] 04:00:05     INFO - Total resource usage - Wall time: 449s; CPU: 22%; Read bytes: 27755880448; Write bytes: 4098371584; Read time: 466848; Write time: 9856348
[task 2022-01-14T04:00:13.209Z] 04:00:05     INFO - TinderboxPrint: CPU usage<br/>21.8%
[task 2022-01-14T04:00:13.209Z] 04:00:05     INFO - TinderboxPrint: I/O read bytes / time<br/>27,755,880,448 / 466,848
[task 2022-01-14T04:00:13.209Z] 04:00:05     INFO - TinderboxPrint: I/O write bytes / time<br/>4,098,371,584 / 9,856,348
[task 2022-01-14T04:00:13.209Z] 04:00:05     INFO - TinderboxPrint: CPU idle<br/>1,116.6 (62.7%)
[task 2022-01-14T04:00:13.209Z] 04:00:05     INFO - TinderboxPrint: CPU iowait<br/>275.0 (15.4%)
[task 2022-01-14T04:00:13.209Z] 04:00:05     INFO - TinderboxPrint: CPU softirq<br/>18.5 (1.0%)
[task 2022-01-14T04:00:13.209Z] 04:00:05     INFO - TinderboxPrint: CPU system<br/>196.6 (11.0%)
[task 2022-01-14T04:00:13.209Z] 04:00:05     INFO - TinderboxPrint: CPU user<br/>173.8 (9.8%)
[task 2022-01-14T04:00:13.209Z] 04:00:05     INFO - TinderboxPrint: Swap in / out<br/>17,461,248 / 1,107,316,736
[task 2022-01-14T04:00:13.209Z] 04:00:05     INFO - install-chrome-android - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2022-01-14T04:00:13.209Z] 04:00:05     INFO - install-chromium-distribution - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2022-01-14T04:00:13.209Z] 04:00:05     INFO - install - Wall time: 26s; CPU: 21%; Read bytes: 0; Write bytes: 63901696; Read time: 0; Write time: 58056
[task 2022-01-14T04:00:13.209Z] 04:00:05     INFO - run-tests - Wall time: 391s; CPU: 22%; Read bytes: 27750092800; Write bytes: 4023189504; Read time: 465348; Write time: 9795504
[task 2022-01-14T04:00:13.209Z] 04:00:05  WARNING - returning nonzero exit status -9
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → INCOMPLETE
Summary: Intermittent Error copying results /builds/task_164213225211120/workspace/build/raptor.json to upload dir /builds/task_164213225211120/workspace/build/blobber_upload_dir/perfherder-data.json → Intermittent Error copying results /builds/task_"id#"/workspace/build/raptor.json to upload dir /builds/task_"id#"/workspace/build/blobber_upload_dir/perfherder-data.json | [Errno 2] No such file or directory: '/builds/task_"id#"/<...>/raptor.json'
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 3 years ago2 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 2 years ago1 year ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 1 year ago10 months ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 10 months ago8 months ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.