Open Bug 1680214 Opened 3 years ago Updated 3 days ago

Intermittent raptor-main Critical: TEST-UNEXPECTED-FAIL: no raptor test results were found for <random>

Categories

(Testing :: Raptor, defect, P5)

Product:
Testing
Component:
Raptor
Type:
defect

Tracking

(Not tracked)

Status:
REOPENED

People

(Reporter: intermittent-bug-filer, Unassigned)

References

(Depends on 1 open bug)

Details

(Keywords: intermittent-failure, leave-open, Whiteboard: [stockwell unknown])

Attachments

(1 file)

Bug 1680214 - Disable cnn live-site test. r?#perftest
48 bytes, text/x-phabricator-request
Details | Review

Filed by: archaeopteryx [at] coole-files.de
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=323263058&repo=fenix
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/N7FQU4u4QiChwkG2XkTDcA/runs/0/artifacts/public/logs/live_backing.log

[task 2020-12-02T05:47:56.245Z] 05:45:54     INFO -  raptor-browsertime Info: Starting a browsertime pageload
[task 2020-12-02T05:47:56.245Z] 05:45:54     INFO -  raptor-browsertime Info: Waiting for 1000 ms (post_startup_delay)
[task 2020-12-02T05:47:56.245Z] 05:45:55     INFO -  raptor-browsertime Info: Navigating to about:blank
[task 2020-12-02T05:47:56.245Z] 05:45:55     INFO -  raptor-browsertime Info: Navigating to url about:blank iteration 2
[task 2020-12-02T05:47:56.245Z] 05:47:56     INFO -  raptor-browsertime-android Info: removing reverse socket connections
[task 2020-12-02T05:47:56.245Z] 05:47:56     INFO -  adb command_output: adb -s HT86B1A00982 wait-for-device reverse --remove-all, timeout: None, timedout: None, exitcode: 0, output:
[task 2020-12-02T05:47:56.245Z] 05:47:56     INFO -  raptor-mitmproxy Info: Stopping mitmproxy playback, killing process 1003
[task 2020-12-02T05:47:56.245Z] 05:47:56     INFO -  raptor-mitmproxy Info: Successfully killed the mitmproxy playback process
[task 2020-12-02T05:47:56.245Z] 05:47:56     INFO -  perftest-results-handler Info: retrieving browsertime test results
[task 2020-12-02T05:47:56.245Z] 05:47:56     INFO -  perftest-results-handler Critical: unable to find browsertime results at /builds/task_160688764737577/workspace/build/blobber_upload_dir/browsertime-results/ebay-kleinanzeigen/browsertime.json
[task 2020-12-02T05:47:56.245Z] 05:47:56     INFO -  raptor-perftest Info: Removing temporary directory: /tmp/tmpRzuYSX
[task 2020-12-02T05:47:56.245Z] 05:47:56     INFO -  raptor-perftest Info: Removing temporary directory: /tmp/tmppmKRW2
[task 2020-12-02T05:47:56.245Z] 05:47:56    ERROR -  raptor-main Critical: TEST-UNEXPECTED-FAIL: no raptor test results were found for ebay-kleinanzeigen
[task 2020-12-02T05:47:56.245Z] 05:47:56    ERROR - Return code: 1
[task 2020-12-02T05:47:56.245Z] 05:47:56  WARNING - setting return code to 1
[task 2020-12-02T05:47:56.245Z] 05:47:56     INFO - Killing logcat pid 784.```
Summary: Intermittent raptor-main Critical: TEST-UNEXPECTED-FAIL: no raptor test results were found for ebay-kleinanzeigen → Intermittent raptor-main Critical: TEST-UNEXPECTED-FAIL: no raptor test results were found for <random>

Spike here is caused mainly from two machines that got quarantined.

The last spike on Feb 11 seems to have come from a worker issue on t-linux64-ms-139. The failure rate dropped this week to one failure a day so far.

Windows 2017 failures are primarily responsible for the failures here.

It seems quite likely that this is a crash, but we can't get any crash information from browsertime desktop tests at the moment (bug 1585036).

Depends on: 1585036

cc jesup since the primary failing test here is google-mail when fission is enabled on the mac webrender platform.

No longer depends on: 1585036
Depends on: 1585036

Let's see if this failure rate stays high over this week. We may reduce test to tier-3 since it's a live site test which can be flaky.

Flags: needinfo?(gmierz2)

The failures are not getting any better.

:jesup, would you have an alternative website we could use for this live site test that runs on autoland?

Flags: needinfo?(gmierz2) → needinfo?(rjesup)
Whiteboard: [stockwell needswork:owner]
Whiteboard: [stockwell needswork:owner] → [stockwell needswork:owner], [perftest-triage]
Whiteboard: [stockwell needswork:owner], [perftest-triage] → [stockwell needswork:owner]
Assignee: nobody → gmierz2
Status: NEW → ASSIGNED
Pushed by gmierz2@outlook.com:
https://hg.mozilla.org/integration/autoland/rev/5b14ef0fba04
Disable cnn live-site test. r=perftest-reviewers,kimberlythegeek

Backed out for perfdocs failures

[task 2021-06-14T19:27:38.534Z] /builds/worker/checkouts/gecko/python/mozperftest
[task 2021-06-14T19:27:43.728Z]  0:05.24 INFO Found 4 perfdocs directories in ['/builds/worker/checkouts/gecko/python/mozperftest/perfdocs', '/builds/worker/checkouts/gecko/testing/awsy/perfdocs', '/builds/worker/checkouts/gecko/testing/raptor/raptor/perfdocs', '/builds/worker/checkouts/gecko/testing/talos/perfdocs']
[task 2021-06-14T19:27:43.977Z]  0:05.44 INFO Found 4 perfdocs directories in ['/builds/worker/checkouts/gecko/python/mozperftest/perfdocs', '/builds/worker/checkouts/gecko/testing/awsy/perfdocs', '/builds/worker/checkouts/gecko/testing/raptor/raptor/perfdocs', '/builds/worker/checkouts/gecko/testing/talos/perfdocs']
[task 2021-06-14T19:27:44.969Z]  0:06.48 testing/awsy/perfdocs/config.yaml  0  error  PerfDocs are outdated, run ./mach lint -l perfdocs --fix` to update them.  Flawless performance docs. (perfdocs)
[task 2021-06-14T19:27:44.969Z]  0:06.48 testing/awsy/perfdocs/index.rst  0  error  PerfDocs are outdated, run ./mach lint -l perfdocs --fix` to update them.  Flawless performance docs. (perfdocs)
[task 2021-06-14T19:27:44.969Z]  0:06.48 testing/raptor/raptor/perfdocs/config.yml  0  error  PerfDocs are outdated, run ./mach lint -l perfdocs --fix` to update them.  Flawless performance docs. (perfdocs)
[task 2021-06-14T19:27:44.969Z]  0:06.48 testing/raptor/raptor/perfdocs/index.rst  0  error  PerfDocs are outdated, run ./mach lint -l perfdocs --fix` to update them.  Flawless performance docs. (perfdocs)
[task 2021-06-14T19:27:44.970Z]  0:06.48 testing/talos/perfdocs/config.yml  0  error  PerfDocs are outdated, run ./mach lint -l perfdocs --fix` to update them.  Flawless performance docs. (perfdocs)
[task 2021-06-14T19:27:44.970Z]  0:06.48 testing/talos/perfdocs/index.rst  0  error  PerfDocs are outdated, run ./mach lint -l perfdocs --fix` to update them.  Flawless performance docs. (perfdocs)
[task 2021-06-14T19:27:45.081Z] 19:27:45.81 perfdocs (61) | Finished in 6.74 seconds
[task 2021-06-14T19:27:45.194Z]  0:06.66 python/mozperftest/perfdocs/config.yml  0  error  PerfDocs are outdated, run ./mach lint -l perfdocs --fix` to update them.  Flawless performance docs. (perfdocs)
[task 2021-06-14T19:27:45.194Z]  0:06.66 python/mozperftest/perfdocs/index.rst  0  error  PerfDocs are outdated, run ./mach lint -l perfdocs --fix` to update them.  Flawless performance docs. (perfdocs)
[task 2021-06-14T19:27:45.304Z] 19:27:45.304 perfdocs (59) | Finished in 6.97 seconds
[task 2021-06-14T19:27:45.327Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/python/mozperftest/perfdocs/config.yml:0 | PerfDocs are outdated, run ./mach lint -l perfdocs --fix` to update them. (Flawless performance docs.)
[task 2021-06-14T19:27:45.327Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/python/mozperftest/perfdocs/index.rst:0 | PerfDocs are outdated, run ./mach lint -l perfdocs --fix` to update them. (Flawless performance docs.)
[task 2021-06-14T19:27:45.327Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/testing/awsy/perfdocs/config.yaml:0 | PerfDocs are outdated, run ./mach lint -l perfdocs --fix` to update them. (Flawless performance docs.)
[task 2021-06-14T19:27:45.327Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/testing/awsy/perfdocs/index.rst:0 | PerfDocs are outdated, run ./mach lint -l perfdocs --fix` to update them. (Flawless performance docs.)
[task 2021-06-14T19:27:45.327Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/testing/raptor/raptor/perfdocs/config.yml:0 | PerfDocs are outdated, run ./mach lint -l perfdocs --fix` to update them. (Flawless performance docs.)
[task 2021-06-14T19:27:45.327Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/testing/raptor/raptor/perfdocs/index.rst:0 | PerfDocs are outdated, run ./mach lint -l perfdocs --fix` to update them. (Flawless performance docs.)
[task 2021-06-14T19:27:45.327Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/testing/talos/perfdocs/config.yml:0 | PerfDocs are outdated, run ./mach lint -l perfdocs --fix` to update them. (Flawless performance docs.)
[task 2021-06-14T19:27:45.327Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/testing/talos/perfdocs/index.rst:0 | PerfDocs are outdated, run ./mach lint -l perfdocs --fix` to update them. (Flawless performance docs.)
[taskcluster 2021-06-14 19:27:46.012Z] === Task Finished ===
[taskcluster 2021-06-14 19:27:46.224Z] Unsuccessful task run with exit code: 1 completed in 27.723 seconds
Flags: needinfo?(gmierz2)
Pushed by gmierz2@outlook.com:
https://hg.mozilla.org/integration/autoland/rev/49b1ed67dc07
Disable cnn live-site test. r=perftest-reviewers,kimberlythegeek

https://hg.mozilla.org/mozilla-central/rev/49b1ed67dc07

Status: ASSIGNED → RESOLVED
Closed: 3 years ago
status-firefox91: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 91 Branch

Re-opening since the patch only resolved one issue here. There will still be failures with this signature.

Status: RESOLVED → REOPENED
Flags: needinfo?(gmierz2)
Keywords: leave-open
Resolution: FIXED → ---
Assignee: gmierz2 → fstrugariu

Update:
There have been 32 failures within the last 7 days:
• 8 failures on Windows 10 x64 WebRender Shippable opt
• 6 failures on OS X 10.15 WebRender Shippable opt
• 3 failures on OS X 10.15 Shippable opt
• 3 failures on OS X 10.14 WebRender Shippable opt
• 7 failures on Linux 18.04 x64 WebRender Shippable opt
• 2 failures on Linux 18.04 x64 Shippable opt
• 3 failures on linux1804-64-clang-trunk opt
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=344825362&repo=mozilla-central&lineNumber=1090

[task 2021-07-09T23:16:38.404Z] 23:16:38     INFO -  raptor-mitmproxy Info: Verified mitmproxy CA certificate is installed in Firefox
[task 2021-07-09T23:16:38.404Z] 23:16:38     INFO -  raptor-perftest Info: Playback recording information not available
[task 2021-07-09T23:16:38.405Z] 23:16:38     INFO -  raptor-browsertime Info: test: {'here': '/home/cltbld/tasks/task_162586459810088/build/tests/raptor/raptor/tests/tp6/desktop', 'alert_on': ['fcp', 'loadtime'], 'alert_threshold': '2.0', 'apps': 'firefox, chrome, chromium', 'browser_cycles': '25', 'lower_is_better': True, 'measure': ['fnbpaint', 'fcp', 'dcf', 'loadtime'], 'page_cycles': 2, 'page_timeout': '60000', 'playback': 'mitmproxy', 'playback_pageset_manifest': 'mitm5-linux-firefox-ebay.manifest', 'playback_recordings': 'mitm5-linux-firefox-ebay.mp', 'playback_version': '5.1.1', 'type': 'pageload', 'unit': 'ms', 'use_live_sites': 'false', 'test_url': 'https://www.ebay.com/', 'name': 'ebay', 'manifest': '/home/cltbld/tasks/task_162586459810088/build/tests/raptor/raptor/tests/tp6/desktop/browsertime-tp6.ini', 'manifest_relpath': 'tests/tp6/desktop/browsertime-tp6.ini', 'path': '/home/cltbld/tasks/task_162586459810088/build/tests/raptor/raptor/tests/tp6/desktop/ebay', 'relpath': 'tests/tp6/desktop/ebay', 'ancestor_manifest': 'raptor.ini', 'expected': 'pass', 'cold': True, 'expected_browser_cycles': 25, 'browser_cycle': 1}
[task 2021-07-09T23:16:38.405Z] 23:16:38     INFO -  raptor-browsertime-desktop Info: binary_path: /home/cltbld/tasks/task_162586459810088/build/application/firefox/firefox
[task 2021-07-09T23:16:38.405Z] 23:16:38     INFO -  raptor-browsertime Info: Using Firefox Window Recorder for videos
[task 2021-07-09T23:16:38.405Z] 23:16:38     INFO -  raptor-browsertime Info: timeout (s): 60000
[task 2021-07-09T23:16:38.405Z] 23:16:38     INFO -  raptor-browsertime Info: browsertime cwd: /home/cltbld/tasks/task_162586459810088/build
[task 2021-07-09T23:16:38.406Z] 23:16:38     INFO -  raptor-browsertime Info: browsertime cmd: /home/cltbld/tasks/task_162586459810088/fetches/node/bin/node /home/cltbld/tasks/task_162586459810088/fetches/browsertime/node_modules/browsertime/bin/browsertime.js --firefox.geckodriverPath /home/cltbld/tasks/task_162586459810088/fetches/geckodriver /home/cltbld/tasks/task_162586459810088/build/tests/raptor/raptor/browsertime/../../browsertime/browsertime_pageload.js --viewPort 1024x768 --browser firefox --firefox.binaryPath /home/cltbld/tasks/task_162586459810088/build/application/firefox/firefox --browsertime.page_cycles 2 --browsertime.url https://www.ebay.com/ --browsertime.page_cycle_delay 1000 --browsertime.post_startup_delay 1000 --firefox.profileTemplate /tmp/tmpww23hz9k/profile --skipHar --pageLoadStrategy none --webdriverPageload true --firefox.disableBrowsertimeExtension true --pageCompleteCheckStartWait 5000 --pageCompleteCheckPollTimeout 1000 --timeouts.pageLoad 60000 --timeouts.script 120000 --resultDir /home/cltbld/tasks/task_162586459810088/build/blobber_upload_dir/browsertime-results/ebay --firefox.env MOZ_WEBRENDER=1 --firefox.env MOZ_ACCELERATED=1 --video true --visualMetrics false --firefox.windowRecorder true -n 25
[task 2021-07-09T23:16:38.406Z] 23:16:38     INFO -  raptor-browsertime Info: browsertime_ffmpeg: /home/cltbld/tasks/task_162586459810088/fetches/ffmpeg-4.1.4-i686-static/ffmpeg
[task 2021-07-09T23:16:38.407Z] 23:16:38     INFO -  raptor-browsertime Info: PATH: b'/home/cltbld/tasks/task_162586459810088/fetches/ffmpeg-4.1.4-i686-static:/home/cltbld/tasks/task_162586459810088/build/venv/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin'
[task 2021-07-09T23:16:38.767Z] 23:16:38     INFO -  raptor-browsertime Info: Running tests using Firefox - 25 iteration(s)
[task 2021-07-09T23:16:40.929Z] 23:16:40     INFO -  raptor-browsertime Info: Starting a browsertime pageload
[task 2021-07-09T23:16:40.930Z] 23:16:40     INFO -  raptor-browsertime Info: Waiting for 1000 ms (post_startup_delay)
[task 2021-07-09T23:16:41.931Z] 23:16:41     INFO -  raptor-browsertime Info: Navigating to about:blank
[task 2021-07-09T23:16:41.932Z] 23:16:41     INFO -  raptor-browsertime Info: Navigating to url about:blank iteration 1
[task 2021-07-09T23:16:46.978Z] 23:16:46     INFO -  raptor-browsertime Info: Cycle 0, waiting for 1000 ms
[task 2021-07-09T23:16:47.980Z] 23:16:47     INFO -  raptor-browsertime Info: Cycle 0, starting the measure
[task 2021-07-09T23:16:47.981Z] 23:16:47     INFO -  raptor-browsertime Info: Testing url https://www.ebay.com/ iteration 1
[task 2021-07-09T23:16:47.983Z] 23:16:47     INFO -  raptor-browsertime Info: Start firefox window recorder.
[task 2021-07-09T23:18:48.653Z] 23:18:48     INFO -  raptor-mitmproxy Info: MitmproxyDesktop stop!!
[task 2021-07-09T23:18:48.653Z] 23:18:48     INFO -  raptor-mitmproxy Info: Mitmproxy stop!!
[task 2021-07-09T23:18:48.653Z] 23:18:48     INFO -  raptor-mitmproxy Info: Stopping mitmproxy playback, killing process 3259
[task 2021-07-09T23:18:48.839Z] 23:18:48     INFO -  raptor-mitmproxy Info: Successfully killed the mitmproxy playback process
[task 2021-07-09T23:18:48.839Z] 23:18:48     INFO -  raptor-mitmproxy Info: Turning off the browser proxy
[task 2021-07-09T23:18:48.839Z] 23:18:48     INFO -  raptor-mitmproxy Info: writing: /home/cltbld/tasks/task_162586459810088/build/application/firefox/distribution/policies.json
[task 2021-07-09T23:18:48.840Z] 23:18:48     INFO -  perftest-results-handler Info: retrieving browsertime test results
[task 2021-07-09T23:18:48.841Z] 23:18:48     INFO -  perftest-results-handler Critical: unable to find browsertime results at /home/cltbld/tasks/task_162586459810088/build/blobber_upload_dir/browsertime-results/ebay/browsertime.json
[task 2021-07-09T23:18:48.841Z] 23:18:48     INFO -  raptor-perftest Info: Removing temporary directory: /tmp/tmpww23hz9k
[task 2021-07-09T23:18:48.847Z] 23:18:48    ERROR -  raptor-main Critical: TEST-UNEXPECTED-FAIL: no raptor test results were found for ebay
[task 2021-07-09T23:18:48.908Z] 23:18:48    ERROR - Return code: 1
[task 2021-07-09T23:18:48.908Z] 23:18:48  WARNING - setting return code to 1
[task 2021-07-09T23:18:48.908Z] 23:18:48     INFO - Copying Raptor results to upload dir:
[task 2021-07-09T23:18:48.908Z] 23:18:48     INFO - /home/cltbld/tasks/task_162586459810088/build/blobber_upload_dir/perfherder-data.json
[task 2021-07-09T23:18:48.908Z] 23:18:48     INFO - Copying raptor results from /home/cltbld/tasks/task_162586459810088/build/raptor.json to /home/cltbld/tasks/task_162586459810088/build/blobber_upload_dir/perfherder-data.json
[task 2021-07-09T23:18:48.908Z] 23:18:48 CRITICAL - Error copying results /home/cltbld/tasks/task_162586459810088/build/raptor.json to upload dir /home/cltbld/tasks/task_162586459810088/build/blobber_upload_dir/perfherder-data.json
[task 2021-07-09T23:18:48.908Z] 23:18:48     INFO - [Errno 2] No such file or directory: '/home/cltbld/tasks/task_162586459810088/build/raptor.json'
[task 2021-07-09T23:18:48.909Z] 23:18:48     INFO - Running post-action listener: _package_coverage_data
[task 2021-07-09T23:18:48.909Z] 23:18:48     INFO - Running post-action listener: _resource_record_post_action
[task 2021-07-09T23:18:48.909Z] 23:18:48     INFO - Running post-action listener: process_java_coverage_data
[task 2021-07-09T23:18:48.909Z] 23:18:48     INFO - Running post-action listener: stop_device
[task 2021-07-09T23:18:48.909Z] 23:18:48     INFO - [mozharness: 2021-07-09 23:18:48.909231Z] Finished run-tests step (success)
[task 2021-07-09T23:18:48.909Z] 23:18:48     INFO - Running post-run listener: _resource_record_post_run
[task 2021-07-09T23:18:48.972Z] 23:18:48     INFO - Total resource usage - Wall time: 174s; CPU: 19%; Read bytes: 30069239808; Write bytes: 4179079168; Read time: 370512; Write time: 3952312
[task 2021-07-09T23:18:48.972Z] 23:18:48     INFO - TinderboxPrint: CPU usage<br/>19.2%
[task 2021-07-09T23:18:48.972Z] 23:18:48     INFO - TinderboxPrint: I/O read bytes / time<br/>30,069,239,808 / 370,512
[task 2021-07-09T23:18:48.972Z] 23:18:48     INFO - TinderboxPrint: I/O write bytes / time<br/>4,179,079,168 / 3,952,312
[task 2021-07-09T23:18:48.972Z] 23:18:48     INFO - TinderboxPrint: CPU idle<br/>1,058.3 (76.2%)
[task 2021-07-09T23:18:48.972Z] 23:18:48     INFO - TinderboxPrint: CPU iowait<br/>65.4 (4.7%)
[task 2021-07-09T23:18:48.972Z] 23:18:48     INFO - TinderboxPrint: CPU system<br/>45.9 (3.3%)
[task 2021-07-09T23:18:48.972Z] 23:18:48     INFO - TinderboxPrint: CPU user<br/>215.0 (15.5%)
[task 2021-07-09T23:18:48.973Z] 23:18:48     INFO - TinderboxPrint: Swap in / out<br/>0 / 1,027,600,384
[task 2021-07-09T23:18:48.973Z] 23:18:48     INFO - install-chromium-distribution - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2021-07-09T23:18:48.973Z] 23:18:48     INFO - install - Wall time: 10s; CPU: 13%; Read bytes: 122880; Write bytes: 294916096; Read time: 16; Write time: 4648
[task 2021-07-09T23:18:48.974Z] 23:18:48     INFO - run-tests - Wall time: 165s; CPU: 20%; Read bytes: 30067683328; Write bytes: 3337359360; Read time: 370456; Write time: 3914472
[task 2021-07-09T23:18:49.007Z] 23:18:49  WARNING - returning nonzero exit status 1
[taskcluster 2021-07-09T23:18:49.034Z]    Exit Code: 1
[taskcluster 2021-07-09T23:18:49.034Z]    User Time: 1m35.467704s
[taskcluster 2021-07-09T23:18:49.034Z]  Kernel Time: 11.058752s
[taskcluster 2021-07-09T23:18:49.034Z]    Wall Time: 3m31.259179973s
[taskcluster 2021-07-09T23:18:49.034Z]       Result: FAILED
[taskcluster 2021-07-09T23:18:49.035Z] === Task Finished ===
[taskcluster 2021-07-09T23:18:49.035Z] Task Duration: 3m31.264984282s
[taskcluster 2021-07-09T23:18:49.366Z] Uploading artifact public/logs/localconfig.json from file logs/localconfig.json with content encoding "gzip", mime type "application/json" and expiry 2022-07-09T21:46:07.703Z
[taskcluster 2021-07-09T23:18:50.022Z] Uploading artifact public/test_info/mitm_netlocs_dump.json from file build/blobber_upload_dir/mitm_netlocs_dump.json with content encoding "gzip", mime type "application/json" and expiry 2022-07-09T21:46:07.703Z
[taskcluster 2021-07-09T23:18:50.307Z] Uploading artifact public/test_info/mitmproxy.log from file build/blobber_upload_dir/mitmproxy.log with content encoding "gzip", mime type "text/plain" and expiry 2022-07-09T21:46:07.703Z
[taskcluster 2021-07-09T23:18:50.836Z] Uploading artifact public/test_info/resource-usage.json from file build/blobber_upload_dir/resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2022-07-09T21:46:07.703Z
[taskcluster 2021-07-09T23:18:51.175Z] Uploading redirect artifact public/logs/live.log to URL https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BuJTIQyoTW6OWPw5q_Sgdg/runs/0/artifacts/public%2Flogs%2Flive_backing.log with mime type "text/plain; charset=utf-8" and expiry 2022-07-09T21:46:07.703Z
[taskcluster:error] exit status 1
Whiteboard: [stockwell unknown] → [stockwell unknown][stockwell needswork:owner]

:bebe can you work with :kimberlythegeek to try re-recording ebay, which is one of the failing sites.

Flags: needinfo?(fstrugariu)

Recording done

Flags: needinfo?(fstrugariu)
Assignee: fstrugariu → nobody
See Also: → 1669100
status-firefox91: affected → ---
Target Milestone: 91 Branch → ---
Whiteboard: [stockwell unknown][stockwell needswork:owner] → [stockwell needswork:owner]
Depends on: 1733434
Flags: needinfo?(rjesup)

There are 29 total failures in the last 7 days on

[task 2021-12-21T04:29:33.043Z] 04:29:33     INFO -  raptor-browsertime Info: Stop firefox window recorder.
[task 2021-12-21T04:29:39.706Z] 04:29:39     INFO -  raptor-mitmproxy Info: MitmproxyDesktop stop!!
[task 2021-12-21T04:29:39.706Z] 04:29:39     INFO -  raptor-mitmproxy Info: Mitmproxy stop!!
[task 2021-12-21T04:29:39.706Z] 04:29:39     INFO -  raptor-mitmproxy Info: Stopping mitmproxy playback, killing process 1026
[task 2021-12-21T04:29:40.241Z] 04:29:40     INFO -  raptor-mitmproxy Info: Successfully killed the mitmproxy playback process
[task 2021-12-21T04:29:40.241Z] 04:29:40     INFO -  raptor-mitmproxy Info: Turning off the browser proxy
[task 2021-12-21T04:29:40.241Z] 04:29:40     INFO -  raptor-mitmproxy Info: writing: /Users/cltbld/tasks/task_1640060051/build/application/Firefox Nightly.app/Contents/Resources/distribution/policies.json
[task 2021-12-21T04:29:40.242Z] 04:29:40     INFO -  perftest-results-handler Info: retrieving browsertime test results
[task 2021-12-21T04:29:40.242Z] 04:29:40     INFO -  perftest-results-handler Critical: unable to find browsertime results at /Users/cltbld/tasks/task_1640060051/build/blobber_upload_dir/browsertime-results/reddit-billgates-ama/browsertime.json
[task 2021-12-21T04:29:40.243Z] 04:29:40     INFO -  raptor-perftest Info: Removing temporary directory: /var/folders/ns/8_v12r_52yb3x70z2n37lrbh000017/T/tmpq2jtx8wf
[task 2021-12-21T04:29:40.260Z] 04:29:40    ERROR -  raptor-main Critical: TEST-UNEXPECTED-FAIL: no raptor test results were found for reddit-billgates-ama
[task 2021-12-21T04:29:40.394Z] 04:29:40    ERROR - Return code: 1
[task 2021-12-21T04:29:40.394Z] 04:29:40  WARNING - setting return code to 1
[task 2021-12-21T04:29:40.394Z] 04:29:40     INFO - Copying Raptor results to upload dir:
[task 2021-12-21T04:29:40.394Z] 04:29:40     INFO - /Users/cltbld/tasks/task_1640060051/build/blobber_upload_dir/perfherder-data.json
[task 2021-12-21T04:29:40.394Z] 04:29:40     INFO - Copying raptor results from /Users/cltbld/tasks/task_1640060051/build/raptor.json to /Users/cltbld/tasks/task_1640060051/build/blobber_upload_dir/perfherder-data.json
[task 2021-12-21T04:29:40.394Z] 04:29:40 CRITICAL - Error copying results /Users/cltbld/tasks/task_1640060051/build/raptor.json to upload dir /Users/cltbld/tasks/task_1640060051/build/blobber_upload_dir/perfherder-data.json
[task 2021-12-21T04:29:40.395Z] 04:29:40     INFO - [Errno 2] No such file or directory: '/Users/cltbld/tasks/task_1640060051/build/raptor.json'
[task 2021-12-21T04:29:40.395Z] 04:29:40     INFO - Running post-action listener: _package_coverage_data
[task 2021-12-21T04:29:40.395Z] 04:29:40     INFO - Running post-action listener: _resource_record_post_action
[task 2021-12-21T04:29:40.395Z] 04:29:40     INFO - Running post-action listener: process_java_coverage_data
[task 2021-12-21T04:29:40.396Z] 04:29:40     INFO - Running post-action listener: stop_device
[task 2021-12-21T04:29:40.396Z] 04:29:40     INFO - [mozharness: 2021-12-21 04:29:40.396094Z] Finished run-tests step (success)
[task 2021-12-21T04:29:40.396Z] 04:29:40     INFO - Running post-run listener: _resource_record_post_run
[task 2021-12-21T04:29:40.507Z] 04:29:40     INFO - Total resource usage - Wall time: 879s; CPU: 25%; Read bytes: 478781440; Write bytes: 6633571840; Read time: 10569; Write time: 31460
[task 2021-12-21T04:29:40.507Z] 04:29:40     INFO - TinderboxPrint: CPU usage<br/>24.6%
[task 2021-12-21T04:29:40.507Z] 04:29:40     INFO - TinderboxPrint: I/O read bytes / time<br/>478,781,440 / 10,569
[task 2021-12-21T04:29:40.507Z] 04:29:40     INFO - TinderboxPrint: I/O write bytes / time<br/>6,633,571,840 / 31,460
[task 2021-12-21T04:29:40.507Z] 04:29:40     INFO - TinderboxPrint: CPU idle<br/>2,154.2 (61.3%)
[task 2021-12-21T04:29:40.507Z] 04:29:40     INFO - TinderboxPrint: CPU system<br/>205.3 (5.8%)
[task 2021-12-21T04:29:40.507Z] 04:29:40     INFO - TinderboxPrint: CPU user<br/>1,155.2 (32.9%)
[task 2021-12-21T04:29:40.507Z] 04:29:40     INFO - TinderboxPrint: Swap in / out<br/>667,766,784 / 0
[task 2021-12-21T04:29:40.508Z] 04:29:40     INFO - install-chromium-distribution - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2021-12-21T04:29:40.509Z] 04:29:40     INFO - install - Wall time: 32s; CPU: 38%; Read bytes: 442832384; Write bytes: 447934464; Read time: 28901; Write time: 1136
[task 2021-12-21T04:29:40.514Z] 04:29:40     INFO - run-tests - Wall time: 848s; CPU: 24%; Read bytes: 467918848; Write bytes: 6181672448; Read time: 9907; Write time: 30292
[task 2021-12-21T04:29:40.643Z] 04:29:40  WARNING - returning nonzero exit status 1
[taskcluster 2021-12-21T04:29:40.692Z]    Exit Code: 1
[taskcluster 2021-12-21T04:29:40.692Z]    User Time: 17m38.970908s
[taskcluster 2021-12-21T04:29:40.692Z]  Kernel Time: 2m8.211281s
[taskcluster 2021-12-21T04:29:40.692Z]    Wall Time: 15m28.08080805s
[taskcluster 2021-12-21T04:29:40.692Z]       Result: FAILED

Greg, can you please take a look?

Flags: needinfo?(gmierz2)
Whiteboard: [stockwell unknown] → [stockwell needswork:owner]

Seems like a mitmproxy error. The error doesn't depend in the type of performance test (pageload, live, responsiveness, desktop, mobile). I have a wild guess that it might be from alternate server replay. mitmproxy is killed after several 404 received (mitmlog). kimberly?

Update: seems like the tests are failing after a bunch of calls like below:

127.0.0.1:41912: GET http://detectportal.firefox.com/canonical.html
              << 404 Not Found 0b
Flags: needinfo?(ksereduck)

I'm unsure what caused the recent spike in failures, but I don't think this detectportal.firefox.com call is what is causing the test to fail. It happens on every replay as far as I know.

Also, 9 of these failures are live tests, which shouldn't be affected by kill non-replay requests. (This would put the failures at 9 for live and 12 for non-live, under the 15 failure limit) I suspect that the failures in the live tests are not related to the recorded page load failures, but I will look into this further.

A reason for the espn mobile failure could be this:

127.0.0.1:54557: GET http://www.espn.com/nba/story/_/page/allstarweekend2578…
              << 301 Moved Permanently 2.44k
Flags: needinfo?(gmierz2)

The recent failures are all live tests, except one. These seem to be issues with the cnn website. The failures have steadily gone up over the past few months. The last time the failures were above 20 was three months ago, and all of them were on the cnn live site. I'm unsure if this is something we can resolve on our end.

Flags: needinfo?(ksereduck)

:kshampur, can you look into this failure?

Flags: needinfo?(kshampur)

quick update: majority of the issues from the last two alerts are on Beta + MacOS and consistently with Imgur, where after several iterations it will crash right before going to the secondary_url.

So far unsuccessful in reproducing locally on my Mac, but I have some Try pushes going that I am hopeful will reproduce this and I will review those early next week.

I am also curious in seeing this upcoming Monday's failure bot report, given that it already dropped from 32 - > 20 (though only 26->17 for beta)

keeping ni? active

The failures are down 20 -> 6 now,
I will keep ni? and monitor this for one more week

Flags: needinfo?(kshampur)

The severity field for this bug is set to S4. However, the following bug duplicate has higher severity:

:afinder, could you consider increasing the severity of this bug to S3?

For more information, please visit auto_nag documentation.

Flags: needinfo?(afinder)
Severity: S4 → S3
Flags: needinfo?(afinder)
Duplicate of this bug: 1890445
Duplicate of this bug: 1890461
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: