Closed Bug 1806746 Opened 2 years ago Closed 2 years ago

Intermittent TVw [tier 2] ::: Test verification FAIL | TinderboxPrint: Per-test run of .../RTCRtpReceiver-getStats.https.html<br/>: FAILURE

Categories

(Core :: WebRTC, defect, P5)

Product:
Core
Component:
WebRTC
Type:
defect

Tracking

()

Status:
RESOLVED INCOMPLETE
Tracking Flags:
Tracking Status
firefox-esr102 --- unaffected
firefox108 --- unaffected
firefox109 --- unaffected
firefox110 --- wontfix

People

(Reporter: intermittent-bug-filer, Unassigned)

References

(Regression)

Details

(Keywords: intermittent-failure, regression)

Filed by: mlaza [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=400298680&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LX9KmXaOTNO4dKHrG7kIzQ/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LX9KmXaOTNO4dKHrG7kIzQ/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

[task 2022-12-20T23:06:22.902Z] 23:06:22     INFO - | `/webrtc/RTCRtpReceiver-getStats.https.html` | `receiver.getStats() should work with a closed PeerConnection` | **FAIL: 9/10, PASS: 1/10** | `assert_equals: Expect dictionary.transportId to be string expected "string" but got "undefined"` |
[task 2022-12-20T23:06:22.902Z] 23:06:22     INFO - 
[task 2022-12-20T23:06:22.903Z] 23:06:22     INFO - ::: Running tests in a loop 10 times : FAIL
[task 2022-12-20T23:06:22.903Z] 23:06:22     INFO - :::
[task 2022-12-20T23:06:22.903Z] 23:06:22    ERROR - ::: Test verification FAIL
[task 2022-12-20T23:06:22.904Z] 23:06:22     INFO - :::
[task 2022-12-20T23:06:23.207Z] 23:06:23    ERROR - Return code: 1
[task 2022-12-20T23:06:23.208Z] 23:06:23    ERROR - # TBPL FAILURE #
[task 2022-12-20T23:06:23.208Z] 23:06:23  WARNING - setting return code to 2
[task 2022-12-20T23:06:23.208Z] 23:06:23    ERROR - TinderboxPrint: Per-test run of .../RTCRtpReceiver-getStats.https.html<br/>: FAILURE
[task 2022-12-20T23:06:23.210Z] 23:06:23     INFO - Running command: ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/web-platform/runtests.py', '--log-raw=-', '--log-raw=/builds/worker/workspace/build/blobber_upload_dir/wpt-test2_raw.log', '--log-wptreport=/builds/worker/workspace/build/blobber_upload_dir/wptreport.json', '--log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/wpt-test2_errorsummary.log', '--symbols-path=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BiEN1AzET--M7SEacwf-IA/artifacts/public/build/target.crashreporter-symbols.zip', '--stackwalk-binary=/builds/worker/fetches/minidump-stackwalk/minidump-stackwalk', '--stackfix-dir=/builds/worker/workspace/build/tests/bin', '--no-pause-after-test', '--instrument-to-file=/builds/worker/workspace/build/blobber_upload_dir/wpt_instruments.txt', '--specialpowers-path=/builds/worker/workspace/build/tests/extensions/specialpowers@mozilla.org.xpi', '--processes=1', '--binary=/builds/worker/workspace/build/application/firefox/firefox', '--install-fonts', '--test-type=testharness', '--setpref=media.peerconnection.mtransport_process=false', '--setpref=network.process.enabled=false', '--setpref=layers.d3d11.enable-blacklist=false', '--prefs-root=/builds/worker/workspace/build/tests/web-platform/prefs', '--config=/builds/worker/workspace/build/tests/web-platform/wptrunner.ini', '--ca-cert-path=/builds/worker/workspace/build/tests/web-platform/tests/tools/certs/cacert.pem', '--host-key-path=/builds/worker/workspace/build/tests/web-platform/tests/tools/certs/web-platform.test.key', '--host-cert-path=/builds/worker/workspace/build/tests/web-platform/tests/tools/certs/web-platform.test.pem', '--certutil-binary=/builds/worker/workspace/build/tests/bin/certutil', '--verify-log-full', '--verify', 'tests/web-platform/tests/webrtc/RTCRtpSender-getStats.https.html'] in /builds/worker/workspace/build
[task 2022-12-20T23:06:23.210Z] 23:06:23     INFO - Copy/paste: /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/web-platform/runtests.py --log-raw=- --log-raw=/builds/worker/workspace/build/blobber_upload_dir/wpt-test2_raw.log --log-wptreport=/builds/worker/workspace/build/blobber_upload_dir/wptreport.json --log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/wpt-test2_errorsummary.log --symbols-path=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BiEN1AzET--M7SEacwf-IA/artifacts/public/build/target.crashreporter-symbols.zip --stackwalk-binary=/builds/worker/fetches/minidump-stackwalk/minidump-stackwalk --stackfix-dir=/builds/worker/workspace/build/tests/bin --no-pause-after-test --instrument-to-file=/builds/worker/workspace/build/blobber_upload_dir/wpt_instruments.txt --specialpowers-path=/builds/worker/workspace/build/tests/extensions/specialpowers@mozilla.org.xpi --processes=1 --binary=/builds/worker/workspace/build/application/firefox/firefox --install-fonts --test-type=testharness --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --setpref=layers.d3d11.enable-blacklist=false --prefs-root=/builds/worker/workspace/build/tests/web-platform/prefs --config=/builds/worker/workspace/build/tests/web-platform/wptrunner.ini --ca-cert-path=/builds/worker/workspace/build/tests/web-platform/tests/tools/certs/cacert.pem --host-key-path=/builds/worker/workspace/build/tests/web-platform/tests/tools/certs/web-platform.test.key --host-cert-path=/builds/worker/workspace/build/tests/web-platform/tests/tools/certs/web-platform.test.pem --certutil-binary=/builds/worker/workspace/build/tests/bin/certutil --verify-log-full --verify tests/web-platform/tests/webrtc/RTCRtpSender-getStats.https.html
[task 2022-12-20T23:06:23.210Z] 23:06:23     INFO - Using env: (same as previous command)
[task 2022-12-20T23:06:23.210Z] 23:06:23     INFO - Calling ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/web-platform/runtests.py', '--log-raw=-', '--log-raw=/builds/worker/workspace/build/blobber_upload_dir/wpt-test2_raw.log', '--log-wptreport=/builds/worker/workspace/build/blobber_upload_dir/wptreport.json', '--log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/wpt-test2_errorsummary.log', '--symbols-path=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BiEN1AzET--M7SEacwf-IA/artifacts/public/build/target.crashreporter-symbols.zip', '--stackwalk-binary=/builds/worker/fetches/minidump-stackwalk/minidump-stackwalk', '--stackfix-dir=/builds/worker/workspace/build/tests/bin', '--no-pause-after-test', '--instrument-to-file=/builds/worker/workspace/build/blobber_upload_dir/wpt_instruments.txt', '--specialpowers-path=/builds/worker/workspace/build/tests/extensions/specialpowers@mozilla.org.xpi', '--processes=1', '--binary=/builds/worker/workspace/build/application/firefox/firefox', '--install-fonts', '--test-type=testharness', '--setpref=media.peerconnection.mtransport_process=false', '--setpref=network.process.enabled=false', '--setpref=layers.d3d11.enable-blacklist=false', '--prefs-root=/builds/worker/workspace/build/tests/web-platform/prefs', '--config=/builds/worker/workspace/build/tests/web-platform/wptrunner.ini', '--ca-cert-path=/builds/worker/workspace/build/tests/web-platform/tests/tools/certs/cacert.pem', '--host-key-path=/builds/worker/workspace/build/tests/web-platform/tests/tools/certs/web-platform.test.key', '--host-cert-path=/builds/worker/workspace/build/tests/web-platform/tests/tools/certs/web-platform.test.pem', '--certutil-binary=/builds/worker/workspace/build/tests/bin/certutil', '--verify-log-full', '--verify', 'tests/web-platform/tests/webrtc/RTCRtpSender-getStats.https.html'] with output_timeout 1000
[task 2022-12-20T23:06:23.430Z] 23:06:23     INFO - :::
[task 2022-12-20T23:06:23.431Z] 23:06:23     INFO - ::: Running test verification step "Running tests in a loop 10 times"...
[task 2022-12-20T23:06:23.431Z] 23:06:23     INFO - :::
[task 2022-12-20T23:06:23.670Z] 23:06:23     INFO - mozversion application_buildid: 20221220223543
[task 2022-12-20T23:06:23.670Z] 23:06:23     INFO - mozversion application_changeset: fb1a765ec213b12a3c6aa1474e8dc42585b1c2aa
[task 2022-12-20T23:06:23.671Z] 23:06:23     INFO - mozversion application_display_name: Firefox Nightly
[task 2022-12-20T23:06:23.671Z] 23:06:23     INFO - mozversion application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
[task 2022-12-20T23:06:23.671Z] 23:06:23     INFO - mozversion application_name: Firefox
[task 2022-12-20T23:06:23.671Z] 23:06:23     INFO - mozversion application_remotingname: firefox-default
[task 2022-12-20T23:06:23.672Z] 23:06:23     INFO - mozversion application_repository: https://hg.mozilla.org/integration/autoland
[task 2022-12-20T23:06:23.672Z] 23:06:23     INFO - mozversion application_vendor: Mozilla
[task 2022-12-20T23:06:23.672Z] 23:06:23     INFO - mozversion application_version: 110.0a1
[task 2022-12-20T23:06:23.672Z] 23:06:23     INFO - mozversion platform_buildid: 20221220223543
[task 2022-12-20T23:06:23.672Z] 23:06:23     INFO - mozversion platform_changeset: fb1a765ec213b12a3c6aa1474e8dc42585b1c2aa
[task 2022-12-20T23:06:23.672Z] 23:06:23     INFO - mozversion platform_repository: https://hg.mozilla.org/integration/autoland
[task 2022-12-20T23:06:23.672Z] 23:06:23     INFO - mozversion platform_version: 110.0a1
[task 2022-12-20T23:06:26.358Z] 23:06:26     INFO - Using 1 client processes
[task 2022-12-20T23:06:26.886Z] 23:06:26     INFO - Installed font: Ahem.ttf
[task 2022-12-20T23:06:27.184Z] 23:06:27     INFO - wptserve Starting http server on http://127.0.0.1:8000
[task 2022-12-20T23:06:27.457Z] 23:06:27     INFO - wptserve Starting http server on http://127.0.0.1:8001
[task 2022-12-20T23:06:27.740Z] 23:06:27     INFO - wptserve Starting http server on http://127.0.0.1:8002
[task 2022-12-20T23:06:28.018Z] 23:06:28     INFO - wptserve Starting http server on http://127.0.0.1:8003
[task 2022-12-20T23:06:28.297Z] 23:06:28     INFO - wptserve Starting https server on https://127.0.0.1:8443
[task 2022-12-20T23:06:28.565Z] 23:06:28     INFO - wptserve Starting https server on https://127.0.0.1:8444
[task 2022-12-20T23:06:28.838Z] 23:06:28     INFO - wptserve Starting https server on https://127.0.0.1:8445
[task 2022-12-20T23:06:29.111Z] 23:06:29     INFO - wptserve Starting https server on https://127.0.0.1:8446
[task 2022-12-20T23:06:29.387Z] 23:06:29     INFO - wptserve Create socket on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8888))
[task 2022-12-20T23:06:29.388Z] 23:06:29     INFO - wptserve Bind on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8888))
[task 2022-12-20T23:06:29.388Z] 23:06:29     INFO - wptserve Listen on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8888))
[task 2022-12-20T23:06:29.670Z] 23:06:29     INFO - wptserve Create socket on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8889))
[task 2022-12-20T23:06:29.672Z] 23:06:29     INFO - wptserve Bind on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8889))
[task 2022-12-20T23:06:29.673Z] 23:06:29     INFO - wptserve Listen on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8889))
[task 2022-12-20T23:06:29.945Z] 23:06:29     INFO - wptserve Starting http2 server on https://127.0.0.1:9000
[task 2022-12-20T23:06:30.431Z] 23:06:30     INFO - SUITE-START | Running 1 tests

:bwc, since you are the author of the regressor, bug 1800136, could you take a look?

For more information, please visit auto_nag documentation.

Flags: needinfo?(docfaraday)

That test case has been known intermittent for almost a year. Not sure why TV would notice only now.

Flags: needinfo?(docfaraday)

Set release status flags based on info from the regressing bug 1800136

status-firefox108: --- → unaffected
status-firefox109: --- → unaffected
status-firefox110: --- → affected
status-firefox-esr102: --- → unaffected

TV does not seem to pay attention to the [PASS, FAIL] in the ini file for this test. Is this a known limitation of TV?

Flags: needinfo?(james)

Yeah, it seems like the stability check only checks the actual status, and doesn't care at all about the expected status(es). I filed https://github.com/web-platform-tests/wpt/issues/37736

Flags: needinfo?(james)
Flags: needinfo?(masoud.ahma365)

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: NEW → RESOLVED
Closed: 2 years ago
status-firefox110: affected → wontfix
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.