Closed
Bug 1059240
Opened 10 years ago
Closed 6 years ago
Intermittent test_peerConnection_replaceTrack.html | Test timed out.
Categories
(Core :: WebRTC, defect, P4)
Tracking
()
RESOLVED
WORKSFORME
backlog | webrtc/webaudio+ |
People
(Reporter: cbook, Unassigned)
References
(Depends on 1 open bug, )
Details
(Keywords: intermittent-failure, Whiteboard: [stockwell unknown][stockwell needswork:owner])
Ubuntu VM 12.04 mozilla-inbound opt test mochitest-e10s-3 on 2014-08-27 03:00:23 PDT for push 00db70d58a1a
slave: tst-linux32-spot-1059
https://tbpl.mozilla.org/php/getParsedLog.php?id=46846046&tree=Mozilla-Inbound
11397 INFO TEST-UNEXPECTED-FAIL | /tests/dom/media/tests/mochitest/test_peerConnection_replaceTrack.html | Test timed out.
Comment 1•10 years ago
|
||
Looks like a problem with the OS level fake loopback device drivers, as gUM never returned.
(On which machines do we actually use the OS level device drivers vs. FF build in fake device?)
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment hidden (Legacy TBPL/Treeherder Robot) |
Updated•9 years ago
|
backlog: --- → webRTC+
Rank: 35
Priority: -- → P3
Comment hidden (Legacy TBPL/Treeherder Robot) |
Comment 30•7 years ago
|
||
Mass change P3->P4 to align with new Mozilla triage process.
Priority: P3 → P4
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 33•7 years ago
|
||
Over the last 7 days there are 31 failures present on this bug. These happen on Linux, Linux x64, linux32-stylo-disabled and linux64-stylo-disabled.
Here is the most recent log example: https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=162867993&lineNumber=42889
Here is a relevant part from that log:
[task 2018-02-18T10:50:58.391Z] 10:50:58 INFO - Run step 43: PC_LOCAL_AUDIOVIDEO_REPLACE_VIDEOTRACK_1
[task 2018-02-18T10:50:58.393Z] 10:50:58 INFO - TEST-PASS | dom/media/tests/mochitest/test_peerConnection_replaceTrack.html | We have a sender for video
[task 2018-02-18T10:50:58.397Z] 10:50:58 INFO - TEST-PASS | dom/media/tests/mochitest/test_peerConnection_replaceTrack.html | Shouldn't have any local streams without a corresponding sender
[task 2018-02-18T10:50:58.400Z] 10:50:58 INFO - TEST-PASS | dom/media/tests/mochitest/test_peerConnection_replaceTrack.html | Shouldn't have any remote streams without a corresponding receiver
[task 2018-02-18T10:50:58.402Z] 10:50:58 INFO - Call getUserMedia for {"video":true,"audio":{"autoGainControl":false,"echoCancellation":false,"noiseSuppression":false}}
[task 2018-02-18T10:50:58.403Z] 10:50:58 INFO - Buffered messages finished
[task 2018-02-18T10:50:58.416Z] 10:50:58 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_replaceTrack.html | Test timed out.
[task 2018-02-18T10:50:58.416Z] 10:50:58 INFO - reportError@SimpleTest/TestRunner.js:121:7
[task 2018-02-18T10:50:58.417Z] 10:50:58 INFO - TestRunner._checkForHangs@SimpleTest/TestRunner.js:142:7
Flags: needinfo?(drno)
Updated•7 years ago
|
Whiteboard: [stockwell needswork]
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Updated•7 years ago
|
Depends on: gUM_timeout
Comment 36•7 years ago
|
||
There are 40 failures in the past 7 days.
Platforms: most of them on Linux debug, linux32-stylo-disabled debug, once on Linux x64 asan, linux64-ccov opt and Windows 7 pgo
Most recent failure log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=164796045&lineNumber=9069
Reftest analyzer: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/CgdwHZ1bTEynh-9S5aJJEg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1
Waiting for update on ticket 1441563
Comment 37•7 years ago
|
||
Yes lets try to find an owner for bug 1441563. That seems more useful then trying to solve all of these intermittents individually.
Flags: needinfo?(drno)
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 58•6 years ago
|
||
There have been 38 failures in the last 7 days:
- 3 failures on Linux x64 asan+debug
- 35 failures on Linux x32 debug
Recent failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=190796727&repo=mozilla-inbound&lineNumber=175439
[task 2018-07-29T23:03:09.098Z] 23:03:09 INFO - TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_peerConnection_replaceTrack.html | Test timed out.
[task 2018-07-29T23:03:09.099Z] 23:03:09 INFO - reportError@SimpleTest/TestRunner.js:121:7
[task 2018-07-29T23:03:09.101Z] 23:03:09 INFO - TestRunner._checkForHangs@SimpleTest/TestRunner.js:142:7
[task 2018-07-29T23:03:09.103Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.104Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.105Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.106Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.107Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.108Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.109Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.110Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.111Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.112Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.115Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.115Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.116Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.116Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.117Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.121Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.123Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.127Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.128Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.129Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.134Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.135Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.139Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.143Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.143Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.144Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.155Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.157Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.159Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.166Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.166Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.166Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.166Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.179Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.181Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.186Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.187Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.191Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.192Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.194Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.212Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.216Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.224Z] 23:03:09 INFO - setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:5
[task 2018-07-29T23:03:09.226Z] 23:03:09 INFO - TestRunner.runTests/<@SimpleTest/TestRunner.js:381:9
[task 2018-07-29T23:03:09.233Z] 23:03:09 INFO - promise callback*TestRunner.runTests@SimpleTest/TestRunner.js:368:5
[task 2018-07-29T23:03:09.234Z] 23:03:09 INFO - RunSet.runtests@SimpleTest/setup.js:194:3
[task 2018-07-29T23:03:09.239Z] 23:03:09 INFO - RunSet.runall@SimpleTest/setup.js:173:5
[task 2018-07-29T23:03:09.240Z] 23:03:09 INFO - hookupTests@SimpleTest/setup.js:266:5
[task 2018-07-29T23:03:09.241Z] 23:03:09 INFO - parseTestManifest@http://mochi.test:8888/manifestLibrary.js:36:5
[task 2018-07-29T23:03:09.242Z] 23:03:09 INFO - getTestManifest/req.onload@http://mochi.test:8888/manifestLibrary.js:49:11
[task 2018-07-29T23:03:09.247Z] 23:03:09 INFO - EventHandlerNonNull*getTestManifest@http://mochi.test:8888/manifestLibrary.js:45:3
[task 2018-07-29T23:03:09.249Z] 23:03:09 INFO - hookup@SimpleTest/setup.js:246:5
[task 2018-07-29T23:03:09.250Z] 23:03:09 INFO - EventHandlerNonNull*@http://mochi.test:8888/tests?autorun=1&closeWhenDone=1&consoleLevel=INFO&manifestFile=tests.json&dumpOutputDirectory=%2Ftmp&cleanupCrashes=true:11:1
[task 2018-07-29T23:03:09.255Z] 23:03:09 INFO - GECKO(2958) | [Child 3010: Socket Thread]: D/mtransport Attempting to protect RTP...
[task 2018-07-29T23:03:09.260Z] 23:03:09 INFO - GECKO(2958) | [Child 3010: Socket Thread]: D/mtransport Attempting to unprotect RTP...
[task 2018-07-29T23:03:09.261Z] 23:03:09 INFO - GECKO(2958) | [Child 3010: Socket Thread]: D/mtransport Attempting to unprotect RTP...
[task 2018-07-29T23:03:09.263Z] 23:03:09 INFO - GECKO(2958) | [Child 3010: Socket Thread]: D/mtransport Attempting to protect RTP...
[task 2018-07-29T23:03:09.267Z] 23:03:09 INFO - GECKO(2958) | [Child 3010: Socket Thread]: D/mtransport Attempting to unprotect RTP...
[task 2018-07-29T23:03:09.270Z] 23:03:09 INFO - GECKO(2958) | [Child 3010: Socket Thread]: D/mtransport Attempting to protect RTP...
Flags: needinfo?(drno)
Whiteboard: [stockwell unknown] → [stockwell unknown][stockwell needswork:owner]
Comment 59•6 years ago
|
||
Trying to determine what caused the failure rate to jump...
Comment 60•6 years ago
|
||
Seems to have happened somewhere in this range. Will continue investigating next week.
Flags: needinfo?(docfaraday)
Comment 61•6 years ago
|
||
Flags: needinfo?(docfaraday)
Updated•6 years ago
|
Flags: needinfo?(docfaraday)
Comment 63•6 years ago
|
||
I'm having trouble isolating when this really started. When running lots of retriggers, it seems to come and go. I do not know whether this is because there is some other factor influencing its frequency, or whether several changesets are altering its probability; some making it significantly more common, others making it significantly less so.
Flags: needinfo?(docfaraday)
Comment 64•6 years ago
|
||
I see a big jump here:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&tochange=345ecb7b1895&fromchange=7afb8c53da3ae88fef9b78915a4435a7d6908169&selectedJob=192859199
but the next day, levels have dropped back down, only to jump here on autoland:
https://treeherder.mozilla.org/#/jobs?repo=autoland&tochange=8cce0e7ad6f3f7e91afe13e97e04fc0f0ed400c6&fromchange=2aca3a665afe56e1480c2358c8b9d3047cbbdc32
Comment 65•6 years ago
|
||
After running retriggers on autoland for the commits in the merge that caused the failure rate to increase on inbound, I don't see a culprit. I see a couple of possibilities here:
1. The increase is caused by an interaction between something that landed on autoland around the 19th, and something else that landed on inbound on the 20th.
2. The increase has something to do with infra or some other factor external to the code.
Comment 66•6 years ago
|
||
Because the merge on inbound (let's call this "merge A") was the first time we saw the increase (that I can tell), I have looked through the changesets merged into autoland on the 20th (let's call this "merge B") to see which of them were originally pushed to inbound before merge A happened. Here's that list, with backouts removed:
7afb8c53da3a Marco Castelluccio — Bug 1476574 - Enable reset/dump for wpt and cleanup harness code for supporting reset/dump now that it is supported by all test suites. r=jmaher
5e961986f536 Marco Castelluccio — Bug 1476574 - Support resetting/dumping code coverage counters before/after web-platform-tests. r=jgraham
76308197fa05 Ehsan Akhgari — Bug 1476796 - Enable AntiTrackingCommon::AddFirstPartyStorageAccessGrantedFor() to notify consumers about completion of asynchronous results; r=baku
d10de6036b74 Jed Davis — Bug 1467889 - Adjust some uses of XPCOM strings. r=mrbkap r=mstange
c611225375ec Eric Rahm — Bug 1442765 - Part 2: Switch nsTraceRefcnt's hashtables to use xpcom hashtables. r=mccr8
b60b9d07842e Eric Rahm — Bug 1442765 - Part 1: Add intptr_t hashkey type. r=froydnj
b9193e72be19 Sebastian Hengst — Bug 1354232 - Annotate html/infrastructure/urls/resolving-urls/query-encoding/navigation.sub.html as passing on Linux asan. r=RyanVM on IRC
It seems as though one (or more) of the changesets in this list, when combined with the changesets in merge A, causes the increase. I'm going to push some backouts to try.
Comment 67•6 years ago
|
||
Hang on a minute. The rate of this seems to have dropped to 0 on the 5th. I'm not sure whether this is because the orange-tracking for this bug has been stopped (perhaps due to my retriggers), or whether the orange has simply stopped happening. Checking that now.
Comment 68•6 years ago
|
||
Welp. This seems to have stopped happening. It might be getting masked by another intermittent, or it might actually be fixed, it is hard to say.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WORKSFORME
Comment 69•6 years ago
|
||
This appears to have been fixed by bug 1404977.
Comment hidden (Intermittent Failures Robot) |
Comment 71•5 years ago
•
|
||
The failures in comment 70 were all due to gUM_timeout, all within a two-hour timespan. Infra hiccup maybe?
You need to log in
before you can comment on or make changes to this bug.
Description
•