1066775 - WebRTC mochitests fail in many different configs with "Received end of ICE gather candidate"

Status: RESOLVED FIXED

(Core :: WebRTC, defect)

Description

[Syd Polk :sydpolk]

Attachment: Client 1.log

Many configurations running the steeplechase WebRTC tests fail with this:

steeplechase INFO  | {"action":"test_unexpected_fail","message":"Received end of ICE gathering candidate","time":1410543019291,"source_file":"dom/media/tests/mochitest/test_peerConnection_basicAudio.html"}

This was run (in this test) with host1 being linux32 and host2 being linux64:

python /Users/mozilla/jenkins/workspace/nightly-ubuntu14_32-ubuntu14_64/steeplechase/steeplechase/runsteeplechase.py --binary /home/mozilla/firefoxes/nightly/linux32/firefox/firefox --binary2 /home/mozilla/firefoxes/nightly/linux64/firefox/firefox --save-logs-to /Users/mozilla/jenkins/workspace/nightly-ubuntu14_32-ubuntu14_64/logs --specialpowers-path /home/mozilla/firefoxes/nightly/linux64/tests/steeplechase/specialpowers --prefs-file /home/mozilla/firefoxes/nightly/linux64/tests/steeplechase/prefs_general.js --signalling-server http://172.16.141.52:8080/ --html-manifest /home/mozilla/firefoxes/nightly/linux64/tests/steeplechase/tests/steeplechase.ini --host1 172.16.141.55:20701 --host2 172.16.141.52:20701

Attaching client logs.

Comment 1

[Syd Polk :sydpolk]

Attachment: Client 2.log

Comment 2

[Syd Polk :sydpolk]

Nils, this is the issue I filed as you were leaving on vacation.

Comment 3

[Nils Ohlmeier [:drno]]

Thanks Syd. I actually don't need to see machines which reproduced the problem. I already know what is going on.

Comment 4

[Nils Ohlmeier [:drno]]

Attachment: bug_1066775_fix_end_of_gathering.patch

Turns out that ICE gathering is slower in real world scenarios like steeplechase then I was hoping. Lets remove the check for now.

Try run: https://tbpl.mozilla.org/?tree=Try&rev=da4c92b5aa80

Comment 5

[Syd Polk :sydpolk]

Comment on attachment 8494693 [details] [diff] [review]
bug_1066775_fix_end_of_gathering.patch

Review of attachment 8494693 [details] [diff] [review]:
-----------------------------------------------------------------

This looks good. You are removing a test. It is likely to be too slow in a VM environment. Removing it seems safe.

Comment 6

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ecfe7d352a03

Comment 7

[Nils Ohlmeier [:drno]]

Reminder to request uplift once settled in.

Comment 8

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/ecfe7d352a03

Comment 9

[Nils Ohlmeier [:drno]]

Comment on attachment 8494693 [details] [diff] [review]
bug_1066775_fix_end_of_gathering.patch

Approval Request Comment
[Feature/regressing bug #]: This problem got introduced recently by bug 1041832 and got uplifted by the release train.
[User impact if declined]: If not uplifted our steeplechase tests will continue to show intermittent failures.
[Describe test coverage new/current, TBPL]: This only takes tests steps out which were too aggressive (to optimistic regarding the timing in the network).
[Risks and why]: No risks as we only take out test checkpoints.
[String/UUID change made/needed]: n/a

Comment 10

[Lawrence Mandel [:lmandel] (use needinfo)]

Comment on attachment 8494693 [details] [diff] [review]
bug_1066775_fix_end_of_gathering.patch

Test only fix. Aurora+

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/29e38b914425