1373858 - Intermittent dom/media/tests/mochitest/test_peerConnection_stats.html | candidate-pair.bytesSent was tested.

Status: RESOLVED FIXED

(Core :: WebRTC, defect, P3)

Description

[Treeherder Bug Filer]

Filed by: wkocher [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=107824204&repo=autoland

https://autophone.s3.amazonaws.com/v1/task/HJL4OzcERl-ILOCU9EbZUw/runs/0/artifacts/public/build/ec4a1f59-c34c-43e1-b188-b8d685306e26-autophone.log

Comment 1

[Intermittent Failures Robot]

14 failures in 814 pushes (0.017 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 13
* mozilla-inbound: 1

Platform breakdown:
* android-6-0-armv8-api15: 10
* osx-10-10: 1
* linux64-qr: 1
* linux64: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373858&startday=2017-06-12&endday=2017-06-18&tree=all

Comment 2

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

this is primarily on autophone, but also seen on other platforms as well.  Seems to be consistent since June 16th (3 days, 34 failures)

:mreavy- I see you as the triage owner for this component, could you find someone from the webRTC team to look into this and help resolve this in the next 2 weeks?

Comment 3

[Intermittent Failures Robot]

22 failures in 151 pushes (0.146 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 11
* mozilla-inbound: 5
* mozilla-central: 4
* try: 2

Platform breakdown:
* linux64: 7
* linux64-qr: 6
* android-6-0-armv8-api15: 6
* linux32: 2
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373858&startday=2017-06-19&endday=2017-06-19&tree=all

Comment 4

[Intermittent Failures Robot]

28 failures in 173 pushes (0.162 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 13
* mozilla-inbound: 11
* mozilla-central: 3
* try: 1

Platform breakdown:
* android-6-0-armv8-api15: 14
* linux64-qr: 5
* linux64: 3
* linux32: 2
* android-7-1-armv8-api15: 2
* osx-10-10: 1
* android-4-4-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373858&startday=2017-06-20&endday=2017-06-20&tree=all

Comment 5

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

:bc, could we disable this on autophone for the time being?  That would reduce the failures, 28 total yesterday and 17 were on autophone.

Comment 6

[Bob Clary [:bc] (inactive)]

Just this one test? Sure.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

well there could be others, but this one test seems to be failing quite often.

Comment 8

[Bob Clary [:bc] (inactive)]

Attachment: bug-1373858.patch

I don't think we need a try run, do you?

Comment 9

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

Comment on attachment 8879904 [details] [diff] [review]
bug-1373858.patch

Review of attachment 8879904 [details] [diff] [review]:
-----------------------------------------------------------------

thanks; I don't think we need a try run, worse case scenario the failure is related to something else and we will see more failures in another test in this directory.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7dc1aafc5eff
Disable dom/media/tests/mochitest/test_peerConnection_stats.html for Android failures, r=jmaher.

Comment 11

[Intermittent Failures Robot]

24 failures in 170 pushes (0.141 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 15
* mozilla-inbound: 5
* mozilla-central: 4

Platform breakdown:
* android-6-0-armv8-api15: 12
* osx-10-10: 3
* linux64: 3
* linux64-qr: 2
* windows8-64: 1
* windows7-32-vm: 1
* linux32: 1
* android-7-1-armv8-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373858&startday=2017-06-21&endday=2017-06-21&tree=all

Comment 12

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/7dc1aafc5eff

Comment 13

[Intermittent Failures Robot]

96 failures in 892 pushes (0.108 failures/push) were associated with this bug in the last 7 days. 

This is the #16 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 41
* mozilla-inbound: 28
* mozilla-central: 15
* try: 12

Platform breakdown:
* android-6-0-armv8-api15: 33
* linux64: 22
* linux64-qr: 14
* osx-10-10: 13
* linux32: 8
* android-7-1-armv8-api15: 3
* windows8-64: 1
* windows7-32-vm: 1
* android-4-4-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373858&startday=2017-06-19&endday=2017-06-25&tree=all

Comment 14

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

ok, disabling on android didn't solve the problem, since we have a 62 failures on non android, it would be nice if we could get someone from the team to look at this- I will disable this on my wednesday cleanup if there is no action here.

Comment 15

[Michael Froman [:mjf]]

(In reply to Joel Maher ( :jmaher) from comment #14)
> ok, disabling on android didn't solve the problem, since we have a 62
> failures on non android, it would be nice if we could get someone from the
> team to look at this- I will disable this on my wednesday cleanup if there
> is no action here.

Sorry - I don't know how I missed this particular bug.  It is additional fallout from the previous Orange factor fix I made, and is timing-related.  Please give me a day or two to look at it, as this is the first I'd heard about it rising up in the results.

Comment 16

[Maire Reavy [:mreavy]]

Apologies that this took so long to get to Michael.  Thanks for taking this, Michael.

Comment 17

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

thanks :mjf, let me know if you have troubles reproducing this or need more information.

Comment 18

[Michael Froman [:mjf]]

Attachment: Bug 1373858 - make sure all fields are checked in the cancelled pairs.

Review commit: https://reviewboard.mozilla.org/r/153478/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/153478/

Comment 19

[Nils Ohlmeier [:drno]]

Comment on attachment 8882374 [details]
Bug 1373858 - make sure all fields are checked in the cancelled pairs.

https://reviewboard.mozilla.org/r/153478/#review158656

::: dom/media/tests/mochitest/test_peerConnection_stats.html:379
(Diff revision 1)
>          ok(stat.framesEncoded >= 0 && stat.framesEncoded < 100000, stat.type
>            + ".framesEncoded is a sane number for a short test. value="
>            + stat.framesEncoded);
>        }
>      } else if (stat.type == "candidate-pair") {
> +      info("candidate-pair is: " + JSON.stringify(stat));

Do you want to keep this, or is that development debug message only?

::: dom/media/tests/mochitest/test_peerConnection_stats.html:418
(Diff revision 1)
>  
>        // state
> -      if (stat.state == "succeeded") {
> +      if (stat.state == "succeeded" &&
> +          stat.selected !== undefined &&
> +          stat.selected) {
> +        info("candidate-pair state is succeeded and selected is true");

Same question here.

Comment 20

[Michael Froman [:mjf]]

Comment on attachment 8882374 [details]
Bug 1373858 - make sure all fields are checked in the cancelled pairs.

https://reviewboard.mozilla.org/r/153478/#review158656

> Do you want to keep this, or is that development debug message only?

Yeah, I debated on this, but ultimately decided to leave it in case other bugs come up here.

> Same question here.

Ditto above.

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2a0702d83512
make sure all fields are checked in the cancelled pairs. r=drno

Comment 22

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

we might want to enable this on autophone as well since we disabled it there not too long ago to reduce the failures here.

Comment 23

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/2a0702d83512

Comment 24

[Intermittent Failures Robot]

63 failures in 718 pushes (0.088 failures/push) were associated with this bug in the last 7 days. 

This is the #16 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 26
* mozilla-inbound: 14
* mozilla-central: 13
* try: 10

Platform breakdown:
* linux64: 25
* linux64-qr: 16
* osx-10-10: 11
* linux32: 7
* windows7-32-vm: 2
* windows8-64: 1
* linux64-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373858&startday=2017-06-26&endday=2017-07-02&tree=all