1651601 - Intermittent Assertion failed: (0), function nr_ice_media_stream_start_checks, file /builds/worker/checkouts/gecko/media/mtransport/third_party/nICEr/src/ice/ice_media_stream.c, line 414.

Status: RESOLVED FIXED

(Core :: WebRTC, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: rmaries [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=309097152&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/eoTjFjO5QHGI6-uceCBHug/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-07-09T06:47:08.602Z] 06:47:08 INFO - TEST-START | /webrtc/RTCPeerConnection-restartIce.https.html
[task 2020-07-09T06:47:10.389Z] 06:47:10 INFO - PID 2798 | (ice/INFO) ICE-PEER(PC:1594277230268608 (id=23622320136 url=https://web-platform.test:8443/webrtc/RTCPeerConnection-restartIce.https.html):default)/CAND-PAIR(5Bq9): setting pair to state FROZEN: 5Bq9|IP4:10.49.56.235:65493/UDP|IP4:10.49.56.235:50551/UDP(host(IP4:10.49.56.235:65493/UDP)|candidate:0 1 UDP 2122252543 10.49.56.235 50551 typ host)
[task 2020-07-09T06:47:10.389Z] 06:47:10 INFO - PID 2798 | (ice/INFO) ICE(PC:1594277230268608 (id=23622320136 url=https://web-platform.test:8443/webrtc/RTCPeerConnection-restartIce.https.html))/CAND-PAIR(5Bq9): Pairing candidate IP4:10.49.56.235:65493/UDP (7e7f00ff):IP4:10.49.56.235:50551/UDP (7e7f00ff) priority=9115005270282338815 (7e7f00fffcfe01ff)
[task 2020-07-09T06:47:10.389Z] 06:47:10 INFO - PID 2798 | Assertion failed: (0), function nr_ice_media_stream_start_checks, file /builds/worker/checkouts/gecko/media/mtransport/third_party/nICEr/src/ice/ice_media_stream.c, line 414.
[task 2020-07-09T06:47:10.667Z] 06:47:10 INFO - PID 2798 | [Parent 2798, Unnamed thread 1276f52f0] WARNING: Resource acquired is being released in non-LIFO order; why?
[task 2020-07-09T06:47:10.667Z] 06:47:10 INFO - PID 2798 | : file /builds/worker/checkouts/gecko/xpcom/threads/BlockingResourceBase.cpp, line 292
[task 2020-07-09T06:47:10.667Z] 06:47:10 INFO - PID 2798 | --- Mutex : dumpSafetyLock (currently acquired)
[task 2020-07-09T06:47:10.668Z] 06:47:10 INFO - PID 2798 | calling context
[task 2020-07-09T06:47:10.668Z] 06:47:10 INFO - PID 2798 | [stack trace unavailable]

Comment 1

[Byron Campen [:bwc]]

Assertion is here: https://searchfox.org/mozilla-central/rev/8d55e18875b89cdf2a22a7cba60dc40999c18356/media/mtransport/third_party/nICEr/src/ice/ice_media_stream.c#414

Comment 2

[Byron Campen [:bwc]]

Attachment: Bug 1651601: Filter gathered candidates when their component is obsolete. r?mjf

Comment 3

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=cf711e9fe7ab267e94481ea41be49e7717a558b8

Comment 4

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=442d0d8bc957982bbc897ae9c6eda3325f629088

Comment 5

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b78bea0e5cd54c4fd0a31b349e8b8474db5a5558

Comment 6

[Byron Campen [:bwc]]

Try looks good, seeing if I can hit the assertion with some retriggers.

Comment 7

[Intermittent Failures Robot]

24 failures in 5265 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 17
• mozilla-central: 4
• try: 1
• mozilla-beta: 2

Platform breakdown:

• macosx1014-64: 24

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651601&startday=2020-07-06&endday=2020-07-12&tree=all

Comment 8

[Byron Campen [:bwc]]

Not hitting the assertion. Probably ok now.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3e7a4a1681a1
Filter gathered candidates when their component is obsolete. r=mjf

Comment 10

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/3e7a4a1681a1

Comment 11

[Andreea Pavel [:apavel]]

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=309688409&repo=mozilla-central&lineNumber=47185

Comment 12

[Byron Campen [:bwc]]

Hmm, seems to have not done the trick.

Comment 13

[Byron Campen [:bwc]]

Ah, it looks like there are at least two stacks here. One involves gathering a local candidate when the stream is obsolete, the other involves receiving an ICE check on an obsolete stream. The patch on this bug was intended to fix the former.

Comment 14

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=92ec771506865e5f654f2345022e0fdb06ce7b09

Comment 15

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=44fc15200d1cdfe009aa86ab14d700c28a651f02

Comment 16

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=44b662709e63c1ea99c5c97a4e3456122221ce8e

Comment 17

[Byron Campen [:bwc]]

Attachment: Bug 1651601: Only do the minimum processing for STUN messages on an obsolete stream.

Comment 18

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=54f13985e3a0322e8d433263d4b2bf63632affdf

Comment 19

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=cfd682d29608292d7ad49ab92863bea4bb2b80a2

Comment 20

[Byron Campen [:bwc]]

Looks like we also need some work on remote trickle candidates here.

Comment 21

[Byron Campen [:bwc]]

I wonder why all of these things are happening now... maybe bug 1326005 opened up this window of opportunity?

Comment 23

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c8a4c2c159f1c58c5fa38c6048a3018e4f767d8c

Comment 24

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c0273cdd849adb32a8b9f5fb327133b05bf94d01

Comment 25

[Byron Campen [:bwc]]

Try in comment 23 looks really orange, but it is all pre-existing initermittents. I'm going to do a comparison push on the base revision.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=558488fae4039b2803ba36def8be67e7e23338a8

Comment 26

[Intermittent Failures Robot]

39 failures in 4014 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

This is the #27 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 25
• mozilla-beta: 4
• mozilla-central: 4
• try: 4
• mozilla-esr78: 2

Platform breakdown:

• macosx1014-64: 38
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651601&startday=2020-07-13&endday=2020-07-19&tree=all

Comment 28

[Intermittent Failures Robot]

28 failures in 4310 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

This is the #49 most frequent failure this week.

Repository breakdown:

• mozilla-beta: 3
• autoland: 19
• mozilla-esr78: 3
• mozilla-central: 2
• try: 1

Platform breakdown:

• windows10-64: 20
• macosx1014-64: 7
• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651601&startday=2020-07-20&endday=2020-07-26&tree=all

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/014a5eab2a2c
Only do the minimum processing for STUN messages on an obsolete stream. r=mjf

Comment 30

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/014a5eab2a2c

Comment 31

[Atila Butkovits]

This is still happening: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=311370911&repo=autoland&lineNumber=58468

Comment 32

[Byron Campen [:bwc]]

Looks like we have a lost patch from comment 20. Trying to find it...

Comment 33

[Byron Campen [:bwc]]

Attachment: Bug 1651601: Ignore remote trickle candidates for obsolete streams.

Depends on D83545

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/88b60065bd1e
Ignore remote trickle candidates for obsolete streams. r=mjf

Comment 35

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/88b60065bd1e

Comment 36

[Intermittent Failures Robot]

9 failures in 4289 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-esr78: 1
• mozilla-central: 2

Platform breakdown:

• windows10-64: 6
• macosx1014-64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651601&startday=2020-07-27&endday=2020-08-02&tree=all

Comment 37

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:bwc, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Comment 38

[Byron Campen [:bwc]]

Let's wait a bit and see if we've covered all the ways this can happen.

Comment 39

[Byron Campen [:bwc]]

Looks like this is all sorted out now. It may be worth uplifting to esr.

Comment 40

[Byron Campen [:bwc]]

Comment on attachment 9166819 [details]
Bug 1651601: Ignore remote trickle candidates for obsolete streams.

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: This fixes a common intermittent failure.
• User impact if declined:
• Fix Landed on Version: 81
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky):
• String or UUID changes made by this patch: None

Comment 41

[Byron Campen [:bwc]]

Comment on attachment 9166819 [details]
Bug 1651601: Ignore remote trickle candidates for obsolete streams.

Beta/Release Uplift Approval Request

• User impact if declined: See request for esr
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky):
• String changes made/needed: none

Comment 42

[Julien Cristau [:jcristau] (back August 18)]

Comment on attachment 9166819 [details]
Bug 1651601: Ignore remote trickle candidates for obsolete streams.

approved for 80.0b6

Comment 43

[Julien Cristau [:jcristau] (back August 18)]

https://hg.mozilla.org/releases/mozilla-beta/rev/8e25cc50d91f
https://hg.mozilla.org/releases/mozilla-beta/rev/ffe0c94ffb80

Comment 44

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9162858 [details]
Bug 1651601: Filter gathered candidates when their component is obsolete. r?mjf

Approved for 78.2esr.

Comment 45

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr78/rev/763fdc032ae7
https://hg.mozilla.org/releases/mozilla-esr78/rev/25845cb12302
https://hg.mozilla.org/releases/mozilla-esr78/rev/319862e5d83d

Comment 46

[Intermittent Failures Robot]

3 failures in 4618 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1
• mozilla-esr78: 2

Platform breakdown:

• windows10-64-qr: 1
• macosx1014-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651601&startday=2020-08-03&endday=2020-08-09&tree=all

Comment 47

[Intermittent Failures Robot]

2 failures in 3601 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 2

Platform breakdown:

• windows10-64: 1
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1651601&startday=2020-08-10&endday=2020-08-16&tree=all