1293457 - Intermittent WebRtcIceGatherTest.TestStunTcpServerTrickle | Value of: res

Status: RESOLVED INCOMPLETE

(Core :: WebRTC: Networking, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: philringnalda [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=1593025&repo=autoland

https://queue.taskcluster.net/v1/task/A1JLUkmcR4-XPFMYN749Ow/runs/0/artifacts/public%2Flogs%2Flive_backing.log

Comment 1

[Nils Ohlmeier [:drno]]

Neat: a SEGV on shutdown, because we try to fire a gathering callback on shutdown

20:24:20     INFO -  ASAN:DEADLYSIGNAL
20:24:20     INFO -  =================================================================
20:24:20     INFO -  ==1138==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x7f970fc51808 bp 0x7f9705922130 sp 0x7f9705921fe0 T6)
20:24:21     INFO -      #0 0x7f970fc51807 in RefPtr /home/worker/workspace/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:91:25
20:24:21     INFO -      #1 0x7f970fc51807 in ctx /home/worker/workspace/build/src/media/mtransport/nricectxhandler.h:29
20:24:21     INFO -      #2 0x7f970fc51807 in (anonymous namespace)::IceTestPeer::GatheringStateChange(mozilla::NrIceCtx*, mozilla::NrIceCtx::GatheringState) /home/worker/workspace/build/src/media/mtransport/test/ice_unittest.cpp:1016
20:24:21     INFO -      #3 0x7f9711cfce4b in operator() /home/worker/workspace/build/src/media/mtransport/sigslot.h:2420:5
20:24:21     INFO -      #4 0x7f9711cfce4b in mozilla::NrIceCtx::SetGatheringState(mozilla::NrIceCtx::GatheringState) /home/worker/workspace/build/src/media/mtransport/nricectx.cpp:989
20:24:21     INFO -      #5 0x7f9717cfe12c in nr_ice_gather_finished_cb /home/worker/workspace/build/src/media/mtransport/third_party/nICEr/src/ice/ice_ctx.c:550:9
20:24:21     INFO -      #6 0x7f9717ce84b9 in nr_ice_candidate_destroy /home/worker/workspace/build/src/media/mtransport/third_party/nICEr/src/ice/ice_candidate.c:316:7
20:24:21     INFO -      #7 0x7f9717cf0a95 in nr_ice_component_destroy /home/worker/workspace/build/src/media/mtransport/third_party/nICEr/src/ice/ice_component.c:162:7
20:24:21     INFO -      #8 0x7f9717d000e7 in nr_ice_media_stream_destroy /home/worker/workspace/build/src/media/mtransport/third_party/nICEr/src/ice/ice_media_stream.c:102:7
20:24:21     INFO -      #9 0x7f9717cff0bf in nr_ice_remove_media_stream /home/worker/workspace/build/src/media/mtransport/third_party/nICEr/src/ice/ice_ctx.c:796:10
20:24:21     INFO -      #10 0x7f9711d083ef in mozilla::NrIceMediaStream::Close() /home/worker/workspace/build/src/media/mtransport/nricemediastream.cpp:602:13
20:24:21     INFO -      #11 0x7f9711cf954a in mozilla::NrIceCtx::~NrIceCtx() /home/worker/workspace/build/src/media/mtransport/nricectx.cpp:676:7
20:24:21     INFO -      #12 0x7f9711cf9bed in mozilla::NrIceCtx::~NrIceCtx() /home/worker/workspace/build/src/media/mtransport/nricectx.cpp:672:23
20:24:21     INFO -      #13 0x7f970fc54a74 in Release /home/worker/workspace/build/src/media/mtransport/nricectx.h:333:3
20:24:21     INFO -      #14 0x7f970fc54a74 in Release /home/worker/workspace/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:40
20:24:21     INFO -      #15 0x7f970fc54a74 in Release /home/worker/workspace/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:387
20:24:21     INFO -      #16 0x7f970fc54a74 in ~RefPtr /home/worker/workspace/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:78
20:24:21     INFO -      #17 0x7f970fc54a74 in ~NrIceCtxHandler /home/worker/workspace/build/src/media/mtransport/nricectxhandler.h:44

Comment 2

[Nils Ohlmeier [:drno]]

Looks like we could set the |done_cb| in nr_ice_candidate_destroy(). That should avoid the SEGV. But I'm not sure if that would not have other side effects.

The actual test failure looks like a simple gathering timeout. Probably around the TCP server reflexive gathering. But no idea why.

Comment 3

[Nils Ohlmeier [:drno]]

Attachment: Bug 1293457: stop printing gathering results on shutdown.

Review commit: https://reviewboard.mozilla.org/r/73216/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/73216/

Comment 4

[Nils Ohlmeier [:drno]]

(In reply to Nils Ohlmeier [:drno] from comment #2)
> The actual test failure looks like a simple gathering timeout. Probably
> around the TCP server reflexive gathering. But no idea why.

Apparently the reason is that Twillio's STUN server no longer responds to TCP connections.

Comment 5

[Michael Froman [:mjf]]

Comment on attachment 8783419 [details]
Bug 1293457: stop printing gathering results on shutdown.

https://reviewboard.mozilla.org/r/73216/#review71114

::: media/mtransport/test/ice_unittest.cpp:1009
(Diff revision 1)
>    }
>  
>    // Handle events
>    void GatheringStateChange(NrIceCtx* ctx,
>                              NrIceCtx::GatheringState state) {
> -    (void)ctx;
> +    if (shutting_down_) {

Why remote this line?

Comment 6

[Michael Froman [:mjf]]

Comment on attachment 8783419 [details]
Bug 1293457: stop printing gathering results on shutdown.

https://reviewboard.mozilla.org/r/73216/#review71118

One small question, otherwise it looks good to me.

Comment 7

[Nils Ohlmeier [:drno]]

Comment on attachment 8783419 [details]
Bug 1293457: stop printing gathering results on shutdown.

https://reviewboard.mozilla.org/r/73216/#review71114

> Why remote this line?

Because it compiles without it :-)
Looks like a left over from old times to me. So I thought I would clean it up...

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/389bcc7eb849
stop printing gathering results on shutdown. r=mjf

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/389bcc7eb849

Comment 10

[Ryan VanderMeulen [:RyanVM]]

This is still happening.
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293457

Comment 11

[Nils Ohlmeier [:drno]]

Okay, so the first patch only fixed the seg fault on shutdown, which was triggered by the actual intermittent: for some unknown reason gathering sometimes does not finish with the fake STUN server an TCP.

Comment 12

[Intermittent Failures Robot]

1 failures in 814 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* pine: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293457&startday=2017-06-12&endday=2017-06-18&tree=all

Comment 13

[Intermittent Failures Robot]

1 failures in 892 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293457&startday=2017-06-19&endday=2017-06-25&tree=all

Comment 14

[Intermittent Failures Robot]

2 failures in 718 pushes (0.003 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* try: 1
* mozilla-inbound: 1

Platform breakdown:
* linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293457&startday=2017-06-26&endday=2017-07-02&tree=all

Comment 15

[Intermittent Failures Robot]

1 failures in 720 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293457&startday=2017-07-10&endday=2017-07-16&tree=all

Comment 16

[Intermittent Failures Robot]

1 failures in 822 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293457&startday=2017-07-17&endday=2017-07-23&tree=all

Comment 17

[Intermittent Failures Robot]

2 failures in 888 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* try: 1
* mozilla-central: 1

Platform breakdown:
* linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293457&startday=2017-07-31&endday=2017-08-06&tree=all

Comment 18

[Intermittent Failures Robot]

1 failures in 901 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293457&startday=2017-08-07&endday=2017-08-13&tree=all

Comment 19

[Intermittent Failures Robot]

2 failures in 924 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 2

Platform breakdown:
* linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293457&startday=2017-09-04&endday=2017-09-10&tree=all

Comment 20

[Bulk Bug Changes for mreavy's org]

Mass change P2->P3 to align with new Mozilla triage process.

Comment 21

[Firefox Bug Husbandry Bot]

Bulk priority update of open intermittent test failure bugs. 

P3 => P5

https://bugzilla.mozilla.org/show_bug.cgi?id=1381960

Comment 22

[Intermittent Failures Robot]

1 failures in 1032 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293457&startday=2017-09-11&endday=2017-09-17&tree=all

Comment 23

[Intermittent Failures Robot]

1 failures in 943 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293457&startday=2017-09-18&endday=2017-09-24&tree=all

Comment 24

[Intermittent Failures Robot]

1 failures in 947 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293457&startday=2017-10-09&endday=2017-10-15&tree=all

Comment 25

[Intermittent Failures Robot]

2 failures in 864 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 2

Platform breakdown:
* linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293457&startday=2017-10-16&endday=2017-10-22&tree=all

Comment 26

[Intermittent Failures Robot]

3 failures in 762 pushes (0.004 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1
* mozilla-beta: 1
* autoland: 1

Platform breakdown:
* linux64-nightly: 1
* linux64: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1293457&startday=2017-11-13&endday=2017-11-19&tree=all

Comment 27

[Firefox Bug Husbandry Bot]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup