Open Bug 1653618 Opened 4 months ago Updated 1 month ago

ThreadSanitizer: data race in libusrsctp [@ sctp_handle_tick] vs [@ sctp_handle_sack]

Categories

(Core :: WebRTC: Networking, defect)

defect

Tracking

()

ASSIGNED

People

(Reporter: bwc, Assigned: bwc, NeedInfo)

References

(Blocks 1 open bug)

Details

(Keywords: leave-open)

Attachments

(1 file)

Looks like racy accesses on a flag field. https://searchfox.org/mozilla-central/rev/d6d8fcc22c3820f2ae08229e0d37be19fba74db9/netwerk/sctp/src/netinet/sctp_callout.c#170 vs https://searchfox.org/mozilla-central/rev/d6d8fcc22c3820f2ae08229e0d37be19fba74db9/netwerk/sctp/src/netinet/sctp_indata.c#5221

This might be risky? Looks like two threads might be trying to start a timer simultaneously, which seems bad. Marking security until further analysis can be done.

 1:55.43 GECKO(857032) ==================
 1:55.43 GECKO(857032) WARNING: ThreadSanitizer: data race (pid=857159)
 1:55.43 GECKO(857032)   Write of size 4 at 0x7b54000215d8 by thread T35 (mutexes: write M35861):
 1:55.43 GECKO(857032)     #0 sctp_handle_tick /home/bcampen/checkouts/mozilla-central/netwerk/sctp/src/netinet/sctp_callout.c:170:15 (libxul.so+0x6847515)
 1:55.43 GECKO(857032)     #1 user_sctp_timer_iterate /home/bcampen/checkouts/mozilla-central/netwerk/sctp/src/netinet/sctp_callout.c:214:3 (libxul.so+0x6847652)
 1:55.43 GECKO(857032)   Previous read of size 4 at 0x7b54000215d8 by thread T5 (mutexes: write M35847, write M137777001758742048):
 1:55.43 GECKO(857032)     #0 sctp_handle_sack /home/bcampen/checkouts/mozilla-central/netwerk/sctp/src/netinet/sctp_indata.c:5221:9 (libxul.so+0x68568b4)
 1:55.43 GECKO(857032)     #1 sctp_process_control /home/bcampen/checkouts/mozilla-central/netwerk/sctp/src/netinet/sctp_input.c:5213:6 (libxul.so+0x6864733)
 1:55.43 GECKO(857032)     #2 sctp_common_input_processing /home/bcampen/checkouts/mozilla-central/netwerk/sctp/src/netinet/sctp_input.c:5913:10 (libxul.so+0x6861473)
 1:55.43 GECKO(857032)     #3 usrsctp_conninput /home/bcampen/checkouts/mozilla-central/netwerk/sctp/src/user_socket.c:3340:2 (libxul.so+0x68d65dd)
 1:55.43 GECKO(857032)     #4 mozilla::DataChannelConnection::SctpDtlsInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mozilla::MediaPacket const&) /home/bcampen/checkouts/mozilla-central/netwerk/sctp/datachannel/DataChannel.cpp:949:3 (libxul.so+0x68de6f0)
 1:55.43 GECKO(857032)     #5 void sigslot::_opaque_connection::emitter<mozilla::DataChannelConnection, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mozilla::MediaPacket const&>(sigslot::_opaque_connection const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mozilla::MediaPacket const&) /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/mtransport/sigslot.h:339:5 (libxul.so+0x68f1303)
 1:55.43 GECKO(857032)     #6 emit<const std::__cxx11::basic_string<char> &, const mozilla::MediaPacket &> /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/mtransport/sigslot.h:330:5 (libxul.so+0x7258928)
 1:55.43 GECKO(857032)     #7 emit /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/mtransport/sigslot.h:562:12 (libxul.so+0x7258928)
 1:55.43 GECKO(857032)     #8 operator() /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/mtransport/sigslot.h:566:35 (libxul.so+0x7258928)
 1:55.43 GECKO(857032)     #9 mozilla::MediaTransportHandler::OnPacketReceived(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mozilla::MediaPacket const&) /home/bcampen/checkouts/mozilla-central/media/webrtc/signaling/src/peerconnection/MediaTransportHandler.cpp:943:3 (libxul.so+0x7258928)
 1:55.43 GECKO(857032)     #10 mozilla::MediaTransportChild::RecvOnPacketReceived(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mozilla::MediaPacket const&) /home/bcampen/checkouts/mozilla-central/media/webrtc/signaling/src/peerconnection/MediaTransportHandlerIPC.cpp:378:10 (libxul.so+0x72640d4)
 1:55.43 GECKO(857032)     #11 mozilla::dom::PMediaTransportChild::OnMessageReceived(IPC::Message const&) /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/ipc/ipdl/PMediaTransportChild.cpp:942:63 (libxul.so+0x6cf150c)
 1:55.43 GECKO(857032)     #12 mozilla::ipc::PBackgroundChild::OnMessageReceived(IPC::Message const&) /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/ipc/ipdl/PBackgroundChild.cpp:6080:32 (libxul.so+0x6cc7255)
 1:55.43 GECKO(857032)     #13 mozilla::ipc::MessageChannel::DispatchAsyncMessage(mozilla::ipc::ActorLifecycleProxy*, IPC::Message const&) /home/bcampen/checkouts/mozilla-central/ipc/glue/MessageChannel.cpp:2150:25 (libxul.so+0x69d1352)
 1:55.43 GECKO(857032)     #14 mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message&&) /home/bcampen/checkouts/mozilla-central/ipc/glue/MessageChannel.cpp:2074:9 (libxul.so+0x69cf46e)
 1:55.43 GECKO(857032)     #15 mozilla::ipc::MessageChannel::RunMessage(mozilla::ipc::MessageChannel::MessageTask&) /home/bcampen/checkouts/mozilla-central/ipc/glue/MessageChannel.cpp:1922:3 (libxul.so+0x69d0160)
 1:55.43 GECKO(857032)     #16 mozilla::ipc::MessageChannel::MessageTask::Run() /home/bcampen/checkouts/mozilla-central/ipc/glue/MessageChannel.cpp:1953:13 (libxul.so+0x69d08c9)
 1:55.43 GECKO(857032)     #17 nsThread::ProcessNextEvent(bool, bool*) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThread.cpp:1234:14 (libxul.so+0x5ef7d00)
 1:55.43 GECKO(857032)     #18 NS_ProcessNextEvent(nsIThread*, bool) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThreadUtils.cpp:513:10 (libxul.so+0x5efc8b5)
 1:55.43 GECKO(857032)     #19 mozilla::net::nsSocketTransportService::Run() /home/bcampen/checkouts/mozilla-central/netwerk/base/nsSocketTransportService2.cpp:1195:11 (libxul.so+0x609aefb)
 1:55.43 GECKO(857032)     #20 non-virtual thunk to mozilla::net::nsSocketTransportService::Run() /home/bcampen/checkouts/mozilla-central/netwerk/base/nsSocketTransportService2.cpp (libxul.so+0x609c38d)
 1:55.43 GECKO(857032)     #21 nsThread::ProcessNextEvent(bool, bool*) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThread.cpp:1234:14 (libxul.so+0x5ef7d00)
 1:55.43 GECKO(857032)     #22 NS_ProcessNextEvent(nsIThread*, bool) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThreadUtils.cpp:513:10 (libxul.so+0x5efc8b5)
 1:55.43 GECKO(857032)     #23 mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /home/bcampen/checkouts/mozilla-central/ipc/glue/MessagePump.cpp:302:20 (libxul.so+0x69d61be)
 1:55.43 GECKO(857032)     #24 RunInternal /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:334:10 (libxul.so+0x6909b2c)
 1:55.43 GECKO(857032)     #25 RunHandler /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:327:3 (libxul.so+0x6909b2c)
 1:55.43 GECKO(857032)     #26 MessageLoop::Run() /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:309:3 (libxul.so+0x6909b2c)
 1:55.43 GECKO(857032)     #27 nsThread::ThreadFunc(void*) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThread.cpp:447:10 (libxul.so+0x5ef3948)
 1:55.43 GECKO(857032)     #28 _pt_root /home/bcampen/checkouts/mozilla-central/nsprpub/pr/src/pthreads/ptthread.c:201:5 (libnspr4.so+0x50be0)
 1:55.43 GECKO(857032)   As if synchronized via sleep:
 1:55.43 GECKO(857032)     #0 nanosleep /builds/worker/fetches/llvm-project/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:361:3 (firefox+0x9e1ad)
 1:55.43 GECKO(857032)     #1 user_sctp_timer_iterate /home/bcampen/checkouts/mozilla-central/netwerk/sctp/src/netinet/sctp_callout.c:209:12 (libxul.so+0x684761a)
 1:55.43 GECKO(857032)   Location is heap block of size 624 at 0x7b5400021480 allocated by thread T5:
 1:55.43 GECKO(857032)     #0 malloc /builds/worker/fetches/llvm-project/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:650:5 (firefox+0x9f244)
 1:55.43 GECKO(857032)     #1 sctp_add_remote_addr /home/bcampen/checkouts/mozilla-central/netwerk/sctp/src/netinet/sctp_pcb.c:4442:8 (libxul.so+0x689bfd4)
 1:55.43 GECKO(857032)     #2 sctp_aloc_assoc /home/bcampen/checkouts/mozilla-central/netwerk/sctp/src/netinet/sctp_pcb.c:5079:13 (libxul.so+0x689cdcd)
 1:55.43 GECKO(857032)     #3 sctpconn_connect /home/bcampen/checkouts/mozilla-central/netwerk/sctp/src/netinet/sctp_usrreq.c:8202:9 (libxul.so+0x68bb3cc)
 1:55.43 GECKO(857032)     #4 soconnect /home/bcampen/checkouts/mozilla-central/netwerk/sctp/src/user_socket.c:1901:12 (libxul.so+0x68d40d5)
 1:55.43 GECKO(857032)     #5 user_connect /home/bcampen/checkouts/mozilla-central/netwerk/sctp/src/user_socket.c:1927:10 (libxul.so+0x68d40d5)
 1:55.44 GECKO(857032)     #6 usrsctp_connect /home/bcampen/checkouts/mozilla-central/netwerk/sctp/src/user_socket.c:1984:10 (libxul.so+0x68d425b)
 1:55.44 GECKO(857032)     #7 mozilla::DataChannelConnection::CompleteConnect() /home/bcampen/checkouts/mozilla-central/netwerk/sctp/datachannel/DataChannel.cpp:852:9 (libxul.so+0x68ded59)
 1:55.44 GECKO(857032)     #8 mozilla::DataChannelConnection::TransportStateChange(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mozilla::TransportLayer::State) /home/bcampen/checkouts/mozilla-central/netwerk/sctp/datachannel/DataChannel.cpp:815:7 (libxul.so+0x68de90b)
 1:55.44 GECKO(857032)     #9 void sigslot::_opaque_connection::emitter<mozilla::DataChannelConnection, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mozilla::TransportLayer::State>(sigslot::_opaque_connection const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mozilla::TransportLayer::State) /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/mtransport/sigslot.h:339:5 (libxul.so+0x68f13a3)
 1:55.44 GECKO(857032)     #10 emit<const std::__cxx11::basic_string<char> &, mozilla::TransportLayer::State> /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/mtransport/sigslot.h:330:5 (libxul.so+0x7258e33)
 1:55.44 GECKO(857032)     #11 emit /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/mtransport/sigslot.h:562:12 (libxul.so+0x7258e33)
 1:55.44 GECKO(857032)     #12 operator() /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/mtransport/sigslot.h:566:35 (libxul.so+0x7258e33)
 1:55.44 GECKO(857032)     #13 mozilla::MediaTransportHandler::OnStateChange(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mozilla::TransportLayer::State) /home/bcampen/checkouts/mozilla-central/media/webrtc/signaling/src/peerconnection/MediaTransportHandler.cpp:976:3 (libxul.so+0x7258e33)
 1:55.44 GECKO(857032)     #14 mozilla::MediaTransportChild::RecvOnStateChange(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int const&) /home/bcampen/checkouts/mozilla-central/media/webrtc/signaling/src/peerconnection/MediaTransportHandlerIPC.cpp:391:10 (libxul.so+0x726421e)
 1:55.44 GECKO(857032)     #15 mozilla::dom::PMediaTransportChild::OnMessageReceived(IPC::Message const&) /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/ipc/ipdl/PMediaTransportChild.cpp:1028:63 (libxul.so+0x6cf1605)
 1:55.44 GECKO(857032)     #16 mozilla::ipc::PBackgroundChild::OnMessageReceived(IPC::Message const&) /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/ipc/ipdl/PBackgroundChild.cpp:6080:32 (libxul.so+0x6cc7255)
 1:55.44 GECKO(857032)     #17 mozilla::ipc::MessageChannel::DispatchAsyncMessage(mozilla::ipc::ActorLifecycleProxy*, IPC::Message const&) /home/bcampen/checkouts/mozilla-central/ipc/glue/MessageChannel.cpp:2150:25 (libxul.so+0x69d1352)
 1:55.44 GECKO(857032)     #18 mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message&&) /home/bcampen/checkouts/mozilla-central/ipc/glue/MessageChannel.cpp:2074:9 (libxul.so+0x69cf46e)
 1:55.44 GECKO(857032)     #19 mozilla::ipc::MessageChannel::RunMessage(mozilla::ipc::MessageChannel::MessageTask&) /home/bcampen/checkouts/mozilla-central/ipc/glue/MessageChannel.cpp:1922:3 (libxul.so+0x69d0160)
 1:55.44 GECKO(857032)     #20 mozilla::ipc::MessageChannel::MessageTask::Run() /home/bcampen/checkouts/mozilla-central/ipc/glue/MessageChannel.cpp:1953:13 (libxul.so+0x69d08c9)
 1:55.44 GECKO(857032)     #21 nsThread::ProcessNextEvent(bool, bool*) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThread.cpp:1234:14 (libxul.so+0x5ef7d00)
 1:55.44 GECKO(857032)     #22 NS_ProcessNextEvent(nsIThread*, bool) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThreadUtils.cpp:513:10 (libxul.so+0x5efc8b5)
 1:55.44 GECKO(857032)     #23 mozilla::net::nsSocketTransportService::Run() /home/bcampen/checkouts/mozilla-central/netwerk/base/nsSocketTransportService2.cpp:1195:11 (libxul.so+0x609aefb)
 1:55.44 GECKO(857032)     #24 non-virtual thunk to mozilla::net::nsSocketTransportService::Run() /home/bcampen/checkouts/mozilla-central/netwerk/base/nsSocketTransportService2.cpp (libxul.so+0x609c38d)
 1:55.44 GECKO(857032)     #25 nsThread::ProcessNextEvent(bool, bool*) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThread.cpp:1234:14 (libxul.so+0x5ef7d00)
 1:55.44 GECKO(857032)     #26 NS_ProcessNextEvent(nsIThread*, bool) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThreadUtils.cpp:513:10 (libxul.so+0x5efc8b5)
 1:55.44 GECKO(857032)     #27 mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /home/bcampen/checkouts/mozilla-central/ipc/glue/MessagePump.cpp:302:20 (libxul.so+0x69d61be)
 1:55.44 GECKO(857032)     #28 RunInternal /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:334:10 (libxul.so+0x6909b2c)
 1:55.44 GECKO(857032)     #29 RunHandler /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:327:3 (libxul.so+0x6909b2c)
 1:55.44 GECKO(857032)     #30 MessageLoop::Run() /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:309:3 (libxul.so+0x6909b2c)
 1:55.44 GECKO(857032)     #31 nsThread::ThreadFunc(void*) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThread.cpp:447:10 (libxul.so+0x5ef3948)
 1:55.44 GECKO(857032)     #32 _pt_root /home/bcampen/checkouts/mozilla-central/nsprpub/pr/src/pthreads/ptthread.c:201:5 (libnspr4.so+0x50be0)

Can you weigh in on this?

Flags: needinfo?(tuexen)
Assignee: nobody → docfaraday
Status: NEW → ASSIGNED

Marking leave-open until we can get a fix from upstream.

Keywords: leave-open
Group: core-security → media-core-security

Ok, looking closer at this, I do not think this is a sec bug. Dan, could you unmark this?

It does look like it could prevent a timer from being rescheduled when it should be. It is possible that we will recover soon after, but I don't know this code very well. Lennart/Michael?

Flags: needinfo?(lennart.grahl)
Flags: needinfo?(dveditz)
Group: media-core-security
Flags: needinfo?(dveditz)
Pushed by bcampen@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/0dc16ddee5d7
Suppression for racy flags field access in libusrsctp timer code. r=decoder

I don't know about this one. Hoping Michael can help you there. That being said, when it comes to usrsctp issues, I suggest to go also post it to their issue tracker (at least when it's not security relevant). :)

Flags: needinfo?(lennart.grahl)
Blocks: tsan
You need to log in before you can comment on or make changes to this bug.