1871573 - Crash in [@ mozilla::net::CacheStorageService::AddStorageEntry]

Status: RESOLVED INVALID

(Core :: Networking: Cache, defect, P1)

Description

[Randell Jesup [:jesup] (needinfo me)]

Flagged by PHC, access to freed memory. PHC Kind is FreedPage

Free stack:

#0 mozilla::net::EndsInANumber(nsTString<char> const&) (xul.dll)
#1 mozilla::net::nsStandardURL::SetSpecWithEncoding(nsTSubstring<char> const&, mozilla::Encoding const*) (xul.dll)
#2 mozilla::net::nsStandardURL::TemplatedMutator<mozilla::net::nsStandardURL>::Init(unsigned int, int, nsTSubstring<char> const&, char const*, nsIURI*, nsIURIMutator**) (xul.dll)
#3 NewStandardURI(nsTSubstring<char> const&, char const*, nsIURI*, int, nsIURI**) (xul.dll)
#4 NS_NewURI(nsIURI**, nsTSubstring<char> const&, char const*, nsIURI*) (xul.dll)
#5 mozilla::Permission::ClonePrincipalForPermission(nsIPrincipal*) (xul.dll)
#6 mozilla::net::CookieJarSettings::Deserialize(mozilla::net::CookieJarSettingsArgs const&, nsICookieJarSettings**) (xul.dll)
#7 mozilla::ipc::LoadInfoArgsToLoadInfo(mozilla::net::LoadInfoArgs const&, nsTSubstring<char> const&, nsINode*, mozilla::net::LoadInfo**) (xul.dll)
#8 mozilla::net::NeckoParent::RecvPHttpChannelConstructor(mozilla::net::PHttpChannelParent*, mozilla::dom::PBrowserParent*, IPC::SerializedLoadContext const&, mozilla::net::HttpChannelCreationArgs const&) (xul.dll)
#9 mozilla::net::PNeckoParent::OnMessageReceived(IPC::Message const&) (xul.dll)
#10 mozilla::dom::PContentParent::OnMessageReceived(IPC::Message const&) (xul.dll)
#11 mozilla::ipc::MessageChannel::MessageTask::Run() (xul.dll)
#12 mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex &> const&) (xul.dll)
#13 mozilla::detail::RunnableFunction<`lambda at /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:211:7'>::Run() (xul.dll)
#14 NS_ProcessNextEvent(nsIThread*, bool) (xul.dll)
#15 mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) (xul.dll)

Alloc stack:

#0 moz_xrealloc(void*, unsigned long long) (mozglue.dll)
#1 mozilla::net::EndsInANumber(nsTString<char> const&) (xul.dll)
#2 mozilla::net::nsStandardURL::SetSpecWithEncoding(nsTSubstring<char> const&, mozilla::Encoding const*) (xul.dll)
#3 mozilla::net::nsStandardURL::TemplatedMutator<mozilla::net::nsStandardURL>::Init(unsigned int, int, nsTSubstring<char> const&, char const*, nsIURI*, nsIURIMutator**) (xul.dll)
#4 NewStandardURI(nsTSubstring<char> const&, char const*, nsIURI*, int, nsIURI**) (xul.dll)
#5 NS_NewURI(nsIURI**, nsTSubstring<char> const&, char const*, nsIURI*) (xul.dll)
#6 mozilla::Permission::ClonePrincipalForPermission(nsIPrincipal*) (xul.dll)
#7 mozilla::net::CookieJarSettings::Deserialize(mozilla::net::CookieJarSettingsArgs const&, nsICookieJarSettings**) (xul.dll)
#8 mozilla::ipc::LoadInfoArgsToLoadInfo(mozilla::net::LoadInfoArgs const&, nsTSubstring<char> const&, nsINode*, mozilla::net::LoadInfo**) (xul.dll)
#9 mozilla::net::NeckoParent::RecvPHttpChannelConstructor(mozilla::net::PHttpChannelParent*, mozilla::dom::PBrowserParent*, IPC::SerializedLoadContext const&, mozilla::net::HttpChannelCreationArgs const&) (xul.dll)
#10 mozilla::net::PNeckoParent::OnMessageReceived(IPC::Message const&) (xul.dll)
#11 mozilla::dom::PContentParent::OnMessageReceived(IPC::Message const&) (xul.dll)
#12 mozilla::ipc::MessageChannel::MessageTask::Run() (xul.dll)
#13 mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex &> const&) (xul.dll)
#14 mozilla::detail::RunnableFunction<`lambda at /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:211:7'>::Run() (xul.dll)
#15 NS_ProcessNextEvent(nsIThread*, bool) (xul.dll)

Crash report: https://crash-stats.mozilla.org/report/index/c85179af-6aa9-4b1e-874e-c47b90231124

Reason: EXCEPTION_ACCESS_VIOLATION_WRITE

Top 10 frames of crashing thread:

0  xul.dll  nsTHashtable<nsBaseHashtableET<nsCStringHashKey, RefPtr<mozilla::net::CacheEntry> > >::WithEntryHandle  xpcom/ds/nsTHashtable.h:446
0  xul.dll  nsTHashtable<nsBaseHashtableET<nsCStringHashKey, RefPtr<mozilla::net::CacheEntry> > >::PutEntry  xpcom/ds/nsTHashtable.h:317
0  xul.dll  nsRefCountedHashtable<nsCStringHashKey, RefPtr<mozilla::net::CacheEntry> >::InsertOrUpdate  xpcom/ds/nsRefCountedHashtable.h:191
0  xul.dll  nsRefCountedHashtable<nsCStringHashKey, RefPtr<mozilla::net::CacheEntry> >::InsertOrUpdate  xpcom/ds/nsRefCountedHashtable.h:180
0  xul.dll  mozilla::net::CacheStorageService::AddStorageEntry  netwerk/cache2/CacheStorageService.cpp:1602
1  xul.dll  mozilla::net::CacheStorageService::AddStorageEntry  netwerk/cache2/CacheStorageService.cpp:1516
2  xul.dll  mozilla::net::CacheStorage::AsyncOpenURI  netwerk/cache2/CacheStorage.cpp:59
3  xul.dll  mozilla::net::nsHttpChannel::OpenCacheEntryInternal  netwerk/protocol/http/nsHttpChannel.cpp:3786
3  xul.dll  mozilla::net::nsHttpChannel::OpenCacheEntry  netwerk/protocol/http/nsHttpChannel.cpp:3648
3  xul.dll  mozilla::net::nsHttpChannel::ConnectOnTailUnblock  netwerk/protocol/http/nsHttpChannel.cpp:850

Comment 1

[Andrew McCreight [:mccr8]]

It would be nice if we had line numbers there.

I'm not sure how EndsInANumber could cause a realloc. I guess it must be input because what other values there can escape there?

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

So this bug is really interesting. The alloc and free stacks make sense; it looks like this is from EndsInANumber() in the URL code: https://searchfox.org/mozilla-central/source/netwerk/base/nsStandardURL.cpp#763 - the TArray getting expanded and freed.
However, these pointers never make it out of the function, and have no correlation to the crash (other than this was the last code to free that memory).
My base assumption is that something else freed the memory; it was reallocated by EndsInANumber() (under PHC) and then freed, and then the original code accessed the memory.

#0  moz_xrealloc(void*, size_t) (ptr=0x7fffaed12ca0, size=64) at /home/jesup/src/mozilla/inbound_prof/memory/mozalloc/mozalloc.cpp:72
#1  0x00007fffdc09158d in nsTArrayInfallibleAllocator::Realloc(void*, unsigned long) (aPtr=0x7fffaed12ca0, aSize=64)
    at /home/jesup/src/mozilla/inbound_prof/obj-debug/dist/include/nsTArray.h:258
#2  0x00007fffdc091304 in nsTArray_base<nsTArrayInfallibleAllocator, nsTArray_RelocateUsingMemutils>::EnsureCapacityImpl<nsTArrayInfallibleAllocator>(unsigned long, unsigned long) (this=0x7fffffff1a08, aCapacity=2, aElemSize=16) at /home/jesup/src/mozilla/inbound_prof/obj-debug/dist/include/nsTArray-inl.h:221
#3  0x00007fffdc090f92 in nsTArray_base<nsTArrayInfallibleAllocator, nsTArray_RelocateUsingMemutils>::EnsureCapacity<nsTArrayInfallibleAllocator>(unsigned long, unsigned long) (this=0x7fffffff1a08, aCapacity=2, aElemSize=16) at /home/jesup/src/mozilla/inbound_prof/obj-debug/dist/include/nsTArray.h:442
#4  0x00007fffddc474b2 in nsTArray_Impl<nsTDependentSubstring<char>, nsTArrayInfallibleAllocator>::AppendElementInternal<nsTArrayInfallibleAllocator, nsTDependentSubstring<char> const&>(nsTDependentSubstring<char> const&) (this=0x7fffffff1a08, aItem=...)
    at /home/jesup/src/mozilla/inbound_prof/obj-debug/dist/include/nsTArray.h:2699
#5  0x00007fffddc472ed in nsTArray<nsTDependentSubstring<char> >::AppendElement<nsTDependentSubstring<char> const&>(nsTDependentSubstring<char> const&)
    (this=0x7fffffff1a08, aItem=...) at /home/jesup/src/mozilla/inbound_prof/obj-debug/dist/include/nsTArray.h:2843
#6  0x00007fffddc0822e in mozilla::net::EndsInANumber(nsTString<char> const&) (input="mozilla.cloudflare-dns.com")
    at /home/jesup/src/mozilla/inbound_prof/netwerk/base/nsStandardURL.cpp:767

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

Ditto another report that looks like it was reallocated: 4e94aaa0-c84d-498c-87d6-bde930231124

Free stack:

#0    enum2$<neqo_http3::buffered_send_stream::BufferedStream>::send_buffer(neqo_transport::connection::Connection*) (xul.dll)
#1    neqo_http3::connection::Http3Connection::process_sending(neqo_transport::connection::Connection*) (xul.dll)
#2    neqo_http3::connection_client::Http3Client::process_http3(std::time::Instant) (xul.dll)
#3    neqo_http3::connection_client::Http3Client::process_input(neqo_common::datagram::Datagram, std::time::Instant) (xul.dll)
#4    neqo_glue::neqo_http3conn_process_input(neqo_glue::NeqoHttp3Conn*, neqo_glue::NetAddr*, thin_vec::ThinVec<u8>*) (xul.dll)
#5    mozilla::net::Http3Session::ProcessInput(nsIUDPSocket*) (xul.dll)
#6    mozilla::net::Http3Session::RecvData(nsIUDPSocket*) (xul.dll)
#7    mozilla::net::HttpConnectionUDP::RecvData() (xul.dll)
#8    mozilla::net::HttpConnectionUDP::OnPacketReceived(nsIUDPSocket*) (xul.dll)
#9    mozilla::net::nsSocketTransportService::Run() (xul.dll)
#10    NS_ProcessNextEvent(nsIThread*, bool) (xul.dll)
#11    mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) (xul.dll)
#12    MessageLoop::RunHandler() (xul.dll)
#13    nsThread::ThreadFunc(void*) (xul.dll)
#14    _PR_NativeRunThread(void*) (nss3.dll)
#15    pr_root(void*) (nss3.dll)

Alloc stack:

#0    alloc::raw_vec::finish_grow<alloc::alloc::Global>() (xul.dll)
#1    alloc::raw_vec::impl$1::reserve::do_reserve_and_handle<u8,alloc::alloc::Global>() (xul.dll)
#2    core::fmt::impl$0::write_str<alloc::string::String>() (xul.dll)
#3    core::fmt::Formatter::debug_tuple_field1_finish() (xul.dll)
#4    neqo_transport::stream_id::impl$16::fmt(neqo_transport::stream_id::StreamId*, core::fmt::Formatter*) (xul.dll)
#5    core::fmt::builders::DebugTuple::field() (xul.dll)
#6    core::fmt::Formatter::debug_tuple_field1_finish() (xul.dll)
#7    core::option::impl$50::fmt<neqo_transport::stream_id::StreamId>(enum2$<core::option::Option<neqo_transport::stream_id::StreamId> >*, core::fmt::Formatter*) (xul.dll)
#8    core::fmt::write() (xul.dll)
#9    core::fmt::impl$56::fmt<enum2$<neqo_http3::buffered_send_stream::BufferedStream> >(enum2$<neqo_http3::buffered_send_stream::BufferedStream>**, core::fmt::Formatter*) (xul.dll)
#10    core::fmt::write() (xul.dll)
#11    alloc::fmt::format::format_inner() (xul.dll)
#12    enum2$<neqo_http3::buffered_send_stream::BufferedStream>::send_buffer(neqo_transport::connection::Connection*) (xul.dll)
#13    neqo_http3::connection::Http3Connection::process_sending(neqo_transport::connection::Connection*) (xul.dll)
#14    neqo_http3::connection_client::Http3Client::process_http3(std::time::Instant) (xul.dll)
#15    neqo_http3::connection_client::Http3Client::process_input(neqo_common::datagram::Datagram, std::time::Instant) (xul.dll)

Comment 4

[Randell Jesup [:jesup] (needinfo me)]

I'm going close this as a hardware error. All 3 PHC crashes are from the same machine, though not the same crash - 3 crashes within minutes per uptime. However, in all 3 cases the instruction that crashes is MOV qword ptr [r8],rbx. r8 is from LEA r8,[rsp+120] with rsp == bfcf60 or bfcf70. In all cases the final value of r8 is 13FD080 or 13fD090, which is exactly 0x800000 above the correct value. This apparently gets lucky and hits the PHC memory buffer fairly reliably.

There are other crashes with this signature, though almost all with ESR 115 or earlier, except for a single 116 crash that looks randomish.