1101264 - crash in OOM | large | NS_ABORT_OOM(unsigned int) | Pickle::Pickle(Pickle const&)

Status: RESOLVED FIXED

(Core :: IPC, defect)

Description

[guigs [:guigs] PST (please needinfo me!)]

This bug was filed from the Socorro interface and is 
report bp-8bf164a9-5658-496e-923d-1c54d2141118.
=============================================================
stacktrace: 

Frame 	Module 	Signature 	Source
0 	xul.dll 	NS_ABORT_OOM(unsigned int) 	xpcom/base/nsDebugImpl.cpp
1 	xul.dll 	Pickle::Pickle(Pickle const&) 	ipc/chromium/src/base/pickle.cc
2 	xul.dll 	IPC::Message::Message(IPC::Message const&) 	ipc/chromium/src/chrome/common/ipc_message.cc
3 	xul.dll 	std::_Cons_val<std::allocator<IPC::Message>, IPC::Message, IPC::Message const&>(std::allocator<IPC::Message>&, IPC::Message*, IPC::Message const&) 	c:/program files (x86)/microsoft visual studio 10.0/vc/include/xmemory:280
4 	xul.dll 	std::deque<IPC::Message, std::allocator<IPC::Message> >::push_back(IPC::Message const&) 	c:/program files (x86)/microsoft visual studio 10.0/vc/include/deque:1267
5 	xul.dll 	mozilla::ipc::MessageChannel::OnMessageReceivedFromLink(IPC::Message const&) 	ipc/glue/MessageChannel.cpp
6 	xul.dll 	mozilla::ipc::ProcessLink::OnMessageReceived(IPC::Message const&) 	ipc/glue/MessageLink.cpp
7 	xul.dll 	IPC::Channel::ChannelImpl::ProcessIncomingMessages(base::MessagePumpForIO::IOContext*, unsigned long) 	ipc/chromium/src/chrome/common/ipc_channel_win.cc
8 	xul.dll 	IPC::Channel::ChannelImpl::OnIOCompleted(base::MessagePumpForIO::IOContext*, unsigned long, unsigned long) 	ipc/chromium/src/chrome/common/ipc_channel_win.cc
9 	xul.dll 	base::MessagePumpForIO::WaitForIOCompletion(unsigned long, base::MessagePumpForIO::IOHandler*) 	ipc/chromium/src/base/message_pump_win.cc
10 	xul.dll 	base::MessagePumpForIO::DoRunLoop() 	ipc/chromium/src/base/message_pump_win.cc
11 	xul.dll 	base::MessagePumpWin::RunWithDispatcher(base::MessagePump::Delegate*, base::MessagePumpWin::Dispatcher*) 	ipc/chromium/src/base/message_pump_win.cc
12 	xul.dll 	base::MessagePumpWin::Run(base::MessagePump::Delegate*) 	ipc/chromium/src/base/message_pump_win.h
13 	xul.dll 	MessageLoop::RunHandler() 	ipc/chromium/src/base/message_loop.cc
14 	xul.dll 	MessageLoop::Run() 	ipc/chromium/src/base/message_loop.cc
15 	xul.dll 	base::Thread::ThreadMain() 	ipc/chromium/src/base/thread.cc
16 	xul.dll 	`anonymous namespace'::ThreadFunc(void*) 	ipc/chromium/src/base/platform_thread_win.cc
17 	kernel32.dll 	BaseThreadInitThunk 	
18 	ntdll.dll 	__RtlUserThreadStart 	
19 	ntdll.dll 	_RtlUserThreadStart


Adapter Description: NVIDIA GeForce GTX 680 Adapter Drivers: nvd3dumx,nvwgf2umx,nvwgf2umx nvd3dum,nvwgf2um,nvwgf2um Adapter RAM: 2048 Device ID: 0x1180 DirectWrite Enabled: false (6.2.9200.16492) Driver Date: 11-3-2014 Driver Version: 9.18.13.4465 GPU #2 Active: false GPU Accelerated Windows: 0/1 Basic (OMTC) Vendor ID: 0x10de WebGL Renderer: Google Inc. -- ANGLE (NVIDIA GeForce GTX 680 Direct3D9Ex vs_3_0 ps_3_0) windowLayerManagerRemote: true AzureCanvasBackend: skia AzureContentBackend: cairo AzureFallbackCanvasBackend: cairo AzureSkiaAccelerated: 0

Comment 1

[Andrew McCreight [:mccr8]]

There's some discussion about this in bug 1092010.

Two theories that have been presented for why we are seeing these OOM crashes in non-e10s browsers is the compositor thread or the thumbnail thread may be sending large messages.

Comment 2

[(Away)]

The larger problem is that this user's Firefox pretty consistently consumes >3gb of memory within the first 30 seconds of startup. These IPC messages may just be the first victims due to their size.

Comment 3

[Nicholas Nethercote [inactive]]

Changing

Comment 4

[(Away)]

Long story short it was adware/malware. When opening about:newtab, an instance of plugin-container shows up, allocates several gigs of memory within a second, and disappears. But the Pickles stay allocated on the FF side.

Anybody want me to pull any more information before I pave the infected machine?

Comment 5

[(Away)]

Well I did find some interesting things nonetheless.

Going to about:newtab causes an instance of plugin-container to appear (not exactly sure why, I vaguely recall the stacks saying RemoteBrowser or some such). That plugin-container encounters a bunch of CSS errors. It's basically the same problem as bug 976777 except the troublesome variable is not |sourceLine| but rather |sourceName| which is a massive data: url from an addon.

The other thing is that these IPC messages accumulate on the firefox.exe side, but I don't see them ever get deallocated. Haven't yet debugged this super closely.

Comment 6

[Benjamin Smedberg]

That plugin-container is a content process charged with generating thumbnails for about:newtab (this is in all builds, not just e10s builds).

Does that mean this is solved with bug 976777 or is there still work to be done?

Comment 7

[(Away)]

There is still (easy) work to be done. Bug 976777 fixed the problem with |sourceLine|; we just need to do the same with |sourceName|.

Comment 8

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Well, we should also try to harden the parent process against this by not explicitly aborting... I would much rather it trigger the death of the child process.

Comment 9

[(Away)]

Attachment: Truncate long sourceName

This seems like a straightforward fix that we can just go ahead and make.

bent, if you have ideas for attacking the issue from another angle, then I totally support that too, either as a leave-open or a spinoff bug or what have you.

Comment 10

[(Away)]

Oh no! This fell off my radar and I never landed it.

It's unclear whether that adware is still going around, but it seems like we're still susceptible to badness here, so I'll push...

Comment 11

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/4c0e0c5176e9

Comment 12

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/4c0e0c5176e9