Closed Bug 613472 Opened 14 years ago Closed 11 years ago

OOM Crash while reading files (about 2GB) file with the DOM File API - should use the fallible allocator and propagate errors

Categories

(Core :: DOM: Core & HTML, defect)

Product:
Core
Component:
DOM: Core & HTML
Platform:
x86
Windows 7
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla30

People

(Reporter: atiware, Assigned: lpy)

References

(Blocks 1 open bug,
URL
)

Details

(Keywords: crash, Whiteboard: [mentor=benjamin@smedbergs.us][lang=c++][crashkill:P2])

Crash Data

Attachments

(1 file, 1 obsolete file)

bug613472-V2.patch
3.71 KB, patch
benjamin
: review+
Details | Diff | Splinter Review 
User-Agent:       Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0b7) Gecko/20100101 Firefox/4.0b7
Build Identifier: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0b7) Gecko/20100101 Firefox/4.0b7

Crashing not only with the imageUploader site (file.getAsDataURL)
it crashes every time if the file huge enough is in function file.getAsBinary too


Reproducible: Always

Steps to Reproduce:
1.create a 2GB fake image file: dd if=/dev/zero of=2GB-zeros.png bs=4096 count=524288
2.open http://demos.hacks.mozilla.org/openweb/imageUploader/
3.drop the 2GB file into the drop zone
4.wait about 1-2 minutes (while FF loading file into memory)
Actual Results:  
Crash in nsAString_internal::SetCapacity

Expected Results:  
load the file or trigger some exception

Crash ID: bp-2abf2801-8cf4-44d6-811e-9e4412101119
Keywords: crash, crashreportid
Version: unspecified → Trunk
Signature	nsAString_internal::SetCapacity
UUID	2abf2801-8cf4-44d6-811e-9e4412101119
Time	2010-11-19 05:01:48.452125
Uptime	251
Last Crash	274 seconds (4.6 minutes) before submission
Install Age	693628 seconds (1.1 weeks) since version was first installed.
Product	Firefox
Version	4.0b7
Build ID	20101104131842
Branch	2.0
OS	Mac OS X
OS Version	10.6.4 10F569
CPU	amd64
CPU Info	family 6 model 37 stepping 5
Crash Reason	EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
Crash Address	0xffffffffa0ceea3c
User Comments	drop a 2 GB file
App Notes	Renderers: 0x22600,0x24300,0x20400
Processor Notes	
EMCheckCompatibility	False
Bugzilla - Report this Crash

Crashing Thread

Frame	Module	Signature [Expand]	Source
0	XUL	nsAString_internal::SetCapacity	xpcom/string/src/nsTSubstring.cpp:572
1	XUL	nsAString_internal::EnsureMutable	xpcom/string/src/nsTSubstring.cpp:581
2	XUL	AppendASCIItoUTF16	 nsTSubstring.h:501
3	XUL	nsDOMFileReader::GetAsDataURL	content/base/src/nsDOMFileReader.cpp:644
4	XUL	nsDOMFileReader::OnStopRequest	content/base/src/nsDOMFileReader.cpp:463
5	XUL	nsBaseChannel::OnStopRequest	netwerk/base/src/nsBaseChannel.cpp:727
6	XUL	nsInputStreamPump::OnStateStop	netwerk/base/src/nsInputStreamPump.cpp:578
7	XUL	nsInputStreamPump::OnInputStreamReady	netwerk/base/src/nsInputStreamPump.cpp:403
8	XUL	nsInputStreamReadyEvent::Run	xpcom/io/nsStreamUtils.cpp:112
9	XUL	nsThread::ProcessNextEvent	xpcom/threads/nsThread.cpp:609
10	XUL	NS_ProcessPendingEvents_P	 nsThreadUtils.cpp:200
11	XUL	nsBaseAppShell::NativeEventCallback	widget/src/xpwidgets/nsBaseAppShell.cpp:131
12	XUL	nsAppShell::ProcessGeckoEvents	widget/src/cocoa/nsAppShell.mm:399
13	CoreFoundation	CoreFoundation@0x4de90	
14	CoreFoundation	CoreFoundation@0x4c088	
Show/hide other threads
Component: General → DOM
Keywords: crashreportid
Product: Firefox → Core
QA Contact: general → general
Summary: Crash while reading a huge (about 2GB) file with the file api → Crash while reading a huge (about 2GB) file with the file api [@ nsAString_internal::SetCapacity]
how much memory / swap / vm do you have on your system? (there might be relevant output visible in Console.app for firefox)

because DOM is utf-16 which means that your 2gb file costs at least 4gb to manage, not counting any extra overhead involved in encoding it as a data:url (which should probably make it much larger) and not counting the other original encodings of the data.
Summary: Crash while reading a huge (about 2GB) file with the file api [@ nsAString_internal::SetCapacity] → Crash while reading a huge (about 2GB) file with the DOM File api [@ nsAString_internal::SetCapacity]
my system:
MacBookPro6,2
Intel Core i7, 2.66 GHz, 4GB Ram

terminal output after starting FF4.0b11 only one line:
FTS0: INITIALIZATION_TIME: 516ms

crashed after 20 minutes at 13:38 while using all memory and about 8-9-10 GB Swap
the VM was about 188 GB (but FF was not the only running app)
(the VM size without FF is 178 GB)

Crash id: bp-1534a3cd-d167-49a5-94d5-d0de72110222

in kernel.log:
Feb 22 13:12:00 atest kernel[0]: (default pager): [KERNEL]: Switching ON Emergency paging segment
Feb 22 13:12:30 atest kernel[0]: (default pager): [KERNEL]: Recovered emergency paging segment
Feb 22 13:33:05 atest kernel[0]: (default pager): [KERNEL]: Switching ON Emergency paging segment
Feb 22 13:33:17 atest kernel[0]: (default pager): [KERNEL]: System is out of paging space.
Feb 22 13:33:52 atest kernel[0]: (default pager): [KERNEL]: Recovered emergency paging segment

output of top 10 minutes before crash:

Processes: 76 total, 4 running, 1 stuck, 71 sleeping, 329 threads  13:28:46
Load Avg: 1.46, 1.68, 1.58  CPU usage: 4.79% user, 10.7% sys, 85.13% idle  SharedLibs: 1800K resident, 1144K data, 0B linkedit.
MemRegions: 11964 total, 2906M resident, 6516K private, 406M shared.
PhysMem: 600M wired, 2237M active, 1119M inactive, 3957M used, 8444K free.
VM: 188G vsize, 1042M framework vsize, 2038483(1710) pageins, 712734(792) pageouts. Swap: 6531M + 1661M free.
Purgeable: 7952K 41581(4) pages purged. Networks: packets: 198428/45M in, 54531/29M out. Disks: 267024/15G read, 167909/16G written

PID    COMMAND      %CPU TIME     #TH  #WQ  #POR #MREG RPRVT  RSHRD  RSIZE  VPRVT  VSIZE  PGRP  PPID  STATE    UID  FAULTS   COW
61596  firefox-bin  19.0 03:39.70 22   1    185  868   2857M+ 41M    2658M+ 5189M  10G    61596 282   sleeping 502  5580532+ 1374
Signature	nsReadFromRawBuffer
UUID	1534a3cd-d167-49a5-94d5-d0de72110222
Time	2011-02-22 04:38:29.207283
Uptime	1608
Last Crash	4072394 seconds (6.7 weeks) before submission
Install Age	1105615 seconds (1.8 weeks) since version was first installed.
Product	Firefox
Version	4.0b11
Build ID	20110203140743
Branch	2.0
OS	Mac OS X
OS Version	10.6.6 10J567
CPU	amd64
CPU Info	family 6 model 37 stepping 5
Crash Reason	EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
Crash Address	0xffffffffb0b81000
User Comments	
App Notes	Renderers: 0x22600,0x24300,0x20400
Processor Notes	
EMCheckCompatibility	False
Bugzilla - Report this Crash

Crashing Thread

Frame	Module	Signature [Expand]	Source
0		@0x7fffffe00830	
1	XUL	nsReadFromRawBuffer	xpcom/io/nsPipe3.cpp:1171
2	XUL	nsPipeOutputStream::WriteSegments	xpcom/io/nsPipe3.cpp:1137
3	XUL	nsDataChannel::OpenContentStream	netwerk/protocol/data/nsDataChannel.cpp:113
4	XUL	nsBaseChannel::BeginPumpingData	netwerk/base/src/nsBaseChannel.cpp:240
5	XUL	nsBaseChannel::AsyncOpen	netwerk/base/src/nsBaseChannel.cpp:591
6	XUL	imgLoader::LoadImage	modules/libpr0n/src/imgLoader.cpp:1660
7	XUL	nsContentUtils::LoadImage	content/base/src/nsContentUtils.cpp:2531
8	XUL	nsImageLoadingContent::LoadImage	content/base/src/nsImageLoadingContent.cpp:733
9	XUL	nsImageLoadingContent::LoadImage	content/base/src/nsImageLoadingContent.cpp:658
10	XUL	nsHTMLImageElement::SetAttr	content/html/content/src/nsHTMLImageElement.cpp:521
11	XUL	nsIDOMHTMLImageElement_SetSrc	 dom_quickstubs.cpp:17444
12	XUL	js_SetPropertyHelper	js/src/jscntxtinlines.h:751
13	XUL	js::mjit::stubs::SetName<0>	js/src/methodjit/StubCalls.cpp:261
14	XUL	js::mjit::ic::SetProp	js/src/methodjit/PolyIC.cpp:1741
15		@0x103202ae7	
16	XUL	js::mjit::JaegerShot	js/src/methodjit/MethodJIT.cpp:748
17	XUL	js::Invoke	js/src/jsinterp.cpp:658
18	XUL	js::ExternalInvoke	js/src/jsinterp.cpp:862
19	XUL	JS_CallFunctionValue	js/src/jsapi.cpp:5053
20	XUL	nsXPCWrappedJSClass::CallMethod	js/src/xpconnect/src/xpcwrappedjsclass.cpp:1701
21	XUL	nsXPCWrappedJS::CallMethod	js/src/xpconnect/src/xpcwrappedjs.cpp:588
22	XUL	PrepareAndDispatch	xpcom/reflect/xptcall/src/md/unix/xptcstubs_x86_64_darwin.cpp:153
23	XUL	XUL@0xe1a2da	
24	XUL	nsEventListenerManager::HandleEventSubType	content/events/src/nsEventListenerManager.cpp:1127
25	XUL	nsEventListenerManager::HandleEventInternal	content/events/src/nsEventListenerManager.cpp:1222
Status: UNCONFIRMED → NEW
Ever confirmed: true
OS: Mac OS X → Windows 7
Summary: Crash while reading a huge (about 2GB) file with the DOM File api [@ nsAString_internal::SetCapacity] → Crash while reading a huge (about 2GB) file with the DOM File API [@ nsAString_internal::SetCapacity][@ nsReadFromRawBuffer]
Crash Signature: [@ nsAString_internal::SetCapacity] [@ nsReadFromRawBuffer]
This is showing up as a common OOM crash with the signature NS_ABORT_OOM(unsigned int) | AppendASCIItoUTF16(nsACString_internal const&, nsAString_internal&)

see bp-190b1a58-636c-4fe5-9522-f05312131210

I believe http://hg.mozilla.org/releases/mozilla-aurora/annotate/37f2d83153a5/content/base/src/nsDOMFileReader.cpp#l526 should be using fallible methods. The Base64Encode method is already correctly fallible.

It doesn't appear that the GetAsText codepath is affected because nsDOIMFileReader::ConvertStream uses fallible SetLength.
Crash Signature: [@ nsAString_internal::SetCapacity] [@ nsReadFromRawBuffer] → [@ NS_ABORT_OOM(unsigned int) | AppendASCIItoUTF16(nsACString_internal const&, nsAString_internal&) ]
Summary: Crash while reading a huge (about 2GB) file with the DOM File API [@ nsAString_internal::SetCapacity][@ nsReadFromRawBuffer] → OOM Crash while reading files (about 2GB) file with the DOM File API - should use the fallible allocator and propagate errors
Whiteboard: [mentor=benjamin@smedbergs.us][lang=c++][crashkill:P2]
Blocks: 943017
Attached patch bug613472.patch (obsolete) — — Splinter Review 
Assignee: nobody → pylaurent1314

        Attachment #8370040 -
        Flags: review?(benjamin)

    
    

    
  
Comment on attachment 8370040 [details] [diff] [review]
bug613472.patch

This appears correct except that the infallible version of AppendASCIItoUTF16 doesn't appear to crash on failure any more. I'd also expect it to warn about the unused result of calling the fallible version... you should result-check it and NS_ABORT_OOM in the failure case.

        Attachment #8370040 -
        Flags: review?(benjamin) → review-

    
    

    
  


  

        Attachment #8370040 -
        Attachment is obsolete: true

        Attachment #8370574 -
        Flags: review?(benjamin)

    
    

    
  
Comment on attachment 8370574 [details] [diff] [review]
bug613472-V2.patch

Excellent, thanks!

        Attachment #8370574 -
        Flags: review?(benjamin) → review+

    
    

    
  
Thank you! :)
Keywords: checkin-needed

    
    

    
  
https://hg.mozilla.org/mozilla-central/rev/cc975b343079
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla30
Component: DOM → DOM: Core & HTML

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: