FileReader does not work with big files on Windows due to OOM (returns error code 24)

NEW
Unassigned

Status

()

Core
DOM
8 years ago
5 years ago

People

(Reporter: Giorgio, Unassigned)

Tracking

unspecified
x86
Windows Server 2003
Points:
---

Firefox Tracking Flags

(blocking2.0 -)

Details

(Whiteboard: WINDOWS ONLY)

Attachments

(1 attachment)

1.07 KB, text/html
Details
(Reporter)

Description

8 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.2; WOW64; en-US; rv:1.9.3a5pre) Gecko/20100510 Minefield/3.7a5pre ( .NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.2; WOW64; en-US; rv:1.9.3a5pre) Gecko/20100510 Minefield/3.7a5pre ( .NET CLR 3.5.30729)

hi, if you try to read (async) a big file ( ~> 60mb) the filereader object returns error code:

NOT_READABLE_ERR 	24 	File could not be read.

i'm on xp x64 

var reader = new FileReader();

reader.addEventListener("error", function(evt)
{
     alert(evt.target.error.code);
},false);

reader.readAsBinaryString(file);

Reproducible: Always
Works fine for me.  I just did the above with a 256MB file, and it worked with no problems.
(Reporter)

Comment 2

8 years ago
Created attachment 444657 [details]
test x64

heres my test

progress will change from 10% to 20% on a ~700mb avi file
That testcase works fine for me over here.
(Reporter)

Comment 4

8 years ago
hi boris, what is your system?

two friends tests on seven 64 bit (firefox 3.6) and windows xp x64 (firefox 3.6) and they get not_readable

ive 4gb of ram and running on developer preview 3.7...
(Reporter)

Comment 5

8 years ago
another friends tests with Seven 32 bit, same error
I'm on a Mac (10.5, running current trunk Firefox nightlies).

Let me grab my 64-bit Win7 system and see whether I can reproduce.
(Reporter)

Comment 7

8 years ago
ok thank you

again... tests were made with 700~ mb files (avis)
Ah, I see.  That's not what comment 0 said.
(Reporter)

Comment 9

8 years ago
i'm sorry... the length in bytes readed is about 60mb on 700mb files, and is variable
Component: Networking: File → DOM
QA Contact: networking.file → general
OK, I can reproduce this on Windows (64-bit Win7) with a 700mb file.  The error I get in nsDOMFileReader::DispatchError is NS_ERROR_OUT_OF_MEMORY.

Note that when I tried to create the 700mb file in Emacs I also got out-of-memory errors.
(Reporter)

Comment 11

8 years ago
the disk memory (paging file) is not used?

off topic: is possible to send data with xhr chuck per chunk?
OK, so in nsDOMFileReader::OnDataAvailable, this call:

  mResult.GetMutableData(&buf, oldLen + aCount);

makes buf null.

At this point, oldLen is 134201344 and aCount is 65536.  Or in hex, 0x7ffc000 and 0x10000 respectively, for a sum of 0x800c000.

  0x8000000 == 2^27 or 128MB.

This isn't consistent; sometimes oldLen is 0xffff000 here, and sometimes 0x7ffb000.

In any case, it's definitely just us trying to allocate a large buffer and the OS saying "can't do that".

Confirming, but this doesn't seem like something we can easily change while still exposing a single buffer with the data, if the OS just won't allow allocating a buffer that big.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Summary: html5 FileReader does not works with big files (returns error code 24) → html5 FileReader does not works with big files on Windows due to OOM (returns error code 24)
Whiteboard: WINDOWS ONLY
Apparently at least for a 32-bit process (which all the Firefox builds you tested are) the OS on Windows limits the entire address space to 2GB by default (there's an OS setting to change it).

So depending on the exact layout of things in that 2GB address space, there may not be anywhere to put a large single buffer.  The page file is irrelevant, since this is a matter of address space, not physical memory.  I'm running this on a machine with 12GB of ram, for example, but the OS won't let any given process use more than 2.

So we really need to not store the whole file in RAM at once if we can avoid it.  Even in discontiguous buffers...
blocking2.0: --- → ?
(Reporter)

Comment 14

8 years ago
i know the os limits!

event.target.result contains the entire file, not a part

"in my opinion" (i'm a high-level-language developer, so i don't really know how this should work... and i'm sorry if i say some of dumb)

the .result on filereader must contain not the entire binary data of the file, but just the "last readed part" in the progress step

and xhr should be able to send an http request by steps (freeing memory on each step, probably how the classic form submission with multipart/form-data works) 

xhr is originally designed to send small data, not big binary files

a second process on windows can help, but is not a good solution
> the .result on filereader must contain not the entire binary data of the file,
> but just the "last readed part" in the progress step

That's something to take up with the relevant working group, right?

> and xhr should be able to send an http request by steps 

Yes, it does just that if you pass it a File.  But here you're trying to get the file into a JS String for some reason, no?

Updated

8 years ago
Summary: html5 FileReader does not works with big files on Windows due to OOM (returns error code 24) → FileReader does not work with big files on Windows due to OOM (returns error code 24)
(Reporter)

Comment 16

8 years ago
as i say i don't know nothing about low level programming and mozilla internals and i'm not sure if the filereader api should be modified, maybe the windows problem can be resolved in other ways, what do you think about?

i'm trying to post binary data (file upload) via xmlhttprequest (multipart/form-data)

"
Content-Disposition: form-data; etc
Content-Type: application/octet-stream

binarystring

"

if i can send the binarystring in chunks 
by multiple sendAsBinary(), the problem is resolved, but only if the target.result property gives me only the last chunk, not the entire file data

I'm sorry Boris but my English is poor, i really want to help Mozilla, but with those language difficulties you need to "read between the lines"...

i will STANDARDIZE my English

thank you again for your participation
> i'm trying to post binary data (file upload) via xmlhttprequest
> (multipart/form-data)

Ok...  So 

  var formData = new FormData();

then append the values you want (strings, File objects, whatever). Then pass the formData to send().  If you put a File in there, it'll read the file off your disk in chunks inside send() and send it in chunks.
(Reporter)

Comment 18

8 years ago
ok, so the mine main problem is resolved... thank you
i hope you will resolve the memory error
thank you again!
You can also simply do

xhr = new XMLHttpRequest;
xhr.open(...);
xhr.send(file);

This works in Firefox 3.6 and sends the raw binary data in the file, without you having to worry about running out of memory.

FormData sends file contents multiple/form-data encoded (like a <form>).
(Reporter)

Comment 20

8 years ago
hi, ive tested FormData, but there is no progress event for this

xhr.addEventListener("progress", function (evt){log.innerHTML += (evt.loaded / evt.total) + "%<br>";}, false);

in plus, it still does not works with big files
Please file two separate bugs for those two issues.
Giogio, did you mean "uploadprogress" instead of "progress" there?

Please do file a separate bug on formdata not working with big files; cc me.
Btw, please don't use "uploadprogress". Instead use

xhr.upload.addEventListener("progress", ...);

"uploadprogress" is a mozilla-ism and not standardized.

And please cc me too on anything you file.

Updated

8 years ago
Duplicate of this bug: 561980
(Reporter)

Comment 25

8 years ago
ok jonas and boris, thank you for this
i will open a new bug when i can do good tests, ive no time now :(

off topic: how i can be always updated on firefox every single improvement (like a goals list?) upload property in xhr is new to me and reading MDC page per page is impossible. please do not reply here but in private mail! thank you
Not blocking 1.9.3 on this as it's not obvious that this is something we can even fix, given the limitations of the OS.
blocking2.0: ? → -
You need to log in before you can comment on or make changes to this bug.