678648 - Cannot upload large files with the XMLHttpRequest 2 send(File) method

Status: RESOLVED DUPLICATE of bug 215450

(Core :: Networking, defect)

Description

[Russell Teglas]

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:6.0) Gecko/20100101 Firefox/6.0
Build ID: 20110811165603

Steps to reproduce:

Using FF 6.0b5 and an HTML page that I have written to investigate HTML 5 support for file upload, I attempt to upload files using the XMLHttpRequest 2 send(File) method.


Actual results:

I observed on Windows XP SP3 using sysinternals Process Explorer that the entire contents of the file is read into memory before the browser begins uploading the file.  This means that Firefox does not support large file (e.g. > 2GB) upload using AJAX.  [Note that the same behavior obtains when using the XMLHttpRequest 2 send(FormData) method (i.e. file is uploaded using multipart mime form submit)].


Expected results:

Firefox should stream the file up to the server.  Chrome 13 and Safari 5.1 currently do this:  using these browsers with my HTML 5 test page, I am able to upload 8 GB files without a problem, and Process Explorer reveals that the memory footprint of the browser process increases negligibly while the file is being uploaded.

Comment 1

[Thomas Ahlblom]

Are you able to attach a simple test case or supply an URL where people can try to reproduce the problem?

Comment 2

[Russell Teglas]

Thomas:  thanks for taking a look at this.  Unfortunately, my nodeJS HTTP file server that I wrote for testing file upload/download functionality is behind a firewall, so I cannot supply a URL.  I am willing to share my HTML 5 test page and my nodeJS server code (which consists of a dozen or so nodeJS javascript files) with anyone who is willing to investigate and work on this problem.  This requires installing cygwin32 and nodeJS on a Windows 32bit operating system or just nodeJS on Linux.  Alternatively, if one has access to and control over a web server capable of handling file upload requests, the HTML 5 test page can readily be modified to use that server.  I'll upload the HTML 5 test page for now.

Comment 3

[Russell Teglas]

Attachment: HTML page for testing HTML 5 file upload/download functionality

I use this page in conjunction with a nodeJS HTTP file server that I have written to learn about nodeJS as well as to investigate the file upload/download capabilities of various browsers.  If need be, I am willing to share the source for my test nodeJS HTTP file server.

Comment 4

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

The testcase contains the following code:

    function uploadFile(file)
    {
        filename = file.name;
        reader = new FileReader();
        reader.onloadend = onFileLoadEnd;
        reader.readAsArrayBuffer(file);
    }

That explicitly loads the contents of a file into memory. That code appears to be run when you hit the "Upload Files (Array)" button.

If you instead hit the "Upload Files (Form)" or "Upload Files" buttons, it *appears* that the testcase uses methods which allow us to stream the data to the server without loading the whole file into memory.

If you're still seeing the problem when pushing one of those buttons, please provide a simpler testcase which doesn't use FileReaders to ensure that you're not accidentally hitting the FileReader code paths.

Comment 5

[Boris Zbarsky [:bzbarsky]]

Necko's file upload APIs don't support files larger than 2GB in size (in particular, nsIInputStream uses 32-bit signed ints for its available length and such).

There are existing bugs on this....

Comment 6

[Russell Teglas]

Jonas & Boris:  thanks for taking a look at this.  

Jonas:  my test page has three ways of testing file upload: (1) xhr.send(File), (2) xhr.send(FormData), and (3) xhr.send(ArrayBuffer).  Of course (3) requires that one read the entire file into memory, but I observed (using sysinternals Process Explorer on Win32) that both (1) and (2) apparently read the entire file into memory as well.  I can provide a simpler HTML 5 test page, but I think it is easy to reproduce the problem as stated.

Boris:  I tried searching for a bug on this but could not find one that matched the problem as I described it.  Aside from the problem with 32-bit ints (signed or unsigned) being used to hold the size of a file, there is the problem that FF (using (1) or (2)) evidently loads the entire contents of a file into memory before transmitting it to the server.  Indeed, using sysinternals Process Explore on Win32, I observed that uploading a 256 MB file led to a 256 MB increase in process memory, followed by an interval of time during which the contents of the file was read from the disk into memory.  At the end of this interval, one clearly sees a spike in network I/O where the bytes in memory are sent to the server, and once the request is complete, the 256 MB of process memory are relinquished to the OS.

Comment 7

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Ok, let's make this bug *only* handle the read-all-of-the-file-into-memory problem. The 32bit problem is covered elsewhere and not what this bug is about anyway.

If we are indeed loading the whole file into memory, then that sounds like a problem in necko. Reassigning.

Russell: Do you see the same behavior when you do a normal <form> submission if the form contains a <input type=file> where you've attached the large file.

Jason: could you have a look at this. If we are indeed reading the whole file into memory, then that defeats the purpose of these XHR APIs.

Comment 8

[Boris Zbarsky [:bzbarsky]]

Russel, do you have any extensions installed?  In particular, anything like Firebug?

Comment 9

[Russell Teglas]

New information:  it appears that the whole file is read into memory only under certain circumstances where it is not too large.  I now observe that FF 6.0b5 can upload files with size <= 4GB (where 4 GB == 4*(1<<30) bytes) using xhr.send(File) without reading the entire file into memory.  So, some form of streaming is being used, which is a good sign.  However, when I attempt to upload a file whose size > 4GB, I observe that the upload abruptly terminates as soon as 4GB have been uploaded.

Jonas:  using a normal <form> submission as you described, I observe that I can upload files whose size < 2GB (without apparently reading the whole file into browser memory), but attempting to upload a file whose size == 2GB exactly fails.

Boris:  yes, I have the latest Firebug installed, which I have been using regularly in my work for some years now.  Could this be a problem, or is there something that you think I should look at with Firebug?

It appears that what I am seeing after all is just the 32bit signed/unsigned int problem (mentioned by Boris in a previous comment) with representing the length of a file:  apparently a 32 bit signed int is used when uploading a file using the traditional <form> submit, and a 32 bit unsigned int is used with XHR.

My real interest here is to be able to say (by the beginning of 2012 at least) that uploading files with size > 4GB works with Firefox XHR (more specifically, xhr.send(File)).

Comment 10

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Ok, if the 32bit limit is all that you're seeing, then it sounded like this was a dupe.

And yes, I would recommend trying to uninstall firebug to see if that changes the memory behavior that you are seeing.

Comment 11

[Matthias Versen [:Matti]]

I think you mean bug 215450.

Comment 13

[Boris Zbarsky [:bzbarsky]]

> Could this be a problem

It could; Firebug could be caching upload streams in memory for debugging purposes...

Comment 14

[Russell Teglas]

(In reply to Boris Zbarsky (:bz) from comment #13)
> > Could this be a problem
> 
> It could; Firebug could be caching upload streams in memory for debugging
> purposes...

I found no difference in behavior after disabling or uninstalling Firebug.  It is clear that there is a problem with how the size of a file is represented by Firefox:  when attempting to upload an 8 GB file, the content-length request header has value 4294967295 (== (1<<32)-1) rather than 8589934592 (== 8 * (1<<32))

Comment 15

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

You originally talked about two distinct problems:

1. Memory consumption didn't seem to be what it should be. I.e. we appeared to load more of the file into memory than needed.

2. You couldn't send the file at all if it was too big.

These are two distinct problems. People have been trying to help you track down both issues.

2 is a known problem and is filed as bug 215450, no need to discuss it more here.

1 might be affected by firebug. But your comments have been inconsitent enough that i'm no longer convinced that it exists at all, with or without firebug.

Comment 16

[Russell Teglas]

Jonas:  I apologize for the confusion and inconsistency here.  There is no issue with 1, and indeed what I was seeing before was exactly what Boris suggested:  Firebug apparently does hold upload streams in memory.  With Firebug disabled, it is clear that traditional <form> upload and xhr.send(File) upload both stream the contents up to the server without any appreciable memory footprint on the Firefox.

Regarding bug 215450, it is clear that there is a currently a 2 GB limit for traditional <form> upload and a 4 GB limit for xhr.send(File) upload.  Presumably both of these limits will be removed once bug 215450 is fixed.

Comment 17

[Boris Zbarsky [:bzbarsky]]

Russell, thanks for the clarification about the memory behavior.  That's exactly what I was asking about in terms of Firebug.

And yes, bug 215450 should address both limits (with possibly a followup needed for XMLHttpRequest specific code).