681931 - sendAsBinary might be leaking memory (in our tests it does)

Status: RESOLVED WORKSFORME

(Core :: DOM: Core & HTML, defect)

Description

[Davit Barbakadze]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.112 Safari/535.1

Steps to reproduce:

This is a sequel to an effort here: https://bugzilla.mozilla.org/show_bug.cgi?id=680847.

Initially when we noticed the leakage, we blamed it on FileReader, since it was first in the chain and showed some symptoms (in our implementation). But now as we moved on to actual upload, another problem popped out, now for XMLHttpRequest and sendAsBinary.

This is the code:

function sliceMe() {
    var file = document.getElementById('file').files[0];
    var fr = new FileReader;
    var xhr = new XMLHttpRequest;
    var chunkSize = 2097152, chunks = Math.ceil(file.size / chunkSize), chunk = 0;
    var mimeType = 'application/octet-stream';
    
    function sendMultipart(bin) {
        var multipartDeltaSize = 0, boundary = '----boundary764876871231', dashdash = '--', crlf = '\r\n', multipartBlob = '',
            args;
            
        args = {
            name : file.name,
            chunk: chunk,
            chunks: chunks    
        }
        
        xhr.open("post", 'upload.php', true);
        
        xhr.onreadystatechange = function() {    
            if (xhr.readyState === 4) {            
                if (++chunk < chunks) {
                    uploadNext();
                }
            }            
        };
                
        xhr.setRequestHeader('Content-Type', 'multipart/form-data; boundary=' + boundary);
        
        for (var name in args) {
            multipartBlob += dashdash + boundary + crlf + 'Content-Disposition: form-data; name="' + name + '"' + crlf + crlf;
            multipartBlob += unescape(encodeURIComponent(args[name])) + crlf;
        }            

        // Build RFC2388 blob
        multipartBlob += dashdash + boundary + crlf +
            'Content-Disposition: form-data; name="file"; filename="' + unescape(encodeURIComponent(file.name)) + '"' + crlf +
            'Content-Type: ' + mimeType + crlf + crlf + bin + crlf +
            dashdash + boundary + dashdash + crlf;    

        xhr.sendAsBinary(multipartBlob);
    }
    
    
    function uploadNext() {
        var start, end;
        
        start = chunk * chunkSize;
        end = start + chunkSize >= file.size ? file.size : start + chunkSize;
                
        fr.onload = function() {
            sendMultipart(fr.result);    
        };
        fr.readAsBinaryString(file.mozSlice(start, end));
    }
    
    uploadNext();    
}

Fiddle: http://jsfiddle.net/8zzrT/3/


Actual results:

After each call to xhr.sendAsBinary memory usage crawls up. If upload happens slowly enough, browser manages to drop it time to time by some amount (but never completely), but when it's fast and is initiated for big files, that trigger hundreds of requests, it quickly eats up all available memory, chokes and makes workstation unusable.

Re-using of XMLHttpRequest didn't help in this case and it always is sendAsBinary, that causes jump in memory usage.


Expected results:

Memory usage must have been dropped after each sendAsBinary cycle.

Might this be related: https://bugzilla.mozilla.org/show_bug.cgi?id=681479 ?

Comment 1

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

What is "some amount"?  I uploaded a 700 MB file and saw maybe a 10 MB rise in memory usage ...

Comment 2

[Davit Barbakadze]

Have you uploaded it chunk by chunk, using the above code?

My tests had the following results:

Uploading: 702.5 mb in 2 mb chunks

Initial memory consumption: 173.4 mb
After upload: 2.42 gb

But then I thought you might have been checking on windows machine. So I tried it in XP (through Parallels) and wow, sendAsBinary behaves really good on there. Rise in memory for every cycle is around 100 mb on average, but more importantly it drops down memory usage back to initial point (or somewhere near). So probably it doesn't leak on Windows.

Comment 3

[Davit Barbakadze]

Tried on FF4 with deprecated slice method, also very stable memory consumption, drop after every cycle, etc. But it suddenly crashed after 400 mb got uploaded... :| But maybe it's due to virtual environment.

Comment 4

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

It's possible it's Mac only.  I have been attempting to reproduce on Windows.

Marcia, could you or somebody else from QA attempt to reproduce this on Mac?

Comment 5

[Norah Smith]

Davit, I was having the same problem with my variable-size chunk uploading that used SendAsBinary and I noticed the a similarly bad leak. I was testing on Firefox 6 and 7; Aurora 8a2 doesn't seem to have the same memory creep for me.

What I realized after testing though, was that sending the blobs using the Safari-style simpler post format explained here: http://www.deadmarshes.com/Blog/20110413023355.html allowed me to post chunks of the file without having to use FileReader at all. Just slice out a blob and append to the form object. When using this approach while uploading a 1GB file, Firefox memory only rose 100MB across the entire upload.

Comment 6

[Davit Barbakadze]

Norah, but they are using FormData in the example you've referenced... Aren't you encountering this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=649150 for all Firefoxes prior to version 7, when uploading blobs with FormData?.. 

In fact that bug here: https://bugzilla.mozilla.org/show_bug.cgi?id=680847, under which you placed your first comment, was an effort to overcome FormData+Blob problem for current and older Geckos...

Comment 7

[Norah Smith]

Hi there, so actually, our upload logic doesn't require the FormData-uploaded blobs to have filenames. We get that and other file metadata through other means, so it works out okay here. I can understand how that could be a problem though in other instances.

Comment 8

[Davit Barbakadze]

If you do not require multipart on server-side, you could send blobs directly.

Comment 9

[Norah Smith]

Also true, but for some reason when sending raw binary data, the servers I have to communicate with had their own set of annoyances going on, so I ended up back at multipart.

Comment 10

[Ioana (away)]

WFM on:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:29.0) Gecko/20100101 Firefox/29.0