1700187 - Encrypted downloads from mega.io / mega.co.nz cause high transient memory usage

Status: NEW

(Core :: DOM: File, defect, P3)

Description

[pwosj]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:86.0) Gecko/20100101 Firefox/86.0

Steps to reproduce:

• Set firefox download settings to "ask every time".
• Create a file of about 2 GB. Appended is some code to do it fast.
• Upload it to mega.io / mega.co.nz https://en.wikipedia.org/wiki/Mega_(service)
• Download it using right click -> download -> standard download
• Wait for the confirmation dialog that asks to open/save. When that shows up, see the memory usage. 8 GB in one process, 2 in another.

#include <iostream>
#include <fstream>
#include <ostream>

const int TOTAL{1000 * 1000 * 1000 * 2};

int main(int argc, const char *argv[])
{
  std::ofstream outfile_s{"FF_DL_MEM_USAGE.txt"};
  if (outfile_s.bad()) {
    std::cerr << "Couldn't open file." << std::endl;
    return -1;
  }
  for (int i = 0; i < TOTAL; i++) {
    outfile_s << "b";
  }
  printf("done");
  return 0;
}

Actual results:

High memory usage, as much as 4 times the file size.

Expected results:

low memory pressure.

Comment 1

[BugBot [:suhaib / :marco]]

The Bugbug bot thinks this bug should belong to the 'Firefox::File Handling' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 2

[Andrei Purice]

Hey pwosj,
Tried reproducing this on iMac with 10.15 following your steps but the memory was at a constant 600-700 for me on Firefox Nightly 89.0a1 (2021-03-23), beta 88.0b1 and release 87.0.

Can you test the issue while in Safe Mode? You can find helpful info here : https://support.mozilla.org/en-US/kb/troubleshoot-firefox-issues-using-safe-mode .
Also a fresh new profile could help. You can find more about creating a new profile here : https://support.mozilla.org/en-US/kb/troubleshoot-and-diagnose-firefox-problems#w_6-create-a-new-firefox-profile .
If possible, you can test this issue on the nightly build as well. Download the build from : https://www.mozilla.org/en-US/firefox/nightly/all/ .

Comment 3

[pwosj]

I tried it on the same browser version as before, in safe mode, in a new profile.

https://1drv.ms/v/s!AkOZGlm0dk2ulmXiSB68s9_xUMV1?e=81xcoH

The video is pretty boring for the most part. A 2.7 GB zip file is being downloaded with memory usage in Activity Monitor.app is in the foreground.

Better scrub it quickly in a competent video player instead of watching the whole thing. Part after 08:00 minutes is interesting. When the save dialogue appears, there's a spike to ~7.2 GB. After a while the memory is swapped and it settles at ~3.2 GB in the main process. Then I cancel the saving.

For a 1 GB file I created using the code above, the spike is there. It's just not noticeable enough. Using a memory profiler would surely show that.

I doubt it has just been fixed in nightly so I don't intend to test it there; unless you say that you don't notice the spike either.

Comment 4

[Andrei Purice]

We will wait for the devs input since this issue already has a set component. Hopefully they will have a better on point answer towards this issue.

Comment 5

[:Gijs (he/him)]

This is a result of mega's "encryption" service, which serves you all of the 2gb or whatever via content in their webpage as a blob: URI, which is why it is hanging on to all the downloading data. It's not related to Firefox's download code directly. I would somewhat expect other browsers with similar sandboxing solutions to have the same issues, though it's certainly possible that mega's code is different for Firefox and that that is causing leaks. But I'm not really in a position to investigate this; over to people who know about blobs and how they are kept alive...

Comment 6

[Andrew Sutherland [:asuth] (he/him)]

Unfortunately I think some level of transient usage spike is expected, but we can definitely try and optimize things further in terms of using flow control on the pipes/IPCStreams that are used in this scenario, as well as spilling memory-backed Blobs created by new Blob to disk rather than only doing so for XHR/fetch/other Blob-originating APIs. I've filed bug 1706356 as a meta bug for this general high transient use scenario.

Note that I think we explicitly have had bugs about mega before, but I think in those cases we (:baku!) were able to come up with mitigations, ex: bug 1584898 was about mega and pipe memory freeing was improved and bug 1202006 gave us XHR spilling large blobs to disk via temporary files.