Closed Bug 1532446 Opened 5 years ago Closed 3 years ago

[meta] Hundreds of MB of stale temp files in app_tmpdir

Categories

(GeckoView :: General, defect)

65 Branch
Unspecified
Android
defect

Tracking

(firefox66 wontfix, firefox67 affected)

RESOLVED FIXED
Tracking Status
firefox66 --- wontfix
firefox67 --- affected

People

(Reporter: jan-bugreport, Unassigned)

References

Details

(Keywords: meta)

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0

Steps to reproduce:

  1. Received a warning that my phone storage is full and my phone may not work correctly.
  2. Saw that Firefox has >500 MB of "user data" in addition to cache.
  3. Opened file:///data/data/org.mozilla.firefox/app_tmpdir/

Actual results:

Saw hundreds of megabytes in various temporary files, most of them last modified more than three months ago.

Most are named "mozilla-temp-xxxxxxx" where xxxxxxx is a random number of varying length.

There's also tmp-ekv.xpi (apparently a copy of an addon I have installed), "tmpaddon", and a series of "tmpaddon-xxxxxx" files where xxxxxx are 5-6 hex digits.

Expected results:

Anything in app_tmpdir, or anything in app_tmpdir older than x days, should be deleted on every startup.

Blocks: fatfennec
OS: Unspecified → Android

Thanks to a friendly soul on IRC who encouraged me to use USB debugging and native code, I found
https://mail.mozilla.org/pipermail/mobile-firefox-dev/2013-November/000414.html
which mentions a cleanup extension, which is archived on the Web Archive
https://web.archive.org/web/20150304170154/http://people.mozilla.org/~mfinkle/fennec/addons/mobileprofiles.xpi
view source to get the actual download link if it changes, mine was
https://web.archive.org/web/20150304170154if_/http://people.mozilla.org/~mfinkle/fennec/addons/mobileprofiles.xpi

The code gives a good idea how the APIs work, so either review and then trust and install it, or "simply" start a USB debug session with WebIDE, click "main process" (on the desktop in the WebIDE), open the console, and use:


const { classes: Cc, interfaces: Ci, utils: Cu } = Components;
Cu.import("resource://gre/modules/FileUtils.jsm");

tmpdir = FileUtils.getDir("TmpD", [], false)
console.log('tmpdir: ' + tmpdir.path);
it = tmpdir.directoryEntries
while (it.hasMoreElements()) {
f = it.getNext().QueryInterface(Ci.nsIFile);
console.log(f.path);
// f.remove(false);
}


(when you're happy with what will be deleted, uncomment the remove line and paste the code again starting from tmpdir - if there are directories, either clean those up first, or change the parameter to f.remove to true if you feel adventurous and want to nuke them recursively)

This freed up ~400 MB on my phone.

I've noticed something similar recently - most of it was dozens of copies of the same add-on.

If we end up seeing duplicate reports we may want to bump up the priority.
David, is this something the addons team can look into, since it may be addon related?

Flags: needinfo?(ddurst)
Priority: -- → P2

The tmpaddon part of this bug is not specific to Android. I have also been seeing it on Linux, and there is even a bug report about it: bug 1385303

Depends on: 1385303

There are two locations that create tmpaddon files.
This bug's root cause is probably the same as bug 1495021 : https://searchfox.org/mozilla-central/rev/8ff2cd0a27e3764d9540abdc5a66b2fb1e4e9644/toolkit/mozapps/extensions/internal/XPIInstall.jsm#3411

No longer depends on: 1385303
See Also: → 1495021, 1385303

Since this is a temp dir that shouldn't have old files, I'd recommend adding code to clean it, instead of just fixing the individual issues that left these specific files. There will always be edge cases like crashes or bugs that leave garbage there, and the code is needed anyways to clean up existing installs.

Note that some of the files were named mozilla-temp-xxxxxxx (xxxxxxx being a placeholder), not something with tmpaddon, so the tmpaddon bug is likely NOT the only issue.

While this may seem like "papering over" the issue, I'd consider it good defense in depth that also mitigates the biggest part of the problem while the individual issues are being identified and addressed.

Flags: needinfo?(ddurst)

I can reliably reproduce the mozilla-temp-xxxxxxxxx files. Bug 1555115

See Also: → 1555115

I too had the symptoms Jan mentions in comment 0 : hundreds of tmp-???.xpi files, all about 1MB each, dating back over a couple of years (I think), but some were from this month.

Using Jan's instructions from comment 1 , I could delete them all (I had no other way of cleaning them out). Saved over 500MB! My FFA app storage data went from 1GB to 450MB.

Depends on: 1670503

(In reply to Paul Gregg from comment #8)

I too had the symptoms Jan mentions in comment 0 : hundreds of tmp-???.xpi files, all about 1MB each, dating back over a couple of years (I think), but some were from this month.

  • Bug 1670503 addresses tmp-???.xpi files.
  • Bug 1555115 addresses mozilla-temp-* files.
  • Bug 1385303 addresses tmpaddon-* files (not exclusively in Android, but at least in Linux they go to the /tmp/ dir which is wiped at every boot).

I'm not sure if this should be considered as 3 separate bugs or as 3 symptoms of the same bug—the fact that app_tmpdir should be cleaned up periodically (at startup, or when closing Firefox, or upon user request).

(In any case, I hope it's fixed soon because my phone is running out of space!)

Status: UNCONFIRMED → NEW
Ever confirmed: true
We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → INCOMPLETE
Product: Firefox for Android → Firefox for Android Graveyard

A similar bug has been reported in GitHub as https://github.com/mozilla-mobile/fenix/issues/14646 but with much less information as this bug report.
Should this bug be re-reported in the GitHub issue tracker with all the information we have, or just move this conversation to the already existing issue?

PS: Unfortunately, I am currently unable to check if the issue has been fixed in Nightly because my phone doesn't have enough space to install any more apps because of this bug.

(In reply to cousteau from comment #13)

A similar bug has been reported in GitHub as https://github.com/mozilla-mobile/fenix/issues/14646 but with much less information as this bug report.
Should this bug be re-reported in the GitHub issue tracker with all the information we have, or just move this conversation to the already existing issue?

The two known causes of the reported observation in this bug are tracked separately, at bug 1670503 and bug 1555115. These two issues need to be resolved at the Gecko/GeckoView side, and since they are an implementation detail of GeckoView, there is probably not much point in opening a new issue in Fenix.

PS: Unfortunately, I am currently unable to check if the issue has been fixed in Nightly because my phone doesn't have enough space to install any more apps because of this bug.

How do you know that your issue is caused by this bug? If you were able to check files through a file manager or adb, then you can delete the files to free up some space.

(In reply to Rob Wu [:robwu] from comment #14)

How do you know that your issue is caused by this bug? If you were able to check files through a file manager or adb, then you can delete the files to free up some space.

Because I can point Firefox to file:///data/data/org.mozilla.firefox_beta/app_tmpdir/ and see the files, which match the filename patterns, and whose sizes add up (roughly) to the amount of storage Firefox is using. So the "file manager" I'm using is the Firefox app itself (which doesn't have a feature to delete files).
Other file managers don't show these files (they seem to be "owned" by the Firefox Beta app so other apps can't touch them), but some apps such as Applications and Drive Storage Analyzer show that Firefox is using 0.96 GB of storage (without showing the individual files, only the total size the app is using).
I will try adb to see if I can delete the files.

The two known causes of the reported observation in this bug are tracked separately, at bug 1670503 and bug 1555115. These two issues need to be resolved at the Gecko/GeckoView side, and since they are an implementation detail of GeckoView, there is probably not much point in opening a new issue in Fenix.

  1. Are those 2 issues already in the right category? I see one of them is in Toolkit and the other in Core. Have these bugs been reported in Gecko/GeckoView, or should they be reported? (or have these bugs moved to the right category)
  2. I suspect that solving the bug won't "cure the symptoms": bug 1385303 was already solved but the files it created didn't get deleted; I assume the same will happen once the other two bugs are solved.
    I would like to report a bug requesting app_tmpdir to be wiped (periodically or upon request), since having that directory use that much storage space is a problem by itself. What's the right platform/component to report it? (I don't want to report it on the wrong place and then have it closed 2 years later without any progress.)

(In reply to cousteau from comment #15)

I will try adb to see if I can delete the files.

Not too surprisingly, I don't seem to have permission to rm these files if I don't root the phone. (I can't ls the content of /data/data/org.mozilla.firefox_beta/app_tmpdir/ either; but I can ls the individual files if I know their path.)

I'm reopening this bug because it has not been resolved, and an actionable thing for GeckoView is to somehow provide a way to clear app_tmpdir. Either automatically (once the the bugs have been fixed), or an option do it manually (e.g. by somehow putting it under the "Cache" category of the Storage section of Android's built-in storage reporting, from where the storage can also be cleared by the user).

Until this bug is fixed, a convoluted way to manually clean up the data is to enable remote debugging in Firefox for Android, use about:debugging or web-ext to attach a debugger, inspect the main process and execute JavaScript code to enumerate and/or delete the stale temp files (e.g. using a code snippet similar to comment 1).

Severity: normal → --
Status: RESOLVED → REOPENED
Priority: P2 → --
Product: Firefox for Android Graveyard → GeckoView
Resolution: INCOMPLETE → ---
Version: Firefox 65 → 65 Branch

(In reply to Rob Wu [:robwu] from comment #17)

Until this bug is fixed, a convoluted way to manually clean up the data is to enable remote debugging in Firefox for Android, use about:debugging or web-ext to attach a debugger, inspect the main process and execute JavaScript code to enumerate and/or delete the stale temp files (e.g. using a code snippet similar to comment 1).

Thank you; that worked! Not precisely a user-friendly fix :) but at least I managed to make it work. Also thank you for reopening the bug until it is properly fixed.

I'm a bit disappointed that this only freed ~570 MB of space, still leaving >400 MB of disk used by Firefox, but I imagine that using the debugger I could figure out a way to analyze the whole content of the /data/data/org.mozilla.firefox_beta/ directory and see which subdirs are taking that much space. (Or even easier; copy it to the external SD card and analyze that copy with my tool of choice.)

While cleaning out the directory again, I found three types of files:

  1. mozilla-temp-DDDDD where DDDDD is a variable number of digits, many were from 2020-11-11 (the newest date I saw), but there was also a bunch from 2020-04-xx, the oldest I saw was from December 2019.
  2. tmpaddon and tmpaddon-HHHHHH where HHHHHH are exactly 6 hex digits, all from 2019
  3. tmp-XXX.xpi where XXX are exactly 3 alphanumeric characters, dates ranging from 2019 to 2021.

So I guess it could be only those two issues and the tmpaddon files may be ancient leftovers. Either way, they should be cleaned up. My Firefox had piled up ~1 GB of garbage there again :(

It's sad to see that an issue that permanently eats significant amounts of storage on users' phones with no good way to get rid of it (the only way normal users can do it is deleting all their browser data) has remained unfixed for over two years.

This is really multiple bug so I'm tagging this as a meta.

Keywords: meta
Summary: Hundreds of MB of stale temp files in app_tmpdir → [meta] Hundreds of MB of stale temp files in app_tmpdir

(In reply to Agi Sferro | :agi | ni? for questions | ⏰ PST | he/him from comment #20)

This is really multiple bug so I'm tagging this as a meta.

While there are other bugs that cause/caused the unbounded growth of app_tmpdir, the feature request to clean up these stale files is separate and a valid request on its own. Should I file a new bug / feature request?
From comment 17:

(In reply to Rob Wu [:robwu] from comment #17)

an actionable thing for GeckoView is to somehow provide a way to clear app_tmpdir. Either automatically (once the the bugs have been fixed), or an option do it manually (e.g. by somehow putting it under the "Cache" category of the Storage section of Android's built-in storage reporting, from where the storage can also be cleared by the user).

Flags: needinfo?(agi)

Yes please! We can move the app_tmpdir to the cache folder so that users can use "Clear Cache" on Android.

Flags: needinfo?(agi)
Depends on: 1685556

(In reply to Rob Wu [:robwu] from comment #17)

I'm reopening this bug because it has not been resolved, and an actionable thing for GeckoView is to somehow provide a way to clear app_tmpdir. Either automatically (once the the bugs have been fixed), or an option do it manually (e.g. by somehow putting it under the "Cache" category of the Storage section of Android's built-in storage reporting, from where the storage can also be cleared by the user).

Until this bug is fixed, a convoluted way to manually clean up the data is to enable remote debugging in Firefox for Android, use about:debugging or web-ext to attach a debugger, inspect the main process and execute JavaScript code to enumerate and/or delete the stale temp files (e.g. using a code snippet similar to comment 1).

Hi!

Can you please give us the code snippet you used to clean up space of your Fenix,please? I'm not familiar to Firefowmx developmentand I can't code it, although I could run it thanks to your explanations.
Thanks!

Flags: needinfo?(rob)

This bug has been fixed by bug 1685556, starting from Fenix based on 89 (to be released soon I guess). When you update to that Fenix version, the stale files will automatically be removed.

Can you please give us the code snippet you used to clean up space of your Fenix,please? I'm not familiar to Firefowmx developmentand I can't code it, although I could run it thanks to your explanations.

Wait a few days and update to the latest version of Fenix, based on version 89.

Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Flags: needinfo?(rob)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.