[meta] Hundreds of MB of stale temp files in app_tmpdir
Categories
(GeckoView :: General, defect)
Tracking
(firefox66 wontfix, firefox67 affected)
People
(Reporter: jan-bugreport, Unassigned)
References
Details
(Keywords: meta)
User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0
Steps to reproduce:
- Received a warning that my phone storage is full and my phone may not work correctly.
- Saw that Firefox has >500 MB of "user data" in addition to cache.
- Opened file:///data/data/org.mozilla.firefox/app_tmpdir/
Actual results:
Saw hundreds of megabytes in various temporary files, most of them last modified more than three months ago.
Most are named "mozilla-temp-xxxxxxx" where xxxxxxx is a random number of varying length.
There's also tmp-ekv.xpi (apparently a copy of an addon I have installed), "tmpaddon", and a series of "tmpaddon-xxxxxx" files where xxxxxx are 5-6 hex digits.
Expected results:
Anything in app_tmpdir, or anything in app_tmpdir older than x days, should be deleted on every startup.
Thanks to a friendly soul on IRC who encouraged me to use USB debugging and native code, I found
https://mail.mozilla.org/pipermail/mobile-firefox-dev/2013-November/000414.html
which mentions a cleanup extension, which is archived on the Web Archive
https://web.archive.org/web/20150304170154/http://people.mozilla.org/~mfinkle/fennec/addons/mobileprofiles.xpi
view source to get the actual download link if it changes, mine was
https://web.archive.org/web/20150304170154if_/http://people.mozilla.org/~mfinkle/fennec/addons/mobileprofiles.xpi
The code gives a good idea how the APIs work, so either review and then trust and install it, or "simply" start a USB debug session with WebIDE, click "main process" (on the desktop in the WebIDE), open the console, and use:
const { classes: Cc, interfaces: Ci, utils: Cu } = Components;
Cu.import("resource://gre/modules/FileUtils.jsm");
tmpdir = FileUtils.getDir("TmpD", [], false)
console.log('tmpdir: ' + tmpdir.path);
it = tmpdir.directoryEntries
while (it.hasMoreElements()) {
f = it.getNext().QueryInterface(Ci.nsIFile);
console.log(f.path);
// f.remove(false);
}
(when you're happy with what will be deleted, uncomment the remove line and paste the code again starting from tmpdir - if there are directories, either clean those up first, or change the parameter to f.remove to true if you feel adventurous and want to nuke them recursively)
This freed up ~400 MB on my phone.
Comment 2•5 years ago
|
||
I've noticed something similar recently - most of it was dozens of copies of the same add-on.
Updated•5 years ago
|
Comment 3•5 years ago
|
||
If we end up seeing duplicate reports we may want to bump up the priority.
David, is this something the addons team can look into, since it may be addon related?
Comment 4•5 years ago
|
||
The tmpaddon part of this bug is not specific to Android. I have also been seeing it on Linux, and there is even a bug report about it: bug 1385303
Comment 5•5 years ago
|
||
There are two locations that create tmpaddon files.
This bug's root cause is probably the same as bug 1495021 : https://searchfox.org/mozilla-central/rev/8ff2cd0a27e3764d9540abdc5a66b2fb1e4e9644/toolkit/mozapps/extensions/internal/XPIInstall.jsm#3411
Since this is a temp dir that shouldn't have old files, I'd recommend adding code to clean it, instead of just fixing the individual issues that left these specific files. There will always be edge cases like crashes or bugs that leave garbage there, and the code is needed anyways to clean up existing installs.
Note that some of the files were named mozilla-temp-xxxxxxx (xxxxxxx being a placeholder), not something with tmpaddon, so the tmpaddon bug is likely NOT the only issue.
While this may seem like "papering over" the issue, I'd consider it good defense in depth that also mitigates the biggest part of the problem while the individual issues are being identified and addressed.
Updated•5 years ago
|
I can reliably reproduce the mozilla-temp-xxxxxxxxx files. Bug 1555115
Comment 8•4 years ago
|
||
I too had the symptoms Jan mentions in comment 0 : hundreds of tmp-???.xpi files, all about 1MB each, dating back over a couple of years (I think), but some were from this month.
Using Jan's instructions from comment 1 , I could delete them all (I had no other way of cleaning them out). Saved over 500MB! My FFA app storage data went from 1GB to 450MB.
Comment hidden (spam) |
Comment hidden (spam) |
Comment 11•3 years ago
|
||
(In reply to Paul Gregg from comment #8)
I too had the symptoms Jan mentions in comment 0 : hundreds of tmp-???.xpi files, all about 1MB each, dating back over a couple of years (I think), but some were from this month.
- Bug 1670503 addresses
tmp-???.xpi
files. - Bug 1555115 addresses
mozilla-temp-*
files. - Bug 1385303 addresses
tmpaddon-*
files (not exclusively in Android, but at least in Linux they go to the/tmp/
dir which is wiped at every boot).
I'm not sure if this should be considered as 3 separate bugs or as 3 symptoms of the same bug—the fact that app_tmpdir
should be cleaned up periodically (at startup, or when closing Firefox, or upon user request).
(In any case, I hope it's fixed soon because my phone is running out of space!)
Updated•3 years ago
|
Comment 12•3 years ago
|
||
We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.
Assignee | ||
Updated•3 years ago
|
Comment 13•3 years ago
|
||
A similar bug has been reported in GitHub as https://github.com/mozilla-mobile/fenix/issues/14646 but with much less information as this bug report.
Should this bug be re-reported in the GitHub issue tracker with all the information we have, or just move this conversation to the already existing issue?
PS: Unfortunately, I am currently unable to check if the issue has been fixed in Nightly because my phone doesn't have enough space to install any more apps because of this bug.
Comment 14•3 years ago
|
||
(In reply to cousteau from comment #13)
A similar bug has been reported in GitHub as https://github.com/mozilla-mobile/fenix/issues/14646 but with much less information as this bug report.
Should this bug be re-reported in the GitHub issue tracker with all the information we have, or just move this conversation to the already existing issue?
The two known causes of the reported observation in this bug are tracked separately, at bug 1670503 and bug 1555115. These two issues need to be resolved at the Gecko/GeckoView side, and since they are an implementation detail of GeckoView, there is probably not much point in opening a new issue in Fenix.
PS: Unfortunately, I am currently unable to check if the issue has been fixed in Nightly because my phone doesn't have enough space to install any more apps because of this bug.
How do you know that your issue is caused by this bug? If you were able to check files through a file manager or adb
, then you can delete the files to free up some space.
Comment 15•3 years ago
|
||
(In reply to Rob Wu [:robwu] from comment #14)
How do you know that your issue is caused by this bug? If you were able to check files through a file manager or
adb
, then you can delete the files to free up some space.
Because I can point Firefox to file:///data/data/org.mozilla.firefox_beta/app_tmpdir/
and see the files, which match the filename patterns, and whose sizes add up (roughly) to the amount of storage Firefox is using. So the "file manager" I'm using is the Firefox app itself (which doesn't have a feature to delete files).
Other file managers don't show these files (they seem to be "owned" by the Firefox Beta app so other apps can't touch them), but some apps such as Applications and Drive Storage Analyzer show that Firefox is using 0.96 GB of storage (without showing the individual files, only the total size the app is using).
I will try adb
to see if I can delete the files.
The two known causes of the reported observation in this bug are tracked separately, at bug 1670503 and bug 1555115. These two issues need to be resolved at the Gecko/GeckoView side, and since they are an implementation detail of GeckoView, there is probably not much point in opening a new issue in Fenix.
- Are those 2 issues already in the right category? I see one of them is in
Toolkit
and the other inCore
. Have these bugs been reported in Gecko/GeckoView, or should they be reported? (or have these bugs moved to the right category) - I suspect that solving the bug won't "cure the symptoms": bug 1385303 was already solved but the files it created didn't get deleted; I assume the same will happen once the other two bugs are solved.
I would like to report a bug requestingapp_tmpdir
to be wiped (periodically or upon request), since having that directory use that much storage space is a problem by itself. What's the right platform/component to report it? (I don't want to report it on the wrong place and then have it closed 2 years later without any progress.)
Comment 16•3 years ago
|
||
(In reply to cousteau from comment #15)
I will try
adb
to see if I can delete the files.
Not too surprisingly, I don't seem to have permission to rm
these files if I don't root the phone. (I can't ls
the content of /data/data/org.mozilla.firefox_beta/app_tmpdir/
either; but I can ls
the individual files if I know their path.)
Comment 17•3 years ago
|
||
I'm reopening this bug because it has not been resolved, and an actionable thing for GeckoView is to somehow provide a way to clear app_tmpdir
. Either automatically (once the the bugs have been fixed), or an option do it manually (e.g. by somehow putting it under the "Cache" category of the Storage section of Android's built-in storage reporting, from where the storage can also be cleared by the user).
Until this bug is fixed, a convoluted way to manually clean up the data is to enable remote debugging in Firefox for Android, use about:debugging
or web-ext
to attach a debugger, inspect the main process and execute JavaScript code to enumerate and/or delete the stale temp files (e.g. using a code snippet similar to comment 1).
Comment 18•3 years ago
|
||
(In reply to Rob Wu [:robwu] from comment #17)
Until this bug is fixed, a convoluted way to manually clean up the data is to enable remote debugging in Firefox for Android, use
about:debugging
orweb-ext
to attach a debugger, inspect the main process and execute JavaScript code to enumerate and/or delete the stale temp files (e.g. using a code snippet similar to comment 1).
Thank you; that worked! Not precisely a user-friendly fix :) but at least I managed to make it work. Also thank you for reopening the bug until it is properly fixed.
I'm a bit disappointed that this only freed ~570 MB of space, still leaving >400 MB of disk used by Firefox, but I imagine that using the debugger I could figure out a way to analyze the whole content of the /data/data/org.mozilla.firefox_beta/
directory and see which subdirs are taking that much space. (Or even easier; copy it to the external SD card and analyze that copy with my tool of choice.)
Reporter | ||
Comment 19•3 years ago
|
||
While cleaning out the directory again, I found three types of files:
- mozilla-temp-DDDDD where DDDDD is a variable number of digits, many were from 2020-11-11 (the newest date I saw), but there was also a bunch from 2020-04-xx, the oldest I saw was from December 2019.
- tmpaddon and tmpaddon-HHHHHH where HHHHHH are exactly 6 hex digits, all from 2019
- tmp-XXX.xpi where XXX are exactly 3 alphanumeric characters, dates ranging from 2019 to 2021.
So I guess it could be only those two issues and the tmpaddon files may be ancient leftovers. Either way, they should be cleaned up. My Firefox had piled up ~1 GB of garbage there again :(
It's sad to see that an issue that permanently eats significant amounts of storage on users' phones with no good way to get rid of it (the only way normal users can do it is deleting all their browser data) has remained unfixed for over two years.
Comment 20•3 years ago
|
||
This is really multiple bug so I'm tagging this as a meta.
Comment 21•3 years ago
|
||
(In reply to Agi Sferro | :agi | ni? for questions | ⏰ PST | he/him from comment #20)
This is really multiple bug so I'm tagging this as a meta.
While there are other bugs that cause/caused the unbounded growth of app_tmpdir
, the feature request to clean up these stale files is separate and a valid request on its own. Should I file a new bug / feature request?
From comment 17:
(In reply to Rob Wu [:robwu] from comment #17)
an actionable thing for GeckoView is to somehow provide a way to clear
app_tmpdir
. Either automatically (once the the bugs have been fixed), or an option do it manually (e.g. by somehow putting it under the "Cache" category of the Storage section of Android's built-in storage reporting, from where the storage can also be cleared by the user).
Comment 22•3 years ago
|
||
Yes please! We can move the app_tmpdir
to the cache folder so that users can use "Clear Cache" on Android.
Comment 23•3 years ago
|
||
(In reply to Rob Wu [:robwu] from comment #17)
I'm reopening this bug because it has not been resolved, and an actionable thing for GeckoView is to somehow provide a way to clear
app_tmpdir
. Either automatically (once the the bugs have been fixed), or an option do it manually (e.g. by somehow putting it under the "Cache" category of the Storage section of Android's built-in storage reporting, from where the storage can also be cleared by the user).Until this bug is fixed, a convoluted way to manually clean up the data is to enable remote debugging in Firefox for Android, use
about:debugging
orweb-ext
to attach a debugger, inspect the main process and execute JavaScript code to enumerate and/or delete the stale temp files (e.g. using a code snippet similar to comment 1).
Hi!
Can you please give us the code snippet you used to clean up space of your Fenix,please? I'm not familiar to Firefowmx developmentand I can't code it, although I could run it thanks to your explanations.
Thanks!
Comment 24•3 years ago
|
||
This bug has been fixed by bug 1685556, starting from Fenix based on 89 (to be released soon I guess). When you update to that Fenix version, the stale files will automatically be removed.
Can you please give us the code snippet you used to clean up space of your Fenix,please? I'm not familiar to Firefowmx developmentand I can't code it, although I could run it thanks to your explanations.
Wait a few days and update to the latest version of Fenix, based on version 89.
Description
•