944310 - Every update of firefox creates a new huge temporary file and doesn't clear out the old one

Status: RESOLVED INCOMPLETE

(Firefox :: Untriaged, defect)

Description

[Andreas Kohn]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0 (Beta/Release)
Build ID: 20131127030201

Steps to reproduce:

I use nightly, and update it pretty much every day. 




Actual results:

Today I noticed that my /tmp directory was full according to df (4G used out of 4), even though du only showed about 50M used.

Checking with lsof showed multiple files referenced by the firefox process, with names like /tmp/mozilla-temp-(some number). These files were already deleted, explaining the du/df difference. Each of them was reported with a size of ~500M.

Now, I killed firefox, and the files went away. After restart just one appeared, quickly filling up to 500M again. 

And then an update came in, so I let the updater restart firefox. Now two of those files were shown in lsof, with the old one still there, and the new one growing fast.




Expected results:

The restart should remove the old file (I assume it is old), obviously. But also this looks quite huge, out of interest: what is in that file? :)

Comment 1

[Andreas Kohn]

Attachment: lsof-before-restart.txt

Comment 2

[Andreas Kohn]

Attachment: lsof-just-after-restart.txt

Comment 3

[Andreas Kohn]

Attachment: lsof-after-5min.txt

Comment 4

[Andreas Kohn]

I've attached an lsof |grep /tmp/mozilla-temp, taken just before pressing "restart nightly", then just after all windows were opened and youtube continued playing  (I take this as "restart worked fine"). The last one was taken about 5min after the restart completed.

I don't think I've noticed this behavior before, so it might be a fairly recent change (~1 week or so?)

Comment 5

[:Gijs (he/him)]

(In reply to Andreas Kohn from comment #4)
> I've attached an lsof |grep /tmp/mozilla-temp, taken just before pressing
> "restart nightly", then just after all windows were opened and youtube
> continued playing  (I take this as "restart worked fine"). The last one was
> taken about 5min after the restart completed.
> 
> I don't think I've noticed this behavior before, so it might be a fairly
> recent change (~1 week or so?)

I suspect this is the media cache. The temporary files are created by use of http://mxr.mozilla.org/mozilla-central/source/xpcom/io/nsAnonymousTemporaryFile.cpp . The two consumers I'm seeing are listed here: http://mxr.mozilla.org/mozilla-central/ident?i=NS_OpenAnonymousTemporaryFile and the media cache seems the most likely to create huge files...

As far as I can tell these are meant to be removed when the process exits. This should include updates.

How sure are you about the recency? Could you maybe test a nightly from the ~19th (available from http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-11-19-03-02-02-mozilla-central/ ) and check if it has the same behaviour? nsAnonymousTemporaryFile was refactored slightly on the 20th, but in principle that shouldn't have caused something like this... But if you're sure about the 1-week thing that change is quite suspicious. :-)

Comment 6

[Andreas Kohn]

Not sure about the recency at all, apart from that I haven't rebooted my machine for a long time, and usually press the "restart to update" every morning (good timing of updates :D) Might still have restarted firefox for other reasons in the mean time though, so it never managed to acquire the amount of files needed for the system to fail visibly (no space left on /tmp).

I grabbed that nightly from 19th, extracted into a new directory, started it up, and then just let it download the update and restart. This lead to the same behavior:
* before restart I see 1 file, referenced multiple times, and quickly growing
* after restart I have two files, the new one quickly growing, the old one no longer changing in size.

Comment 7

[Andreas Kohn]

Attachment: lsof-restart.txt

Comment 8

[:Gijs (he/him)]

Hrm. That makes me very unsure about what's going on, then. :-(

If you have the time/energy, it could be interesting to see if this behaviour occurred on earlier releases of Firefox, but I'm out of ideas on how this could be happening. We tell the OS to delete the file while it's still being referenced (when Firefox is running), at which point it should be doing that when the process dies... I don't understand how restarting could have us still referencing the temp files created by the old (dead) process.

That said, I don't actually pretend to understand lsof output, so maybe I'm missing something when looking at the output you've attached?

Comment 9

[David Teller [:Yoric] - still alive but not very active]

Theoretically mozilla-temp-* files should be deleted automatically once they are closed. So I guess this might mean that we fail to close the file.

Comment 10

[Andreas Kohn]

So, the header line in lsof on my system (sorry, forgot about that):
> $ lsof | head -1
> COMMAND     PID   TID    USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME

One thing I do find odd that might explain the issue is that even after restart the PID owning the files doesn't change. Going to guess a bit here: there is two processes involved, a wrapper, and a child. When the restart happens only the child gets restarted, not the wrapper. But, the temporary files are owned by the wrapper, so they stay around until that guy is killed.

I can do a bit of trying with different versions of nightly from the site you pointed at earlier, will get back about that.

Comment 11

[Andreas Kohn]

Reproduces with Build identifier: Mozilla/5.0 (X11; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0 (http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-10-28-03-02-05-mozilla-central/). 

Trying with a release in the evening, as soon as I figure out how to make it restart in the same way (is there a button somewhere to do a 'restart' without waiting for an update?)

Comment 12

[:Gijs (he/him)]

(In reply to Andreas Kohn from comment #11)
> Reproduces with Build identifier: Mozilla/5.0 (X11; Linux x86_64; rv:27.0)
> Gecko/20100101 Firefox/27.0
> (http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-10-28-03-02-05-
> mozilla-central/). 
> 
> Trying with a release in the evening, as soon as I figure out how to make it
> restart in the same way (is there a button somewhere to do a 'restart'
> without waiting for an update?)

No, but you can use nightlies from http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/ (and earlier years) - they should all offer an update to latest when you open the about window (Help > About Firefox).

I'd actually recommend starting with something pretty old, like from 2010, and then bisect from there. We have a tool to do this called mozregression, but because this bug requires restarting the browser, I suspect it won't work well.

This would also be a good time to doublecheck - are you seeing this with a clean profile (or at least without add-ons installed) ?

Comment 13

[Paul Silaghi, QA [:pauly]]

> This would also be a good time to doublecheck - are you seeing this with a
> clean profile (or at least without add-ons installed) ?
http://support.mozilla.com/kb/Safe+Mode
http://support.mozilla.org/en-US/kb/Managing-profiles#w_starting-the-profile-manager

Comment 14

[Robert Strong (they/them - no direct email)]

(In reply to Paul Silaghi, QA [:pauly] from comment #13)
Pau, application update has nothing to do with these files and as noted in comment #5 this is likely caused by media cache so moving back to untriaged so it can be triaged properly.

Comment 15

[Robert Strong (they/them - no direct email)]

Note: you can also install or uninstall an extension that is not restartless to perform a restart.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

No significant activity here for almost 2 years now and a lot of cache changes in the mean time. I'm going to close this as incomplete, but feel free to reopen it if it still reproduces and there's new information to report.