431065 - Repeated use of safe file output streams conflicts with file monitoring tools

Status: RESOLVED FIXED

(Toolkit :: Add-ons Manager, defect)

Description

[Pardal Freudenthal (:ShareBird)]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.15pre) Gecko/20080416 BonEcho/2.0.0.15pre
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9pre) Gecko/2008042606 Minefield/3.0pre

If you delete the extensions.rdf file in the profile folder, EM will not rebuild it consistently if you have more than 3 addons installed. Sometimes will not show any extensions, sometimes will show all or partially extensions which will be installed on restart.

Reproducible: Always

Steps to Reproduce:
1. Have at least 4 addons installed.
2. Close the app.
3. Delete the file extensions.rdf in your profile folder.
4. Restart the app
Actual Results:  
Inconsistent behavior at registering extensions.

Expected Results:  
Rebuild the extensions.rdf file nicely.

It works fine with Firefox 2.0.X. I can't figure out when it's stopped to work. Build Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b2pre) Gecko/2007111204 Minefield/3.0b2pre doesn't work...

Comment 1

[Dave Townsend [:mossop]]

It is looking increasingly like something has changed recently that is causing errors during the extension manager startup. In the past 5 days I have heard 3 reports of this on IRC (compared to essentially none previously) of failures either in nsISafeOutputStream.finish (for extensions.cache) or nsIRDFRemoteDataSource.Flush (for the main extensions.rdf, which uses an nsISafeOutputStream internally). Both are access denied errors and I believe will be getting thrown from attempting to overwrite the actual file with the temporary copy already written out. Likely this is blocked when the actual file is already open for reading.

In bug 430579 this appeared to be caused by Avira AVGuard. In the second instance it appeared to be caused by AVG. In this case there hasn't been a determined cause.

In my tests I'm unable to reproduce this in a clean Windows XP SP2 environment with some 50 extensions installed.

Comment 2

[Dave Townsend [:mossop]]

Tony, do QA have machines with popular antivirus tools installed on? STR should be:

1. Install a lot of extensions.
2. Restart etc. to make sure they were installed correctly.
3. Exit and delete extensions.rdf.
4. Start Firefox and check all the extensions are working and dont have "needs to restart" against them in the extension manager.

Comment 3

[Pardal Freudenthal (:ShareBird)]

So I've made some more tests and I still having issues in this.

At first I've unloaded the anti-virus service (Symantec Antivirus Corporate). And then:

1. I've backed up my extensions folder with 21 extensions in it.
2. Created a new profile
3. Started the browser
4. Closed the browser
5. Moved the 21 folders to inside the new extensions folder
6. Restarted the browser

Result:

1. Just a couple extensions are listed on Addons Manager as need-install (7)
2. I got this message on Error Console:

Error: [Exception... "Component returned failure code: 0x80520015 (NS_ERROR_FILE_ACCESS_DENIED) [nsIRDFRemoteDataSource.Flush]"  nsresult: "0x80520015 (NS_ERROR_FILE_ACCESS_DENIED)"  location: "JS frame :: file:///D:/Programme/Mozilla/Firefox/3.0/en_US/Firefox/components/nsExtensionManager.js :: anonymous :: line 8663"  data: no]
Source File: file:///D:/Programme/Mozilla/Firefox/3.0/en_US/Firefox/components/nsExtensionManager.js
Line: 8663

All the 21 folders still there, but just 7 are listed on Addons Manager

Than I did:

1. Delete all extension related files on profile.
2. Restarted the browser

Result

1. Just 5 extensions are listed on Addons Manager as need-install (will be installed when restarted)
2. Error Console shows the same message

To make things work properly I needed to:

1. Delete all folders inside extensions folder.
2. Restart the browser
3. Close the browser
4. Delete some empty folders which were created inside extensions folder (the folders were named as extensions IDs but were empty)
5. Restart and close the browser
6. Copy just three extensions to inside extensions folder
7. Restart the browser
8. The three extensions are working properly and listed as installed (no need-install)
9. Close the browser and move three more extensions
10. Restart the browser and verify all six extensions are working properly, no messages on Error Console
11. And so on...

After this, if I delete extensions.rdf file things doesn't work anymore (just a couple extensions listed, all as need-install).

If I have just three extensions installed, everything work as expected, I can delete the extensions.rdf file and my 3 extensions will be registered and will work properly. If I have just one more, the same uggly behavior.

I'm seeing on our Forum a couple similar reports so I'm afraid we're running into big issues after release...

Comment 4

[Pardal Freudenthal (:ShareBird)]

So. After a few tests at this weekend I've figured out that my problems are two: Symantec AntiVirus and TortoiseSVN's feature Icons Overlay. If I disable the AntiVirus Programm AND this feature from TortoiseSVN I'm not able to reproduce the bug anymore. All works properly.

OK. I can deal with this since I can configure my programs with a couple exception rules. But what about the average user? 

Comment 5

[Dave Townsend [:mossop]]

You have listed both Firefox 2 and 3 in the initial comment. Is this a problem with both?

Comment 6

[Pardal Freudenthal (:ShareBird)]

(In reply to comment #5)
> You have listed both Firefox 2 and 3 in the initial comment. Is this a problem
> with both?
> 
No Mossop, it only affects Firefox 3.0. Sorry, I don't know why Firefox 2.0 was listed on my report...

Comment 7

[Dave Townsend [:mossop]]

I can now reproduce this fairly reliably on 2.0, 3.0 and trunk builds. I have file access traces confirming that things fail when the safe file output stream attempts to overwrite the real file with the temporary copy when the real file is being accessed by an external process.

We see it badly because during registration of a new add-on at startup we flush the datasource and other caches over and over again for every tiny change (bug 401229). Most of the file monitoring tools spot when a file has been written to and closed and then open the file to process it. The more often we write out an entire copy of the file the more likely the next time we try to write it the external process will still have it open.

Reports of this have increased significantly for Firefox 3 (ie. we have accurate reports of the failure where before we didn't), however I can still reproduce on Firefox 2 although not as easily. I believe that the changes in bug 375778 have made our IO perform differently and more likely to hit problems with external processes.

One potential option is simply to remove the use of safe file output streams, though that leaves the potential of corrupt files to deal with.

Comment 11

[Luke Boyling]

Is there a temporary fix for advanced users until this bug is sorted out?

Comment 13

[Anton van Bohemen]

This is definately a problem with several anti-virus and/or anti-spyware products. I have (actually had) Trend Micro Internet Security 2008 installed, and I could not install one single extension. It caused the exact same symptoms as in bug 437914 which was duped against this one. Now I'm using McAfee Virusscan Enterprise with Antispyware module 8.5, and this scanner doesn't have this problem.

I also started this topic on the forums: http://forums.mozillazine.org/viewtopic.php?f=23&t=682115

Comment 16

[Mac Franks]

Avira would warn on the multiple file syndrome but I now run PCtools Spyware with Antivirus and it has no problems.  They may have some specific code that deals with this specifically for FF 3.  You might ask them what they did.
 

Comment 17

[Bobby Dennett]

Short of switching/disabling antivirus software, is there any way to workaround this issue?

Comment 19

[Pardal Freudenthal (:ShareBird)]

Just as feedback: I'm not getting any problems since the last updates from TortoiseSVN.

Comment 20

[Dave Townsend [:mossop]]

The new extension manager uses just one safefileoutputstream to write extensions.ini one time so I think this should be fixed.