752732 - Recover from -wal and -shm files being directories in OpenUnsharedDatabase error handling logic

Status: RESOLVED INACTIVE

(Toolkit :: Storage, enhancement)

Description

[Andrew Sutherland [:asuth] (he/him)]

In bug 750781 in Thunderbird, it was discovered that sometimes, somehow, (probably involving a seriously messed up anti-virus software), the Places -wal and -shm files end up as directories.  In versions of SQLite prior to 3.7.12 (not yet released), error handling retry code would burn a lot of cycles trying to wait out a virus checker because it thought the problem was that the files were actual files but being touched by some other process.  SQLite 3.7.12 will detect the file is a directory and fail immediately.

It is desirable to clean up these messed up directories, and to do so without introducing any file stats in normal operation.  Happily, it appears we should be able to detect the error in OpenUnsharedDatabase only when it's a problem, and then detect and delete these rogue folders.  Since the problem is theorized to be bad anti-virus software, and that can happen to anyone, mozStorage seems like the appropriate location to deal with this.

A realistic test would likely need to create a new database, invoke "PRAGMA journal_mode=WAL", close the database, create the bogus directories, then re-open the database at which point the error cascade should be triggered and we verify success by a successful open and the directories disappearing.  The fix would want to land now, but the test would want to hang out on the tracker bug for planning to land when we take SQLite 3.7.12 or higher.  (The fix is beneficial even with the current version of SQLite, since it will at least avoid the problem from recurring on next startup or next time places tries to initialize itself, etc.)

Comment 1

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

(In reply to Andrew Sutherland (:asuth) from comment #0)
> In bug 750781 in Thunderbird, it was discovered that sometimes, somehow,
> (probably involving a seriously messed up anti-virus software), the Places
> -wal and -shm files end up as directories.

Ugh! Looks like this happened on Windows right?

> Since the
> problem is theorized to be bad anti-virus software, and that can happen to
> anyone, mozStorage seems like the appropriate location to deal with this.

Any idea which antivirus software? This looks quite bad, and yeah, it may be bad for cookies too.

I'm going to file the bug for the SQLite 3.7.12 upgrade.

Comment 2

[David :Bienvenu]

(In reply to Marco Bonardo [:mak] from comment #1)
 
> Ugh! Looks like this happened on Windows right?
yes, we've only heard about Windows, but 90% of our users are on Windows.
> 
> 
> Any idea which antivirus software? This looks quite bad, and yeah, it may be
> bad for cookies too.

We've heard about ESET NOD32 and Avira Antivir, though it seems unlikely that two different anti-virus programs would have the same issue.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

New information on bug 750781 suggests this might be Thunderbird running amok when badly configured to view the root of the profile dir as a local folders store.  Since that's a more comprehensive problem that probably justifies a new profile, I'm deferring writing a fix for this for now.  If it turns out after more investigation (by other people :) there is rogue software other than Thunderbird itself at work, I'll provide the fix, otherwise I'll close the bug.

Comment 4

[richard]

12-05-22 11:10 EDT USA. I'm having similar problems with TB. It goes into "Not Responding" if it isn't given 30 to 50 seconds to boot up; if you click on a message while it is searching for the server, it goes grey and you can wait anywhere from 30 seconds to 10 minutes for it to become "live" again. Once it's live again, clicking on a message gets you a blank screen (not grey) and in anywhere from 10 seconds to 2 minutes, the message and any attachments appear. Replies seem more normal. The directory tree I established has been corrupted with some favorite folders immovable now at the bottom of the tree, not at the top. I searched for and found some. ".wal" and ".shm" files, (not Directories) and deleted them. I did not succeed in finding any such Directories and I don't have "OpenUnsharedDataBase" available to me, nor would I know how to use it, if I did. I disabled my Norton 360 AV, the only AV I have. I've uninstalled and reinstalled 12.0.1 twice. Nothing has helped. I'm online with Broadband at 5mbps; w7 64 pro; Intel i7 with 8G ram running at 160Ghz. The slowdown, began 5/19/12.

Comment 5

[Mark Banner (:standard8)]

(In reply to richard from comment #4)
> The directory tree I
> established has been corrupted with some favorite folders immovable now at
> the bottom of the tree, not at the top. I searched for and found some.
> ".wal" and ".shm" files, (not Directories) and deleted them. I did not
> succeed in finding any such Directories

Richard, if you're not seeing those directories, then it isn't this bug that is affecting you, please ask for support on http://getsatisfaction.com/mozilla_messaging/ or file a new bug specific to your issue. Thanks.

Comment 6

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

(In reply to Andrew Sutherland (:asuth) from comment #3)
> New information on bug 750781 suggests this might be Thunderbird running
> amok when badly configured to view the root of the profile dir as a local
> folders store.

could you point me to the relevant comment there? I can't see how TB may transform a file into a folder...

Comment 7

[Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland]

(In reply to Marco Bonardo [:mak] from comment #6)
> (In reply to Andrew Sutherland (:asuth) from comment #3)
> > New information on bug 750781 suggests this might be Thunderbird running
> > amok when badly configured to view the root of the profile dir as a local
> > folders store.
> 
> could you point me to the relevant comment there? I can't see how TB may
> transform a file into a folder...

hi marco:
I believe you are looking for: "have pointed your Local directory (in the account settings -> server settings) to your profile folder (which you should never do)" from:
https://bugzilla.mozilla.org/show_bug.cgi?id=750781#c24

Comment 8

[richard]

I've achieved a full resolution of this problem.

I deleted the following Files AS WELL AS THE FOLDERS WITH THE SAME NAMES, and have restored full functionality of Thunderbird: (All quote marks added by me.)
   "places.sqlite-shm" and "places.sqlite-wal"
   "cookies.sqlite-shm" and "cookies.sqlite-wal".
   I also deleted the file "prefs.js".
TB no longer hangs, I do not get the "Not Responding" message and I can click on links in email messages to go to sites. Speed of TB responses is back to normal in all respects.

Comment 9

[BMO Automation]

Per policy at https://wiki.mozilla.org/Bug_Triage/Projects/Bug_Handling/Bug_Husbandry#Inactive_Bugs. If this bug is not an enhancement request or a bug not present in a supported release of Firefox, then it may be reopened.

Comment 10

[Wayne Mery (:wsmwk)]

I assume this still exists - I'll retest to make certain