Last Comment Bug 750781 - Thunderbird 12 veryyyyy slow when opening Message. Sometimes shows "Not Responding". (because places.sqlite-wal/-shm is created as mail directory by Tb if Tb's profile directory is specified by user as Local Directory: of an account)
: Thunderbird 12 veryyyyy slow when opening Message. Sometimes shows "Not Respo...
Status: RESOLVED FIXED
[gs][
: addon-compat, dataloss, perf, ux-error-prevention
Product: MailNews Core
Classification: Components
Component: Account Manager (show other bugs)
: Trunk
: All All
: -- critical with 1 vote (vote)
: Thunderbird 21.0
Assigned To: :aceman
:
:
Mentors:
https://getsatisfaction.com/mozilla_m...
: 752749 755738 758118 (view as bug list)
Depends on: 752732 946876 577775 848477 921371
Blocks:
  Show dependency treegraph
 
Reported: 2012-05-01 10:08 PDT by Aurélien
Modified: 2014-08-05 14:54 PDT (History)
22 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
-


Attachments
Stranges folders are highlighted (65.02 KB, image/pjpeg)
2012-05-05 10:43 PDT, Aurélien
no flags Details
Process Monitor Log File (71.81 KB, text/plain)
2012-05-12 02:22 PDT, Aurélien
no flags Details
patch (7.65 KB, patch)
2012-07-01 12:42 PDT, :aceman
bwinton: ui‑review+
mozilla: feedback+
iann_bugzilla: feedback-
vseerror: feedback+
Details | Diff | Splinter Review
patch v2 (7.52 KB, patch)
2012-09-27 12:37 PDT, :aceman
iann_bugzilla: review+
Details | Diff | Splinter Review
patch v3 (8.49 KB, patch)
2012-11-06 13:20 PST, :aceman
no flags Details | Diff | Splinter Review
patch v4 (8.55 KB, patch)
2012-11-06 13:30 PST, :aceman
no flags Details | Diff | Splinter Review
patch v5 (8.66 KB, patch)
2012-11-06 13:33 PST, :aceman
no flags Details | Diff | Splinter Review
patch v6 [bitrotted] (9.75 KB, patch)
2012-11-11 12:20 PST, :aceman
bwinton: ui‑review+
Details | Diff | Splinter Review
patch v7 (9.70 KB, patch)
2012-12-06 11:45 PST, :aceman
mkmelin+mozilla: review+
Details | Diff | Splinter Review

Description Aurélien 2012-05-01 10:08:51 PDT
User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; SIMBAR={AE11A89F-8F1A-11DF-A683-0022156856AF}; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 1.1.4322; .NET4.0C; .NET4.0E; Creative AutoUpdate v1.10.10)

Steps to reproduce:

Only updating from Thunderbird 11 to Thunderbird 12


Actual results:

When I click on any message, TB 12 freezes and I have to wait several seconds, sometimes several minutes for the message to open. Heavier the message is, longer it is...

It seems the matter is more general than message opening because when I try to open the "Address book" or the "Get Addons" page it's very slow too. (but other menus are quick to be displayed)

I go back to Thunderbird 11 and it works perfectly. The matter is only with Thunderbird 12 (I use Thunderbird since the 1st release)


OS : Windows XP SP3.
JVM : 1.6.0_31


Expected results:

It should be open messages instantly when click, like in previous release.
Comment 1 Hashem Masoud 2012-05-02 22:56:34 PDT
(In reply to Aurélien from comment #0)
And the message you try to open is local or on IMAP server?
Comment 2 :aceman 2012-05-03 00:23:03 PDT
Can you see if there is CPU usage by Thunderbird in the period where it is "frozen"?
Comment 3 Aurélien 2012-05-03 11:52:03 PDT
Hello !

It's exactly the same on imap server or if it's a classic downloaded POP message.

But like I said, it's not only with Mail, but it seems to be the case with HTML menu like the "Get Addons" page, or even with the Address book.

During the freeze period, there is no CPU Usage !

More characters has the mail, address book or get addons, longer the freeze is.
Comment 4 Aurélien 2012-05-03 11:58:11 PDT
and it the same with Thunderbird 12.0 and 12.0.1 but it works perfectly with TB 11.0.

I tried to delete .msf file, to archive and compact all message. I tried too to delete every mail of my mailbox, but it was the same.
Comment 5 David :Bienvenu 2012-05-04 08:39:56 PDT
Yes, this really sounds like some external program/process interfering with Thunderbird. Sometimes these programs whitelist some applications, and when new versions come out, the whitelisting stops working.

The only other thing I could suggest is creating a new profile and setting up your e-mail accounts again to see if it's something specific to your profile. Running Thunderbird with the -profilemanager command line option allows you to create a new profile.
Comment 6 Joe Sabash [:JoeS1] 2012-05-04 14:35:10 PDT
(In reply to David :Bienvenu from comment #5)
> Yes, this really sounds like some external program/process interfering with
> Thunderbird. Sometimes these programs whitelist some applications, and when
> new versions come out, the whitelisting stops working.
> 

Any clues on how to troubleshoot these sort of program conflicts.
I resolved one of these myself, just by observing cpu usage in the win task manager.
In my case it was Updater.exe installed as an option to Avira Antivir.
I believe there is a common program name in the TB package.
Had something to do with a ask.com search toolbar.

This had a very specific affect on TB only. Other apps seemed to run fine.
CPU nearly pegged but only when TB was running.
Comment 7 Aurélien 2012-05-04 15:48:28 PDT
We advance !!!

I read carefully all your posts ! I test all you said.

So, for information, like Joe, I'm an Avira Antivir user. So I have test to uninstall it but it didn't work. (I try too, to uninstall my Zone Alarm Firewall but no success too)... Task Manager show me that no processus are used during freeze. Process Explorer confirm me that.

So I have test the David advice. I ran TB with -profilemanager command line and... it works ! No Freeze !!! I have test to copy all mails from old profile to new profile, and it works fine too !!

So the matter is linked to my old account... I didn't delete it for the moment,  in the case of you want I take a look to some configuration of this account.

Thanks everyone for your help !
Comment 8 David :Bienvenu 2012-05-04 18:34:56 PDT
(In reply to Aurélien from comment #7)

> So the matter is linked to my old account... I didn't delete it for the
> moment,  in the case of you want I take a look to some configuration of this
> account.

There are a few possibilities as to why a new profile doesn't show the issue. One is some pref that's different in the old profile, or some extension that's not in the new profile. An other possibility is some data file in the old profile has gotten out of control. I would look for a file at the top of the old profile directory that seems unusually large. Most of the files can be deleted safely, other than prefs.js, of course, (panacea.dat, the sqlite files, etc). You could try saving off the files from the top of your profile dir to a different directory and see if removing any of them has an effect. Moving the address book files away (the ones that end in .mab) might also be interesting.

I should stress that I don't have any particular reason to think one of the files is responsible; I'm just trying to trouble-shoot.
Comment 9 Aurélien 2012-05-05 00:00:04 PDT
Hello David !!!!

Now I find what the problem is due ! I have test to move some file from my older and it seems the matter is due to 2 folders that I have in the old profile :
-places.sqlite-wal
-places.sqlite-shm

These 2 folders are empty. When they are not in the profile, everything is normal and quick !!

So... I was going in my new profile folder and create 2 empty folder with the same name. And... Now it freeze now !

Could you make the test on your thunderbird to be sure it's not only with my Thunderbird. 
What are these 2 files ? Why do they slow down only the 12 releases of thunderbird, and not the TB 11 ?

Thank you for your time you spent on my problem !
Comment 10 David :Bienvenu 2012-05-05 07:20:01 PDT
places.sqlite is a sqlite database we use to keep track of the urls you've visited. It's mainly a firefox thing, but we have it as well because we use some pieces of code shared with Firefox that require it. the sqlite-wal and sqlite-shm files are helper files that sqlite uses while Thunderbird is running - the main database is just places.sqlite. When you shut down Thunderbird, do those files go away?

I really have no idea why they would make TB slow for you, per se but we can try to figure it out.
Comment 11 Aurélien 2012-05-05 08:00:56 PDT
I think there is an understanding. I was not clear in my explanation. 
Yes I have 2 FILES places.sqlite-wal and places.sqlite-shm but they are not the problem. 
The problem was due to 2 empty FOLDERS called places.sqlite-wal and places.sqlite-shm !

When I deleted this 2 FOLDERS, freezes have disappeared. They are not be re-created now, so the problem is corrected, but I'm curious to know why I'm the only person to have this problem !

-Is it possible for you to make the test to create in your profile folder 2 empty folders called places.sqlite-wal and places.sqlite-shm and verify if thunderbird is slow down for you too ?
-In your opinion, why this 2 folders was create in my profile ?

I have no extension installed. (Maybe in the past, I have installed Lightin, but I'm not sure on this computer I did it)
Comment 12 David :Bienvenu 2012-05-05 09:22:03 PDT
Oh, folders! Where exactly were these folders in your user profile dir?
Comment 13 Aurélien 2012-05-05 09:25:54 PDT
The 2 folders are at the root of my profile. 

So on my windows XP : C:\Documents and Settings\Administrateur\Application Data\Thunderbird\Profiles\myprofilename.default
Comment 14 David :Bienvenu 2012-05-05 09:39:45 PDT
OK, I'm a bit confused - that's the directory where those files should be, and they should be files, not folders. By folders, do you mean those two files are directories, containing other folders? If that were the case, I don't see how you could have both files and folders with the same name. Do they show up as folders in the Thunderbird UI?
Comment 15 :aceman 2012-05-05 09:54:28 PDT
David, maybe that is the problem? He has folders with those 2 names and when TB tries to show a message it accesses places and sqlite tries to create those 2 files for the places databese and can't as the names are taken by those folders. So it hangs for a long time?
Comment 16 Aurélien 2012-05-05 10:00:39 PDT
Ok, so your last post help me to understand the problem.

In fact, I had the 2 folders with this name. 
Normally, when Thunderbird is launched, it creates 2 files with theses names. But like I had this 2 folders, it didn't create the 2 files (as you implied !). I suppose (But I know nothing at all !) that when a mail is open, TB try to search some information in theses files (maybe for each character in the mail like the freeze was depending of the mail size ?).

About your last question : "Do they show up as folders in the Thunderbird UI?" I'm not sure to understand. The TB tree structure  don't show them because folders are not in Mail folder but in the root of the profile folder.

So, why I had 2 folders here ? What happened to cause this problem ? Why Only TB 12 was slow down with this 2 folder (and not with TB 11 !)
Comment 17 Aurélien 2012-05-05 10:05:54 PDT
aceman and David, can you made this simple test :
1)Close your thunderbird. 

2)Cerate 2 folders at the root of your profile with this name :
-places.sqlite-wal
-places.sqlite-shm

3)Re-launch Thunderbird and try to open mails and address book

Is it slower for you too ?
Comment 18 :aceman 2012-05-05 10:10:23 PDT
(In reply to Aurélien from comment #16)
> In fact, I had the 2 folders with this name. 
> Normally, when Thunderbird is launched, it creates 2 files with theses
> names. But like I had this 2 folders, it didn't create the 2 files (as you
> implied !). I suppose (But I know nothing at all !) that when a mail is
> open, TB try to search some information in theses files (maybe for each
> character in the mail like the freeze was depending of the mail size ?).
The places.sqlite database (to which those 2 files belong too) is used to store your browsing history (even in TB). If your emails have links in them (HTML) the database can be queried if you already accessed those links in the past so they can be marked 'visited'. If the database does not respond (because it can't create those 2 files) then TB could hang.
That is what I think could be a very imaginable explanation. But can't test it right now.
Comment 19 David :Bienvenu 2012-05-05 10:28:26 PDT
(In reply to Aurélien from comment #17)
> aceman and David, can you made this simple test :
> 1)Close your thunderbird. 
> 
> 2)Cerate 2 folders at the root of your profile with this name :
> -places.sqlite-wal
> -places.sqlite-shm

Do the folders really have "-" as the leading character? That's odd, but it explains why you could have the files and the folders, because the folder names are slightly different.
Comment 20 Wayne Mery (:wsmwk, NI for questions) 2012-05-05 10:33:31 PDT
(In reply to Aurélien from comment #17)
> Is it slower for you too ?

I can reproduce. It's horrible. Lots of "not responding" doing almost anything. Not always on every click, but a high percentage.  I haven't attempted to determine frequency.


Aurélien, do you ever see "not responding" in windows title bar?
Comment 21 Aurélien 2012-05-05 10:39:01 PDT
(In reply to David :Bienvenu from comment #19)
> Do the folders really have "-" as the leading character?
> That's odd, but it explains why you could have the files and the folders,
> because the folder names are slightly different.

No, no, without the "-" as the leading character. They are named "places.sqlite-wal" and "places.sqlite-shm" like files

(In reply to Wayne Mery (:wsmwk) from comment #20)
> Aurélien, do you ever see "not responding" in windows
> title bar?
Not ever, but sometime yes !
Comment 22 Aurélien 2012-05-05 10:43:22 PDT
Created attachment 621321 [details]
Stranges folders are highlighted
Comment 23 Aurélien 2012-05-05 10:45:36 PDT
I put in attachment a printscreen of all folders at the root of my profile directory. 

I highlight each "strange" folders I have in my old profile...
Comment 24 :aceman 2012-05-05 11:15:54 PDT
Those marked folders are realz strange. The folders with .sbd extension are created automatically if you have a mail folder with that name and also some subfolder inside it. It looks like your profile and the Mail folder somehow mixed themselves. OR you have pointed your Local directory (in the account settings -> server settings) to your profile folder (which you should never do).
Comment 25 Wayne Mery (:wsmwk, NI for questions) 2012-05-05 11:26:31 PDT
also reported at http://forums.mozillazine.org/viewtopic.php?f=39&t=2466065&p=11965145&e=11965145
Comment 26 Aurélien 2012-05-05 11:42:00 PDT
Thanks Wayne, So I was not alone in this case ^^

About you said aceman, it's possible that long times ago, I did a mishandling which cause the folder bug... Difficult to remember about that !
Comment 27 David :Bienvenu 2012-05-05 13:06:00 PDT
Here's a stack trace that we continually get stuck in once the profile has been somehow horked in this way. We could add code somewhere to check if these places.sqlite-wal and places.sqlite-shm directories (as opposed to files) exist before opening the db, and clear them out - I'm not sure where that code should go...but perhaps the sqlite gurus would have a better idea.

 	mozsqlite3.dll!retryIoerr(int * pnRetry, unsigned long * pError)  Line 33307	C
>	mozsqlite3.dll!winOpen(sqlite3_vfs * pVfs, const char * zName, sqlite3_file * id, int flags, int * pOutFlags)  Line 35263 + 0x4f bytes	C
 	mozsqlite3.dll!winOpen(sqlite3_vfs * pVfs, const char * zName, sqlite3_file * id, int flags, int * pOutFlags)  Line 35292 + 0x1f bytes	C
 	xul.dll!`anonymous namespace'::xOpen(sqlite3_vfs * vfs, const char * zName, sqlite3_file * pFile, int flags, int * pOutFlags)  Line 326 + 0x1f bytes	C++
 	mozsqlite3.dll!sqlite3OsOpen(sqlite3_vfs * pVfs, const char * zPath, sqlite3_file * pFile, int flags, int * pFlagsOut)  Line 14829 + 0x22 bytes	C
 	mozsqlite3.dll!sqlite3WalOpen(sqlite3_vfs * pVfs, sqlite3_file * pDbFd, const char * zWalName, int bNoShm, __int64 mxWalSize, Wal * * ppWal)  Line 46629 + 0x1c bytes	C
 	mozsqlite3.dll!pagerOpenWal(Pager * pPager)  Line 45203 + 0x3e bytes	C
 	mozsqlite3.dll!sqlite3PagerOpenWal(Pager * pPager, int * pbOpen)  Line 45243 + 0x9 bytes	C
 	mozsqlite3.dll!lockBtree(BtShared * pBt)  Line 51719 + 0xf bytes	C
 	mozsqlite3.dll!sqlite3BtreeBeginTrans(Btree * p, int wrflag)  Line 51972 + 0x12 bytes	C
 	mozsqlite3.dll!sqlite3InitOne(sqlite3 * db, int iDb, char * * pzErrMsg)  Line 28324 + 0xe bytes	C
 	mozsqlite3.dll!sqlite3Init(sqlite3 * db, char * * pzErrMsg)  Line 28496 + 0x11 bytes	C
 	mozsqlite3.dll!sqlite3ReadSchema(Parse * pParse)  Line 28534 + 0x10 bytes	C
 	mozsqlite3.dll!sqlite3Pragma(Parse * pParse, Token * pId1, Token * pId2, Token * pValue, int minusFlag)  Line 27258 + 0x9 bytes	C
 	mozsqlite3.dll!yy_reduce(yyParser * yypParser, int yyruleno)  Line 45238 + 0x20 bytes	C
 	mozsqlite3.dll!sqlite3Parser(void * yyp, int yymajor, Token yyminor, Parse * pParse)  Line 45622 + 0x12 bytes	C
 	mozsqlite3.dll!sqlite3RunParser(Parse * pParse, const char * zSql, char * * pzErrMsg)  Line 46458 + 0x20 bytes	C
 	mozsqlite3.dll!sqlite3Prepare(sqlite3 * db, const char * zSql, int nBytes, int saveSqlFlag, Vdbe * pReprepare, sqlite3_stmt * * ppStmt, const char * * pzTail)  Line 28710 + 0x11 bytes	C
 	mozsqlite3.dll!sqlite3LockAndPrepare(sqlite3 * db, const char * zSql, int nBytes, int saveSqlFlag, Vdbe * pOld, sqlite3_stmt * * ppStmt, const char * * pzTail)  Line 28802 + 0x21 bytes	C
 	mozsqlite3.dll!sqlite3_prepare(sqlite3 * db, const char * zSql, int nBytes, sqlite3_stmt * * ppStmt, const char * * pzTail)  Line 28866 + 0x1d bytes	C
 	mozsqlite3.dll!sqlite3_exec(sqlite3 * db, const char * zSql, int (void *, int, char * *, char * *)* xCallback, void * pArg, char * * pzErrMsg)  Line 25324 + 0x17 bytes	C
 	xul.dll!mozilla::storage::Connection::executeSql(const char * aSqlString)  Line 983 + 0x16 bytes	C++
 	xul.dll!mozilla::storage::Connection::initialize(nsIFile * aDatabaseFile, const char * aVFSName)  Line 680 + 0x17 bytes	C++
 	xul.dll!mozilla::storage::Service::OpenUnsharedDatabase(nsIFile * aDatabaseFile, mozIStorageConnection * * _connection)  Line 805 + 0x15 bytes	C++
 	xul.dll!mozilla::places::Database::InitDatabaseFile(nsCOMPtr<mozIStorageService> & aStorage, bool * aNewDatabaseCreated)  Line 524 + 0x3d bytes	C++
 	xul.dll!mozilla::places::Database::Init()  Line 427 + 0x10 bytes	C++
 	xul.dll!mozilla::places::Database::GetSingleton()  Line 377 + 0x87 bytes	C++
 	xul.dll!mozilla::places::Database::GetDatabase()  Line 125 + 0x5 bytes	C++
 	xul.dll!nsNavHistory::Init()  Line 340 + 0x9 bytes	C++
 	xul.dll!nsNavHistory::GetSingleton()  Line 299 + 0x87 bytes	C++
 	xul.dll!nsNavHistoryConstructor(nsISupports * aOuter, const nsID & aIID, void * * aResult)  Line 20 + 0x26 bytes	C++
 	xul.dll!mozilla::GenericFactory::CreateInstance(nsISupports * aOuter, const nsID & aIID, void * * aResult)  Line 48 + 0x14 bytes	C++
 	xul.dll!nsComponentManagerImpl::CreateInstanceByContractID(const char * aContractID, nsISupports * aDelegate, const nsID & aIID, void * * aResult)  Line 1064 + 0x25 bytes	C++
 	xul.dll!nsComponentManagerImpl::GetServiceByContractID(const char * aContractID, const nsID & aIID, void * * result)  Line 1466 + 0x34 bytes	C++
 	xul.dll!CallGetService(const char * aContractID, const nsID & aIID, void * * aResult)  Line 95	C++
 	xul.dll!nsGetServiceByContractID::operator()(const nsID & aIID, void * * aInstancePtr)  Line 278 + 0x13 bytes	C++
 	xul.dll!nsCOMPtr<nsINavHistoryService>::assign_from_gs_contractid(nsGetServiceByContractID gs, const nsID & aIID)  Line 1212 + 0xf bytes	C++
 	xul.dll!nsCOMPtr<nsINavHistoryService>::nsCOMPtr<nsINavHistoryService>(nsGetServiceByContractID gs)  Line 616	C++
 	xul.dll!nsNavHistory::GetHistoryService()  Line 149	C++
 	xul.dll!mozilla::places::`anonymous namespace'::VisitedQuery::Start(nsIURI * aURI, mozIVisitedStatusCallback * aCallback)  Line 351 + 0x5 bytes	C++
 	xul.dll!mozilla::places::History::RegisterVisitedCallback(nsIURI * aURI, mozilla::dom::Link * aLink)  Line 1960 + 0xb bytes	C++
 	xul.dll!mozilla::dom::Link::LinkState()  Line 127 + 0x29 bytes	C++
 	xul.dll!EnumeratePendingLinkUpdates(nsPtrHashKey<mozilla::dom::Link> * aEntry, void * aData)  Line 8092 + 0x13 bytes	C++
 	xul.dll!nsTHashtable<nsPtrHashKey<mozilla::dom::Link> >::s_EnumStub(PLDHashTable * table, PLDHashEntryHdr * entry, unsigned int number, void * arg)  Line 500 + 0x12 bytes	C++
 	xul.dll!PL_DHashTableEnumerate(PLDHashTable * table, PLDHashOperator (PLDHashTable *, PLDHashEntryHdr *, unsigned int, void *)* etor, void * arg)  Line 750 + 0x19 bytes	C++
 	xul.dll!nsTHashtable<nsPtrHashKey<mozilla::dom::Link> >::EnumerateEntries(PLDHashOperator (nsPtrHashKey<mozilla::dom::Link> *, void *)* enumFunc, void * userArg)  Line 251 + 0x12 bytes	C++
 	xul.dll!nsIDocument::FlushPendingLinkUpdates()  Line 8104	C++
 	xul.dll!nsCSSFrameConstructor::ResolveStyleContext(nsStyleContext * aParentStyleContext, nsIContent * aContent, nsFrameConstructorState * aState)  Line 4553	C++
 	xul.dll!nsCSSFrameConstructor::ResolveStyleContext(nsIFrame * aParentFrame, nsIContent * aContent, nsFrameConstructorState * aState)  Line 4542 + 0x18 bytes	C++
 	xul.dll!nsCSSFrameConstructor::AddFrameConstructionItems(nsFrameConstructorState & aState, nsIContent * aContent, bool aSuppressWhiteSpaceOptimizations, nsIFrame * aParentFrame, nsCSSFrameConstructor::FrameConstructionItemList & aItems)  Line 4995 + 0x18 bytes	C++
 	xul.dll!nsCSSFrameConstructor::ContentAppended(nsIContent * aContainer, nsIContent * aFirstNewContent, bool aAllowLazyConstruction)  Line 6582	C++
Comment 28 David :Bienvenu 2012-05-05 14:51:17 PDT
I have to say, this really does make the app unusable...
Comment 29 :aceman 2012-05-06 03:08:27 PDT
Maybe the same problem would be visible in Firefox too. Can anybody try it?
Shouldn't we call some Places/sqlite people to it?

Maybe all the other sqlite databases have the same problem if their names are hijacked.

Maybe we should focus on finding out why those folders appeared.
Comment 30 Aurélien 2012-05-06 03:28:07 PDT
I have just make the test to create manually this 2 empty folders in the firefox profile, and yes, now it's veryyyyyyyy slow too.
Comment 31 :aceman 2012-05-06 05:51:22 PDT
Thanks for testing that.
Let's CC some people that seem to be involved in sqlite and Places.
Comment 32 David :Bienvenu 2012-05-06 06:53:55 PDT
Ugh, I thought I added sdwilsh earlier. FWIW, I doubt that Thunderbird-specific code created the directory places.sqlite-wal or -shm - there's no particular code that turns files into directories, and no reason it would just touch these two files.
Comment 33 Andrew Sutherland [:asuth] 2012-05-06 23:28:05 PDT
(summary for drh: if the WAL/SHM files are directories on windows, SQLite seems to freak out and make mozStorage-using apps unhappy, it's not clear how the files would end up directories.)

From my reading of the SQLite source, there are no calls to CreateDirectory, and the Windows API docs seem to explicitly state that CreateFile is incapable of creating a directory on its own (http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858%28v=vs.85%29.aspx).  Only our Telemetry VFS should apply to the places database, and it does not implement any directory creation, nor should VFS structure disagreements lead to misbehavior.  The SQLite locking code does do some directory magic, but I think that is only the unix-y implementations; at an off-site right now where the hotel internet is being destroyed by a bunch of mozillians and fossil is dying during the clone ("fossil: unknown command: [4cd23fa2fd38f49090d6eeb9b1e521d51d5]"), so it's hard to tell.

It seems most likely that either the virus checker was buggy at one point, an extension that tries to optimize firefox/thunderbird by doing something with places did something dumb, or some type of aggressive trojan/DLL call hooking program glitched its thunks quite badly, and left folders as a side-effect of file accesses gone wrong.

drh, any ideas on what could cause the WAL/SHM files to be created as directories?  More applicably, is this the type of thing SQLite wants to bulletproof itself against?  (Specifically, recovering when the files are directories, as opposed to Byzantine failure of file API calls.)
Comment 34 D. Richard Hipp 2012-05-07 04:36:03 PDT
(In reply to Andrew Sutherland (:asuth) from comment #33)
> (summary for drh: if the WAL/SHM files are directories on windows, SQLite
> seems to freak out and make mozStorage-using apps unhappy, it's not clear
> how the files would end up directories.)
> 
> From my reading of the SQLite source, there are no calls to CreateDirectory,
> and the Windows API docs seem to explicitly state that CreateFile is
> incapable of creating a directory on its own
> (http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858%28v=vs.
> 85%29.aspx).  Only our Telemetry VFS should apply to the places database,
> and it does not implement any directory creation, nor should VFS structure
> disagreements lead to misbehavior.  The SQLite locking code does do some
> directory magic, but I think that is only the unix-y implementations; at an
> off-site right now where the hotel internet is being destroyed by a bunch of
> mozillians and fossil is dying during the clone ("fossil: unknown command:
> [4cd23fa2fd38f49090d6eeb9b1e521d51d5]"), so it's hard to tell.
> 
> It seems most likely that either the virus checker was buggy at one point,
> an extension that tries to optimize firefox/thunderbird by doing something
> with places did something dumb, or some type of aggressive trojan/DLL call
> hooking program glitched its thunks quite badly, and left folders as a
> side-effect of file accesses gone wrong.
> 
> drh, any ideas on what could cause the WAL/SHM files to be created as
> directories?  More applicably, is this the type of thing SQLite wants to
> bulletproof itself against?  (Specifically, recovering when the files are
> directories, as opposed to Byzantine failure of file API calls.)

SQLite never creates directories (except on unix when you select the unix-dotfile VFS that uses dot-file locking instead of traditional file locking, in which case the lock files are really lock directories, but even then the names of the directories never correspond to WAL/SHM filenames.)

In our tests, manually creating directories with the same names as WAL/SHM files causes SQLite to fail to open with an SQLITE_CANTOPEN error.  Though, this failure is delayed by our anti-virus defense mechanism.  To try to work around AV software, if an CreateFile() system call fails, it is retried multiple times with the idea that some AV software might have the file open exclusively during the first try.  By default, these retries cease and the sqlite3_open_v2() request fails with SQLITE_CANTOPEN after about 1200 milliseconds.  But that delay can be adjusted using compile-time options SQLITE_WIN32_IOERR_RETRY and SQLITE_WIN32_IOERR_RETRY_DELAY or by run-time option SQLITE_FCNTL_WIN32_AV_RETRY with the sqlite3_config() interface.  Is Thunderbird increasing the AV timeout?  Might that be part of the problem?  What does Thunderbird do if it gets an SQLITE_CANTOPEN error on places?

We are looking into enhancing the AV-retry logic in SQLite so that it detects when the target file is really a directory and fails immediately rather than retrying for 1200 milliseconds.  Dunno if that will solve this problem, but is something that needs to be done nevertheless.
Comment 35 D. Richard Hipp 2012-05-07 06:39:48 PDT
I have changes in an SQLite branch which cause SQLite open attempts to fail immediately with an SQLITE_CANTOPEN error if the file it is trying to open is really a directory, rather than retrying the open request on the chance that it is AV software that is preventing the open.  The diffs can be seen at http://www.sqlite.org/src/info/03875633f4 and amalgamated source code file can be downloaded from http://www.sqlite.org/tmp/sqlite3-03875633f465e.zip (file may disappear after a few weeks).  Can somebody try out this new version of SQLite and see if it clears the problem?
Comment 36 David :Bienvenu 2012-05-07 07:14:21 PDT
(In reply to D. Richard Hipp from comment #35)
> I have changes in an SQLite branch which cause SQLite open attempts to fail
> immediately with an SQLITE_CANTOPEN error if the file it is trying to open
> is really a directory, rather than retrying the open request on the chance
> that it is AV software that is preventing the open.  The diffs can be seen
> at http://www.sqlite.org/src/info/03875633f4 and amalgamated source code
> file can be downloaded from
> http://www.sqlite.org/tmp/sqlite3-03875633f465e.zip (file may disappear
> after a few weeks).  Can somebody try out this new version of SQLite and see
> if it clears the problem?

Thx, yes, I can try it here.
Comment 37 David :Bienvenu 2012-05-07 09:05:43 PDT
Yes, the patch gets us back to reasonable performance in this situation, thx! asuth, sdwilsh, should we try to incorporate this patch a la carte into mozilla/db/sqlite, or take a new code drop somehow? And/Or try to clean up the directories ourselves?
Comment 38 Wayne Mery (:wsmwk, NI for questions) 2012-05-07 09:32:07 PDT
random reports still coming in. I have ~8 inquiries still open. These two confirm finding directory
https://getsatisfaction.com/mozilla_messaging/topics/not_responding-12utr9
https://getsatisfaction.com/mozilla_messaging/topics/thunderbird_problems-1glh5e
Comment 39 David :Bienvenu 2012-05-07 09:35:24 PDT
(In reply to Wayne Mery (:wsmwk) from comment #38)
> random reports still coming in. I have ~8 inquiries still open. These two
> confirm finding directory
> https://getsatisfaction.com/mozilla_messaging/topics/not_responding-12utr9
> https://getsatisfaction.com/mozilla_messaging/topics/thunderbird_problems-
> 1glh5e

it would be fascinating if they all had the same anti-virus program installed.
Comment 40 Wayne Mery (:wsmwk, NI for questions) 2012-05-07 10:08:41 PDT
confirmed reports are being tagged with bug# 
https://getsatisfaction.com/mozilla_messaging/tags/bug_750781
Comment 41 Andrew Sutherland [:asuth] 2012-05-07 13:37:19 PDT
(In reply to David :Bienvenu from comment #37)
> Yes, the patch gets us back to reasonable performance in this situation,
> thx! asuth, sdwilsh, should we try to incorporate this patch a la carte into
> mozilla/db/sqlite, or take a new code drop somehow? And/Or try to clean up
> the directories ourselves?

We usually only ship released versions of SQLite in the tree.

I think what we should do cleanup-wise depends on whether this has been observed for other databases too.  If it has happened to other databases too, mozStorage should check and fix things.  If it has only happened to Places (possibly because only places uses the write-ahead-log) but happened in FF and TB, then either places should fix the problem or mozStorage.  If it's just places in TB, then just TB.  A lot of the decision factor is that Taras will yell at us if we add additional file-stats to the startup path of Firefox :)

If SQLite decides to automatically try and cleanup the situation, we can leave it to SQLite, but this does seem like an application-level problem.  It's on us to provide the directory and pick filenames that don't collide, etc.
Comment 42 David :Bienvenu 2012-05-07 14:06:08 PDT
Richard, by any chance, was this a known issue with sqlite and other apps?

Asuth, cleanup could be hooked into the db open error handling code for Places (or Mozstorage), couldn't it? So we wouldn't be adding extra fstats for the non-error case.
Comment 43 D. Richard Hipp 2012-05-07 14:51:55 PDT
We have never before heard of anyone having trouble with *-wal and *-shm files being mysteriously changed into directories.  We are still pondering what, if anything, SQLite can and should do about it.

We will be committing changes so that the anti-virus retries are disabled when trying to open a directory as a file - the patch that fixes the delay.  SQLite version 3.7.12 will have these changes and should be out in a few weeks.  But apart from that, we are not sure what SQLite can or should do.  If an external process is messing with database files, what can SQLite do about that, really?

If anybody has any ideas as to why places.sqlite-wal and places.sqlite-shm were changed into directories, we would sure like to hear your theories.
Comment 44 David :Bienvenu 2012-05-07 15:13:55 PDT
(In reply to D. Richard Hipp from comment #43)
> 

> If anybody has any ideas as to why places.sqlite-wal and places.sqlite-shm
> were changed into directories, we would sure like to hear your theories.

The favorite theory is that a virus checker went a bit crazy - https://bugzilla.mozilla.org/show_bug.cgi?id=750781#c33 - I was just curious if you had heard other reports. We know that users who create a new profile don't have the problem re-appear (so far, at least), so it seems to have been something transitory.
Comment 45 Andrew Sutherland [:asuth] 2012-05-07 16:06:01 PDT
(In reply to David :Bienvenu from comment #42)
> Asuth, cleanup could be hooked into the db open error handling code for
> Places (or Mozstorage), couldn't it? So we wouldn't be adding extra fstats
> for the non-error case.

The stack you provided suggests that yes, mozStorage could indeed handle this in the slow-path failure case.  Specifically, although OpenUnsharedDatabase wouldn't actually know whether the database wants to be in WAL mode, it would not be unreasonable for us to probe for directories with the appropriately mangled names if the error code lines up with that type of failure.  We could then unlink and retry the open exactly once (rather than an infinite recursion of death.)

A realistic test would likely need to create a new database, invoke "PRAGMA journal_mode=WAL", close the database, create the bogus directories, then re-open the database at which point the error cascade should be triggered and we verify success by a successful open and the directories disappearing.  The fix would want to land now, but the test would want to hang out on the tracker bug for planning to land when we take SQLite 3.7.12 or higher.

bienvenu, since I'm going to be begging for some of your review time to provide feedback on my B2G IMAP hackery, I'm willing to do a criss-cross where I provide the patch.  This is a B2G work week, but I can get to it this weekend.  I'll file the bug now.
Comment 46 David :Bienvenu 2012-05-07 16:10:31 PDT
(In reply to Andrew Sutherland (:asuth) from comment #45)

> bienvenu, since I'm going to be begging for some of your review time to
> provide feedback on my B2G IMAP hackery

Cool, yes, I'm happy to do that. I keep meaning to find time to see if I can run b2g on windows in firefox...

> I'm willing to do a criss-cross
> where I provide the patch.  This is a B2G work week, but I can get to it
> this weekend.  I'll file the bug now.
Great, thx!
Comment 47 Andrew Sutherland [:asuth] 2012-05-07 16:22:29 PDT
(Thanks very much for your immediate and thorough support, D. Richard Hipp!  Since I think we all agree there is nothing more SQLite can do to address this issue, I'm un-CC'ing you to save you from further bug-spam.  Feel free to re-add yourself if you really do want to keep up with the bug.)
Comment 48 Wayne Mery (:wsmwk, NI for questions) 2012-05-07 16:52:38 PDT
ESET NOD32, currently version 4.2.71.2. - http://forums.mozillazine.org/viewtopic.php?f=39&t=2466065&sid=bc756f6881de1a797d8b7f07214e33f4&p=11970521#p11970521
Comment 49 Roland Tanglao :rolandtanglao 2012-05-07 22:17:19 PDT
(In reply to Wayne Mery (:wsmwk) from comment #48)
> ESET NOD32, currently version 4.2.71.2. -
> http://forums.mozillazine.org/viewtopic.
> php?f=39&t=2466065&sid=bc756f6881de1a797d8b7f07214e33f4&p=11970521#p11970521

So two known occurences:
1. Aurelien using Avira Antivir in this bug, comment 7 & 11: https://bugzilla.mozilla.org/show_bug.cgi?id=750781#c11 https://bugzilla.mozilla.org/show_bug.cgi?id=750781#c7
2. Alex Moss using ESET NOD32, currently version 4.2.71.2. http://forums.mozillazine.org/viewtopic.php?f=39&t=2466065&sid=bc756f6881de1a797d8b7f07214e33f4&p=11970521#p11970521
Comment 50 Roland Tanglao :rolandtanglao 2012-05-07 23:28:20 PDT
I've created a Company Update on Get Satisfaction get input from users to see if it really is just Avira and ESET NOD32 (perhaps it's all anti-virus or perhaps AV is just a red herring!):

https://getsatisfaction.com/mozilla_messaging/topics/tb12_very_slow_when_opening_a_message_due_to_tb_places_sqlite_wal_and_shm_files_and_possibly_other_tb
Comment 51 Wayne Mery (:wsmwk, NI for questions) 2012-05-08 05:54:16 PDT
Trend Micro Titanium Internet Security
http://forums.mozillazine.org/viewtopic.php?f=39&t=2466065&p=11972135#p11972135
Comment 52 Wayne Mery (:wsmwk, NI for questions) 2012-05-08 06:12:46 PDT
(In reply to Wayne Mery (:wsmwk) from comment #51)
> Trend Micro Titanium Internet Security
> http://forums.mozillazine.org/viewtopic.
> php?f=39&t=2466065&p=11972135#p11972135

belay that - user doesn't have the offending directories - just otherwise crappy performance caused by AV
Comment 53 Ludovic Hirlimann [:Usul] 2012-05-11 00:03:47 PDT
*** Bug 752749 has been marked as a duplicate of this bug. ***
Comment 54 Aurélien 2012-05-11 12:13:45 PDT
Hello,

Hmmm... For information, Tonight, I have start my computer and my thunderbird... And... It's slow once again...
I take a look in my thunderbird application data : the two folders places.sqlite-shm and places.sqlite-wal have been re-created anymore.

It seems files have been re-created this morning. When I have closed my thunderbird. (Because the time created folder correspond to the time that I have closed my Thunderbird).

Some other files have been modified at this time (7h03 french time) :
-cert8.db
-folderTree.json
-key3.db
-prefs.js
-virtualFolders.dat

Earlier in the day (6h21) when I have start my Thunderbird, a file cookies.sqlite.bak has been created.
Comment 55 Aurélien 2012-05-11 12:23:47 PDT
The night before,

3 files have been modified

-blocklist.xml
-permissions.sqlite
-history.mab
Comment 56 Aurélien 2012-05-11 12:36:02 PDT
At 6h20 my windows live messenger have crashed. Si my windows "Event viewer" says : 
"La description pour l'ID d'événement ( 1000 ) dans la source (Windows Live Messenger) est introuvable. L'ordinateur local n'a peut-être pas les informations de Registre ou les librairies requises pour afficher les messages émanant d'un ordinateur distant. Vous pourrez peut-être utiliser l'option /AUXSOURCE= pour récupérer cette description. Reportez-vous aux rubriques Aide et support pour plus de détails. Les informations suivantes font partie de l'événement : msnmsgr.exe; 14.0.8117.416; 4bc935af; msnmsgr.exe; 14.0.8117.416; 4bc935af; 0; 0019c72f."

Sorry it's in french but I supprose that the english traduction is probably :


"
The description for Event ID ( 1000 ) in Source ( Windows Live Messenger ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: msnmsgr.exe; 14.0.8117.416; 4bc935af; msnmsgr.exe; 14.0.8117.416; 4bc935af; 0; 0019c72f."

It's the only Error message I have in my Event Viewer since 2 days
Comment 57 Aurélien 2012-05-11 15:17:20 PDT
I think I find how the 2 directories are created !

I have made this test : 
-Create two empty FILE called places.sqlite-wal and places.sqlite-shm
-Launch Thunderbird
-Close Thunderbird

After closing, the 2 files have been deleted and replace by 2 folder with the same name !

If the 2 FILES places.sqlite-shm and places.sqlite-wal exists before the Thunderbird launch, when you will close your thunderbird, the 2 FILES will be transform to 2 FOLDERS !

Can anyone can confirm this on his computer ?

So I suppose that if thunderbird does'nt close properly, the problem can be occured.

Maybe it can help you to find the problem in the sourcecode. 

For additionnal information, I have made the same test in my firefox profile directory, and the problem is not occured. I have created the 2 files manually before the firefox launch, they are correctly deleted after the closing of firefox and not replace by 2 folders unlike that with thunderbird.
Comment 58 WADA 2012-05-11 17:11:04 PDT
(In reply to Aurélien from comment #57)
> I have made this test : 
> -Create two empty FILE called places.sqlite-wal and places.sqlite-shm
> -Launch Thunderbird
> -Close Thunderbird
> After closing, the 2 files have been deleted and replace by 2 folder with
> the same name !
> If the 2 FILES places.sqlite-shm and places.sqlite-wal exists before the
> Thunderbird launch, when you will close your thunderbird, the 2 FILES will
> be transform to 2 FOLDERS !

I can't see "Transform to two FOLDERs(directory) of the 2 FILEs" with Tb 12 on MS Win-XP SP3.
Can you get Process Monitor log for "access to places.sqlite-wal & places.sqlite-shm"?
> http://technet.microsoft.com/en-us/sysinternals/bb545021
> http://technet.microsoft.com/en-us/sysinternals/bb896645
> Monitor of access to pleces relevant files/directories
>   Filter : If path contains "\places."(no quote), Include
> To limit sqlite-wal/sqlite-shm for places only.
>   Filter : If path contains "places.sqlite-wal"(no quote), Include
>            If path contains "places.sqlite-shm"(no quote), Include
Comment 59 Aurélien 2012-05-12 02:22:42 PDT
Created attachment 623402 [details]
Process Monitor Log File
Comment 60 Aurélien 2012-05-12 02:23:24 PDT
Hello everyone,

Thanks WADA for your answer ! I have followed your recommandations.

So, I launched Process Monitor with the filter If path contains "\places."(without quotes). I have created manually the 2 FILES "places.sqlite-wal" AND "places.sqlite-shm".

I have launched Thunderbird, selected a message, closed my thunderbird. I have executed a "dir" command on DOS for listing my profile directory that it confirm me that the 2 files have been replaced by directory, and immediately after, I have done an Export of the Process Monitor results. 

I put the .CSV log export of Process Monitor in attached file.

On this log, I understand it's that it's really the Thunderbird.exe process which replace the 2 files by 2 folders, isn't it ?

So... We can ask ourselves why I have the problem and not you...


In fact, I have made complementary test and it seems everything is due to one file : pref.js. When I delete this file, and replace by a new one, the problem disappear.

I think I have even found the line in my pref.js which cause the problem : 
user_pref("mail.server.server2.directory-rel", "[ProfD]");
When I deleted this line, the 2 files are deleted after closing thunderbird and NEVER replace by the 2 folders.

Does anyone have an idea ?
Comment 61 WADA 2012-05-12 03:56:51 PDT
(In reply to Aurélien from comment #59)
> Created attachment 623402 [details]
> Process Monitor Log File

(A) Following log is seen. (files are searched as local mail folder file)
> "10:52:37,8104645","thunderbird.exe","436","QueryOpen","C: ... \places.sqlite.msf","SUCCESS","CreationTime: 12/05/2012 01:33:26, LastAccessTime: 12/05/2012 01:33:26, LastWriteTime: 12/05/2012 01:33:26, ChangeTime: 12/05/2012 01:33:26, AllocationSize: 0, EndOfFile: 0, FileAttributes: A"
> "10:52:37,8105470","thunderbird.exe","436","QueryOpen","C: ... \places.sqlite","SUCCESS","CreationTime: 12/05/2012 01:27:21, LastAccessTime: 12/05/2012 10:49:16, LastWriteTime: 12/05/2012 01:32:23, ChangeTime: 12/05/2012 01:32:23, AllocationSize: 10 485 760, EndOfFile: 10 485 760, FileAttributes: A"
> "10:52:37,8106704","thunderbird.exe","436","QueryOpen","C: ... \places.sqlite-shm.msf","SUCCESS","CreationTime: 12/05/2012 01:33:26, LastAccessTime: 12/05/2012 01:33:26, LastWriteTime: 12/05/2012 01:33:26, ChangeTime: 12/05/2012 01:33:26, AllocationSize: 0, EndOfFile: 0, FileAttributes: A"
> "10:52:37,8107526","thunderbird.exe","436","QueryOpen","C: ... \places.sqlite-shm","SUCCESS","CreationTime: 12/05/2012 10:52:12, LastAccessTime: 12/05/2012 10:52:12, LastWriteTime: 12/05/2012 01:13:14, ChangeTime: 12/05/2012 01:13:15, AllocationSize: 0, EndOfFile: 0, FileAttributes: A"
> "10:52:37,8108861","thunderbird.exe","436","QueryOpen","C: ... \places.sqlite-wal.msf","SUCCESS","CreationTime: 12/05/2012 01:33:26, LastAccessTime: 12/05/2012 01:33:26, LastWriteTime: 12/05/2012 01:33:26, ChangeTime: 12/05/2012 01:33:26, AllocationSize: 0, EndOfFile: 0, FileAttributes: A"
> "10:52:37,8109688","thunderbird.exe","436","QueryOpen","C: ... \places.sqlite-wal","SUCCESS","CreationTime: 12/05/2012 10:52:12, LastAccessTime: 12/05/2012 10:52:12, LastWriteTime: 12/05/2012 01:13:19, ChangeTime: 12/05/2012 01:13:23, AllocationSize: 0, EndOfFile: 0, FileAttributes: A"
(B) And following log is also seen. (searched as .sbd for subfolders of local mail folder)
> "10:52:43,4398255","thunderbird.exe","436","QueryOpen","C: ... \places.sqlite","SUCCESS","CreationTime: 12/05/2012 01:27:21, LastAccessTime: 12/05/2012 10:52:43, LastWriteTime: 12/05/2012 01:32:23, ChangeTime: 12/05/2012 01:32:23, AllocationSize: 10 485 760, EndOfFile: 10 485 760, FileAttributes: A"
> "10:52:43,4398823","thunderbird.exe","436","QueryOpen","C: ... \places.sqlite.sbd","NAME NOT FOUND",""
(C) "Creation as directory" is requested after (B).  
> "10:52:43,4403999","thunderbird.exe","436","CreateFile","C: ... \places.sqlite-shm","SUCCESS","Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Attributes: N, ShareMode: Read, Write, AllocationSize: 0, OpenResult: Created"
> "10:52:43,4421789","thunderbird.exe","436","CreateFile","C: ... \places.sqlite-wal","SUCCESS","Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Attributes: N, ShareMode: Read, Write, AllocationSize: 0, OpenResult: Created"
 
Why file of places.sqlite/places.sqlite-wal/places.sqlite-shm is still accessed as "file/directory for local mail folder"(.msf, .sbd, and no extension) in your environment?
It may be "server directory creation"(corresponds to path of Local Directory: of Server Settings) due to "mail directory not found".
Comment 62 WADA 2012-05-12 04:03:18 PDT
Oh, you wrote following too.
> I think I have even found the line in my pref.js which cause the problem : 
user_pref("mail.server.server2.directory-rel", "[ProfD]");
> When I deleted this line, the 2 files are deleted after closing thunderbird and NEVER replace by the 2 folders.

You perhaps set server2's "Local Directory: of Server Settings" to "Tb's profile directory" in the past, and continued to keep it.
What was set in mail.server.server2.directory-rel after delete of the line and restart of Tb?
Comment 63 Aurélien 2012-05-12 04:52:32 PDT
Hello Everyone.

WADA, forgot what I say in my previous post. Now I have found a way to easily reproduct the bug with a new account...

So, for more easily explain, I have created a new email address specially for it. You can use it for test !
Email address : "tbbug750781@gmail.com" (without quotes)
password : "placesproblem" (without quotes)

If you want to test try to do EXACTLY this, IN THE SAME ORDER (only one click different can be occured a no-reproducibility of the bug Maybe somes steps are unnecessary, but I didn't try to simplify the test case) :

1)Close your Thunderbird.
2)Launch thunderbird with the command line -profilemanager command line
3)Click on create a new profile and call it "placesproblem" (without quotes)
4)Select your "placesproblem" profile, and click on "Start Thunderbird"
5) Thunderbird ask to Set-up the mail account. So in "Your Name" put : "places" (without quotes), in Email address : "tbbug750781@gmail.com" (without quotes) and in password : "placesproblem" (without quotes). Click on Continue
6) Select the POP3 radio button, and click on "Create Account"
7) Click on the "Write" button to write en email, and send it to "tbbug750781@gmail.com" fill the subject and a body message. And click on "Send."
8) Click on "Get Mail", to recieve the mail. and double click on it to open the mail.
9) Close your Thunderbird.
10) Edit with a text editor (like notepad++) the prefs.js File which is in the directory of this profile.
11) Search the "mail.server.server2.directory-rel" character string in the file. Put "[ProfD]" in the value. (So now you have a line like this :  user_pref("mail.server.server2.directory-rel", "[ProfD]"); )
12) Search the "mail.server.server2.directory" character string in the file. Delete at the end of the value this character string : "\\Mail\\Local Folders". (So now you have a line like this :  user_pref("mail.server.server2.directory", "C:\\Documents and Settings\\Administrateur\\Application Data\\Thunderbird\\Profiles\\26ta72w2.placesproblem");  )
13) Add at the end of this file this 5 lines :
user_pref("places.database.lastMaintenance", 1336674260);
user_pref("places.history.expiration.transient_current_max_pages", 87215);
user_pref("places.history.expiration.transient_optimal_database_size", 139543346);
user_pref("storage.vacuum.last.index", 1);
user_pref("storage.vacuum.last.places.sqlite", 1334205512);

14) Save and Close the prefs.js file.
15) Launch thunderbird with the -profilemanager command line and select once again your "placesproblem" profile, and click on "Start Thunderbird"
16) Close the e-mail you previously open, if is always open, double click anymore on it to reopen it.
17) Open your Task Manager, and KILL the process Thunderbird.exe (If you take a look to your profile directory, you will see that somes files and some directory are created)
18) Re-Launch thunderbird with the -profilemanager command line and select once again your "placesproblem" profile, and click on "Start Thunderbird"
19) In the tree view of thunderbird, go to your Inbox folder, and doubleclick on your email to reopen it.
20) Open your Task Manager, and KILL the process Thunderbird.exe 
21) Re-Re-Launch thunderbird with the -profilemanager command line and select once again your "placesproblem" profile, and click on "Start Thunderbird"
22) Close the e-mail you previously open, if is always open, double click anymore on it to reopen it.
23) Close properly Thunderbird.

Normally at this step, you will have 4 additionnals folders which have been created :
cookies.sqlite-shm
cookies.sqlite-wal
places.sqlite-shm
places.sqlite-wal

The consequences we know them since 1 week, it make Thunderbird veryyyyyy slow.


Please, does anyone can do ALL this step and confirm that he reproduces the problem on his computer ?


I suppose that aceman have been reason here : https://bugzilla.mozilla.org/show_bug.cgi?id=750781#c24
Maybe, I have choosen the bad folder there is 2 years after a Windows system reinstallation. And maybe that it was not a severe problem for previous Thunderbird releaes than the 12.0.

So probably the problem is due to a human error, but maybe it's possible to find a way to catch it to avoid that end-user (like me !) to damage his thunderbird profile ?
Comment 64 :aceman 2012-05-12 05:34:43 PDT
(In reply to Aurélien from comment #60)
> In fact, I have made complementary test and it seems everything is due to
> one file : pref.js. When I delete this file, and replace by a new one, the
> problem disappear.
> 
> I think I have even found the line in my pref.js which cause the problem : 
> user_pref("mail.server.server2.directory-rel", "[ProfD]");
> When I deleted this line, the 2 files are deleted after closing thunderbird
> and NEVER replace by the 2 folders.
> 
> Does anyone have an idea ?

This is exactly what I was hinting at in comment 24 :)
Comment 65 :aceman 2012-05-12 05:42:45 PDT
(In reply to Aurélien from comment #63)
> I suppose that aceman have been reason here :
> https://bugzilla.mozilla.org/show_bug.cgi?id=750781#c24
> Maybe, I have choosen the bad folder there is 2 years after a Windows system
> reinstallation. And maybe that it was not a severe problem for previous
> Thunderbird releaes than the 12.0.
> 
> So probably the problem is due to a human error, but maybe it's possible to
> find a way to catch it to avoid that end-user (like me !) to damage his
> thunderbird profile ?

Yes, I think I could prevent the user to choose the profile folder as the Local Directory for mail in the Account settings. Maybe some other "bad" folders could be rejected (but the problem is with different platforms).
Bienvenu, standard8, would you like that?
Comment 66 David :Bienvenu 2012-05-12 10:35:34 PDT
(In reply to :aceman from comment #65)

> 
> Yes, I think I could prevent the user to choose the profile folder as the
> Local Directory for mail in the Account settings. Maybe some other "bad"
> folders could be rejected (but the problem is with different platforms).
> Bienvenu, standard8, would you like that?

yes, that shouldn't be allowed. I'm surprised things work at all with that setup...
Comment 67 David :Bienvenu 2012-05-12 10:46:42 PDT
In any case, we should be able to somehow detect this situation and deal with it better.
Comment 68 Andrew Sutherland [:asuth] 2012-05-13 13:51:31 PDT
(In reply to David :Bienvenu from comment #67)
> In any case, we should be able to somehow detect this situation and deal
> with it better.

It sounds like we won't need the mozStorage fix then, but I'll wait on closing that bug until additional data points suggest that all of the problems are of this variety.

In terms of fix complexity and potential eternally messed-up profiles, it seems like detecting the broken profile and sending the user to a support page that provides info on how to create a new profile and migrate existing data to that new profile is best.
Comment 69 :aceman 2012-05-13 14:08:28 PDT
I'd think we can try to disallow in in the account manager, but also the proposed fix for sqlite should be implemented. It could just throw an error into the error console if it is unable to create the files it needs. Just not hang for long times.
Comment 70 David :Bienvenu 2012-05-16 15:27:13 PDT
*** Bug 755738 has been marked as a duplicate of this bug. ***
Comment 71 Mark Banner (:standard8, limited time in Dec) 2012-05-22 01:55:27 PDT
Based on the fact this is caused by a non-default user configuration setting for something that isn't likely to be changed much, I'm taking this off the tracking list. It'd be good to get bug 577775 fixed anyway, but I don't think we need to track these specifically.
Comment 72 :aceman 2012-05-22 04:31:53 PDT
Can you decide on the sqlite-side fix and where is the bug for it?

Then I could take this bug and make the AM-side fix.
Comment 73 Mark Banner (:standard8, limited time in Dec) 2012-05-22 05:04:26 PDT
(In reply to :aceman from comment #72)
> Can you decide on the sqlite-side fix and where is the bug for it?

The bug is 752732, but I think the fix isn't high priority, as this is a not common user setting.

> Then I could take this bug and make the AM-side fix.

I think that's bug 577775 right?
Comment 74 :aceman 2012-05-22 05:59:18 PDT
(In reply to Mark Banner (:standard8) from comment #73)
> > Can you decide on the sqlite-side fix and where is the bug for it?
> The bug is 752732, but I think the fix isn't high priority, as 
Right, thanks.

> > Then I could take this bug and make the AM-side fix.
> 
> I think that's bug 577775 right?
No. See bug 577775 comment 5. That one is about really invalid (non-existent) directories.

In the bug here I could try to prevent existing valid directories that we choose to reject due to potential problems (like the profile dir).
Comment 75 Mark Banner (:standard8, limited time in Dec) 2012-05-22 06:03:28 PDT
(In reply to :aceman from comment #74)
> In the bug here I could try to prevent existing valid directories that we
> choose to reject due to potential problems (like the profile dir).

Yeah, that would be fine.
Comment 76 :aceman 2012-06-25 08:06:44 PDT
*** Bug 758118 has been marked as a duplicate of this bug. ***
Comment 77 Wayne Mery (:wsmwk, NI for questions) 2012-06-29 06:40:27 PDT
might it be possible to add something to troubleshooting page?
Comment 78 :aceman 2012-06-29 13:23:16 PDT
Path to the Local Directory. But that may be private info, like the profile path. But there already is a toggle for that.
Comment 79 :aceman 2012-07-01 12:42:22 PDT
Created attachment 638219 [details] [diff] [review]
patch

This patch must be applied on top of patch in bug 577775. It plugs a new set of checks into the Local directory checking infrastructure from that bug.

Even though the directory does exist, this patch prevents the user to use it if it is among some problematic dirs. I have chosen the profile dir and Thunderbird install dir to be bad and some others. I also implemented checks according to comment here and bug 577775 (e.g. that under profile only Mail/ImapMail/News are allowed, etc.

Please see if this is going in a good direction. And if you can come up with any other restrictions.
Comment 80 David :Bienvenu 2012-07-05 16:07:26 PDT
Comment on attachment 638219 [details] [diff] [review]
patch

I haven't run the code, but it looks reasonable. 

+localDirectoryNotAllowed=The directory specified in Local Directory setting is unsuitable for mail storage. Move your messages to another directory to avoid loosing them.

"losing", not "loosing"

and perhaps, instead of saying "move your messages...", you should say to pick a different directory.
Comment 81 :aceman 2012-07-06 02:10:02 PDT
The patch should pick up already used directories that are wrong so they may already contain messages that need to be moved out. But I have no problem using whatever wording bwinton decides on.
Comment 82 Blake Winton (:bwinton) (:☕️) 2012-07-06 08:11:29 PDT
Comment on attachment 638219 [details] [diff] [review]
patch

(UI-Reviewing by inspection, cause I have no idea how to recreate this…)

> localDirectoryNotAllowed=The directory specified in Local Directory setting is unsuitable for mail storage. Move your messages to another directory to avoid loosing them.

Yeah, I think "The directory specified in Local Directory setting is unsuitable for mail storage. To avoid losing your messages, please choose another directory." would be better.

But ui-r=me with that new wording.

Thanks,
Blake.
Comment 83 :aceman 2012-07-07 02:19:53 PDT
Bwinton, try to click Browse on the Local directory and choose your profile folder.
The message should pop up.
Comment 84 Ian Neal 2012-07-07 09:22:44 PDT
Comment on attachment 638219 [details] [diff] [review]
patch

>+++ b/mail/locales/en-US/chrome/messenger/prefs.properties
>+localDirectoryNotAllowed=The directory specified in Local Directory setting is unsuitable for mail storage. Move your messages to another directory to avoid loosing them.
Is it actually "mail storage" or would "message storage" be better as it sounds more general?

>+++ b/mailnews/base/prefs/content/AccountManager.js

>+function checkDirectoryIsAllowed(aLocalPath) {
>+  /**
>+    * Check if the directory contains/is/is a subfolder of the given special
>+    * directory
>+    *
>+    * @param aDirToCheck  array of { dir, below } objects to check.
>+    *                     'dir' is a property to retrieve from Directory service
>+    *                     'below' is a boolean saying if the mail directory
>+    *                     is allowed to be a subfolder of 'dir'.
Couldn't 'below' be an array of allowed subfolders and help avoid some code duplication for "ProfD"?

>+  let forbiddenDirs = [
Nit: not strictly forbiddenDirs, could do with a better variable name.
>+    // profile folder
>+    { dir: "ProfD", below: true },
{ dir: "ProfD", below: [ "Mail", "ImapMail", "News" ] },
>+    // GRE install folder
>+    { dir: "GreD", below: false },
{ dir: "GreD", below: [ "" ] },
>+    // Application install folder
>+    { dir: "CurProcD", below: false },
{ dir: "CurProcD", below: [ "" ] },
>+    // system temporary folder
>+    { dir: "TmpD", below: false } ];
{ dir: "TmpD", below: [ "" ] } ]

f- as I'd like to see the new patch.
Comment 85 Wayne Mery (:wsmwk, NI for questions) 2012-07-16 08:57:27 PDT
Comment on attachment 638219 [details] [diff] [review]
patch

looks good in theory - which is the extent of my expertise.
Comment 86 Wayne Mery (:wsmwk, NI for questions) 2012-07-25 04:57:03 PDT
more examples have been surfacing. for example https://getsatisfaction.com/mozilla_messaging/topics/thunderbird_is_so_slow_it_needs_life_support
Comment 87 :aceman 2012-07-25 05:01:52 PDT
Yes, I'll continue on this.
I'd really would like to see how the folder pane looks for these people...
Comment 88 Wayne Mery (:wsmwk, NI for questions) 2012-08-08 20:28:18 PDT
aceman, ready to rip?
Comment 89 :aceman 2012-08-13 00:39:55 PDT
Need to finish bug 577775 first.
Comment 90 :aceman 2012-09-04 13:22:35 PDT
Ian, should the hardcoded subdirs be taken from these prefs:
user_pref("mail.root.imap-rel", "[ProfD]ImapMail");
user_pref("mail.root.nntp-rel", "[ProfD]News");
user_pref("mail.root.none-rel", "[ProfD]Mail");
user_pref("mail.root.pop3-rel", "[ProfD]Mail");
?

What is their purpose?
Comment 91 roland 2012-09-27 02:29:44 PDT
I am experiencing what looks like the same problem being that it is sometimes extremely slow to open email and it looks like the longer the email is, the slower it is to open.  Sometimes it never opens.

I am on Ubuntu 11.04 using thunderbird 15.  I am using ESET (NOD32) but I have disabled this and the behaviour persists.  I have also tried stopping tb, deleting places.sqlite* and cookies.sqlite* and then restarting tb without any improvement in behaviour.  I am not experiencing the issue whereby places.sqlite-wal and places.sqlite-shm are being created as a directory instead of as files.
Comment 92 :aceman 2012-09-27 02:40:50 PDT
Then it must be a different problem. See e.g. bug 794401.
Comment 93 roland 2012-09-27 03:17:14 PDT
Hi :aceman,

Thanks, it looks like I have resolved it by upgrading to 15.0.1.
Comment 94 :aceman 2012-09-27 03:19:55 PDT
Yeah, there were problems like that fixed in TB15.0.1, I just expected you already have that.
Comment 95 :aceman 2012-09-27 12:37:46 PDT
Created attachment 665591 [details] [diff] [review]
patch v2

We can now move this forward.
Comment 96 Ian Neal 2012-10-07 07:46:09 PDT
Comment on attachment 665591 [details] [diff] [review]
patch v2

>+  // Do not allow some special Linux top directories.
>+  if (Services.appinfo.OS == "Linux") {
>+    let topDir = aLocalPath.clone();
>+    while (topDir.parent.path != "/")
>+      topDir = topDir.parent;
>+    const kForbiddenNames = [ "/dev", "/proc", "/sys" ];
>+    if (kForbiddenNames.indexOf(topDir.path) != -1)
>+      return false;
>+  }
Just wondering if any other OS also have similar special top directories? Worth checking OSX?

r=me with that investigated.
Comment 97 :aceman 2012-10-07 08:02:21 PDT
I can't check it. bwinton, I think you have a Mac, can you please investigate?
Comment 98 Magnus Melin 2012-10-07 11:03:44 PDT
I seem to recall on windows "C:\" will cause massive problems. I think we fixed the "remove" case but anyway, shouldn't be allowed to begin with.
Comment 99 :aceman 2012-10-07 11:24:03 PDT
Yeah, could be done. But I think you can renumber the disks in Windows so that the system disk may not be on C:. But we can add "WinD" and derive the drive letter from that one. Also "ProgF" could be disallowed.
For Mac, not sure what "SysD" returns.

http://mxr.mozilla.org/mozilla-central/source/xpcom/io/nsDirectoryServiceDefs.h
Comment 100 Blake Winton (:bwinton) (:☕️) 2012-10-07 18:44:21 PDT
OS X has "/dev", but not the other two.  On the other hand, it also has "/System".  But on the third hand, I'm not entirely sure that it's Thunderbird's job to not write to or delete those directories.  That really seems like more of an OS's job…

How would I find out what "SysD" returns?

Thanks,
Blake.
Comment 101 :aceman 2012-10-07 23:43:36 PDT
Open Firefox Error console and in the "Code" field paste this:
Components.classes["@mozilla.org/file/directory_service;1"].getService(Components.interfaces.nsIProperties).get("SysD", Components.interfaces.nsIFile).path;

Click "Evaluate".

On Windows, this returns "C:\WINDOWS\system32".
"WinD" returns "C:\WINDOWS". So I can use this to derive the system drive letter.
Comment 102 Blake Winton (:bwinton) (:☕️) 2012-10-08 13:06:08 PDT
It gives me:

NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIProperties.get]
Comment 103 :aceman 2012-10-08 13:14:55 PDT
Well, it should be defined on Mac:
http://mxr.mozilla.org/mozilla-central/source/xpcom/io/nsDirectoryServiceDefs.h#56
Comment 104 Blake Winton (:bwinton) (:☕️) 2012-10-11 06:14:51 PDT
Perhaps it should be, but I only ever get the error I reported.  ;)
Comment 105 :aceman 2012-10-16 14:16:46 PDT
Does any of the other values work?
Comment 106 :aceman 2012-11-06 13:20:25 PST
Created attachment 678866 [details] [diff] [review]
patch v3

So this adds some OS specific dirs into the kDangerousDirs structure.
This disallows e.g. "WinD" on Windows and subsequently that automatically disallows any path that contains it. So if WinD is C:\WINDOWS, then C:\ is not allowed too. At least that is the intention. Magnus, can you please try this on Windows?
Bwinton, please try Mac again. I disallowed /dev for it, so let's see if it works.
Comment 107 :aceman 2012-11-06 13:30:24 PST
Created attachment 678873 [details] [diff] [review]
patch v4
Comment 108 :aceman 2012-11-06 13:33:19 PST
Created attachment 678875 [details] [diff] [review]
patch v5

Sorry, forgot the /System on Mac.
Comment 109 Magnus Melin 2012-11-11 11:02:15 PST
Comment on attachment 678875 [details] [diff] [review]
patch v5

Review of attachment 678875 [details] [diff] [review]:
-----------------------------------------------------------------

I don't do really do windows builds, only on occasion...

::: mail/locales/en-US/chrome/messenger/prefs.properties
@@ +15,5 @@
>  localDirectoryChanged=%1$S needs to restart now to apply the change to the Local directory setting.
>  localDirectoryRestart=Restart
>  serverNameEmpty=Neither the server name nor the user name can be empty.
>  localDirectoryInvalid=The directory specified in the Local Directory setting is invalid. Please pick a different directory.
> +localDirectoryNotAllowed=The directory specified in Local Directory setting is unsuitable for message storage. To avoid losing your messages, please choose another directory.

My suggestion would be:
%1$S is not suitable to use as Local Directory. Please choose another directory.

Better not specify what the problems might be as loosing messages is just one of the issues you'd have.

::: mailnews/base/prefs/content/AccountManager.js
@@ +264,5 @@
>  
>    return true;
>  }
>  
> +/* Even if the local path is usable, there are some special folders we do not

/**
 *

@@ +291,5 @@
> +    if ("dirsvc" in aDirToCheck)
> +      try {
> +        testDir = Services.dirsvc.get(aDirToCheck.dirsvc, Components.interfaces.nsIFile);
> +      } catch (e) {
> +        Components.utils.reportError("The special folder " + aDirToCheck.dirsvc + " is cannot be retrieved on this platform.");

add ": + e" to the message so it's easier to find out what the problem was, in case we ever get here

@@ +302,5 @@
> +        return true;
> +    }
> +
> +    if (!testDir)
> +      return true;

This only applies to the if clause above, so move it there.

@@ +317,5 @@
> +      for each (let subDir in aDirToCheck.safeSubdirs) {
> +        let checkDir = testDir.clone();
> +        checkDir.append(subDir);
> +        if (checkDir.contains(aLocalPath, true))
> +          isInSubdir = true;

Shouldn't his break if finding one unsafe?
Comment 110 :aceman 2012-11-11 11:59:31 PST
(In reply to Magnus Melin from comment #109)
> > +localDirectoryNotAllowed=The directory specified in Local Directory setting is unsuitable for message storage. To avoid losing your messages, please choose another directory.

> My suggestion would be:
> %1$S is not suitable to use as Local Directory. Please choose another
> directory.
> 
> Better not specify what the problems might be as loosing messages is just
> one of the issues you'd have.

This already got some iterations over bwinton and Ian, would they be OK with such shortening? On the other hand mentioning losing messages may scare the user into fixing it quickly :)
Comment 111 :aceman 2012-11-11 12:20:12 PST
Created attachment 680459 [details] [diff] [review]
patch v6 [bitrotted]
Comment 112 Ian Neal 2012-11-11 12:53:34 PST
I'll go with whatever bwinton is happiest with.
Comment 113 Blake Winton (:bwinton) (:☕️) 2012-11-12 10:39:55 PST
Comment on attachment 680459 [details] [diff] [review]
patch v6 [bitrotted]

Yeah, I could live with "%1$S is not suitable to use as Local Directory. Please choose another directory.".  UI-r=me if you want to go with that.

Thanks,
Blake.
Comment 114 Blake Winton (:bwinton) (:☕️) 2012-11-12 10:59:21 PST
Ah, I didn't see you had changed the wording in this patch.  ui-r=me for this version, too.
Comment 115 :aceman 2012-11-30 14:25:43 PST
Magnus, can you finish this?
Comment 116 Magnus Melin 2012-12-06 11:24:19 PST
Like i said, i don't build windows much, i did now but it turns out the patch has bitrotten.
Comment 117 :aceman 2012-12-06 11:45:13 PST
Created attachment 689322 [details] [diff] [review]
patch v7
Comment 118 Magnus Melin 2013-01-02 12:10:46 PST
Comment on attachment 689322 [details] [diff] [review]
patch v7

Review of attachment 689322 [details] [diff] [review]:
-----------------------------------------------------------------

Yup, seems to work fine! Sorry for the very long review delay.
Comment 119 Ryan VanderMeulen [:RyanVM] 2013-01-12 05:07:41 PST
https://hg.mozilla.org/comm-central/rev/85c7ae82d3f6
Comment 120 Kent James (:rkent) 2013-03-06 11:12:05 PST
This is breaking account management for new account types. I have not traced it out precisely, but I suspect that the issue is coming from this:

{ dirsvc: "ProfD",    OS: null,           safeSubdirs: [ "Mail", "ImapMail", "News" ] },

This is a hidden restriction that only allows standard server types. A new account type extension should create a new subfolder of ProfD to use for storage, for example "ExQuilla".

I suppose that I can work around it in my AccountManager.js overlay (and for that reason I probably won't file a new bug), but I would really request that coders and reviewers be sensitive to the concept that you cannot make lists of core account types, even implictly as here, and restrict mailnews operations to only use those types.
Comment 121 Kent James (:rkent) 2013-03-06 11:54:47 PST
Unfortunately the logical variable to override in an extension (kDangerousDirs) is a private variable in a function, so I went ahead and filed the regression, as the overlay for an extension is more complex than it needs.
Comment 122 :aceman 2013-05-02 23:46:49 PDT
I propose mentioning this in the release notes of TB 24 as suddenly some users may be notified their mail data is in improper location and need to move the directories. Preferably the relnotes will contain a link to an article how to the migration. Can anybody do that?
Comment 123 Katie Romanchuk 2014-07-27 15:17:55 PDT
I have same problem with TB 24.2 for Mac. Emails take a long to open. If any fixes for Mac, please let me know, thank you.
Comment 124 Wayne Mery (:wsmwk, NI for questions) 2014-07-27 16:09:09 PDT
(In reply to Katie Romanchuk from comment #123)
> I have same problem with TB 24.2 for Mac. Emails take a long to open. If any
> fixes for Mac, please let me know, thank you.

The specific issue related to this bug was fixed long ago and is not likely to be your problem. You should like for a bug report that is still open (not resolved) or file a new bug report. But before you do that, 1) please update to Thunderbird 31. 2) make sure you can reproduce the issue in safe mode  https://support.mozilla.org/en-US/kb/safe-mode
Comment 125 Hiroyuki Ikezoe (:hiro) 2014-08-05 14:54:40 PDT
Changing severity and adding dataloss keyword for feature reference.

Note You need to log in before you can comment on or make changes to this bug.