Closed Bug 750781 Opened 12 years ago Closed 11 years ago

Thunderbird 12 veryyyyy slow when opening Message. Sometimes shows "Not Responding". (because places.sqlite-wal/-shm is created as mail directory by Tb if Tb's profile directory is specified by user as Local Directory: of an account)

Categories

(MailNews Core :: Account Manager, defect)

defect
Not set
critical

Tracking

(thunderbird13-)

RESOLVED FIXED
Thunderbird 21.0
Tracking Status
thunderbird13 - ---

People

(Reporter: ohms.morel, Assigned: aceman)

References

(Depends on 1 open bug, )

Details

(4 keywords, Whiteboard: [gs][)

Attachments

(3 files, 6 obsolete files)

User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; SIMBAR={AE11A89F-8F1A-11DF-A683-0022156856AF}; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 1.1.4322; .NET4.0C; .NET4.0E; Creative AutoUpdate v1.10.10)

Steps to reproduce:

Only updating from Thunderbird 11 to Thunderbird 12


Actual results:

When I click on any message, TB 12 freezes and I have to wait several seconds, sometimes several minutes for the message to open. Heavier the message is, longer it is...

It seems the matter is more general than message opening because when I try to open the "Address book" or the "Get Addons" page it's very slow too. (but other menus are quick to be displayed)

I go back to Thunderbird 11 and it works perfectly. The matter is only with Thunderbird 12 (I use Thunderbird since the 1st release)


OS : Windows XP SP3.
JVM : 1.6.0_31


Expected results:

It should be open messages instantly when click, like in previous release.
(In reply to Aurélien from comment #0)
And the message you try to open is local or on IMAP server?
Can you see if there is CPU usage by Thunderbird in the period where it is "frozen"?
Hello !

It's exactly the same on imap server or if it's a classic downloaded POP message.

But like I said, it's not only with Mail, but it seems to be the case with HTML menu like the "Get Addons" page, or even with the Address book.

During the freeze period, there is no CPU Usage !

More characters has the mail, address book or get addons, longer the freeze is.
and it the same with Thunderbird 12.0 and 12.0.1 but it works perfectly with TB 11.0.

I tried to delete .msf file, to archive and compact all message. I tried too to delete every mail of my mailbox, but it was the same.
Keywords: perf
Yes, this really sounds like some external program/process interfering with Thunderbird. Sometimes these programs whitelist some applications, and when new versions come out, the whitelisting stops working.

The only other thing I could suggest is creating a new profile and setting up your e-mail accounts again to see if it's something specific to your profile. Running Thunderbird with the -profilemanager command line option allows you to create a new profile.
(In reply to David :Bienvenu from comment #5)
> Yes, this really sounds like some external program/process interfering with
> Thunderbird. Sometimes these programs whitelist some applications, and when
> new versions come out, the whitelisting stops working.
> 

Any clues on how to troubleshoot these sort of program conflicts.
I resolved one of these myself, just by observing cpu usage in the win task manager.
In my case it was Updater.exe installed as an option to Avira Antivir.
I believe there is a common program name in the TB package.
Had something to do with a ask.com search toolbar.

This had a very specific affect on TB only. Other apps seemed to run fine.
CPU nearly pegged but only when TB was running.
We advance !!!

I read carefully all your posts ! I test all you said.

So, for information, like Joe, I'm an Avira Antivir user. So I have test to uninstall it but it didn't work. (I try too, to uninstall my Zone Alarm Firewall but no success too)... Task Manager show me that no processus are used during freeze. Process Explorer confirm me that.

So I have test the David advice. I ran TB with -profilemanager command line and... it works ! No Freeze !!! I have test to copy all mails from old profile to new profile, and it works fine too !!

So the matter is linked to my old account... I didn't delete it for the moment,  in the case of you want I take a look to some configuration of this account.

Thanks everyone for your help !
(In reply to Aurélien from comment #7)

> So the matter is linked to my old account... I didn't delete it for the
> moment,  in the case of you want I take a look to some configuration of this
> account.

There are a few possibilities as to why a new profile doesn't show the issue. One is some pref that's different in the old profile, or some extension that's not in the new profile. An other possibility is some data file in the old profile has gotten out of control. I would look for a file at the top of the old profile directory that seems unusually large. Most of the files can be deleted safely, other than prefs.js, of course, (panacea.dat, the sqlite files, etc). You could try saving off the files from the top of your profile dir to a different directory and see if removing any of them has an effect. Moving the address book files away (the ones that end in .mab) might also be interesting.

I should stress that I don't have any particular reason to think one of the files is responsible; I'm just trying to trouble-shoot.
Hello David !!!!

Now I find what the problem is due ! I have test to move some file from my older and it seems the matter is due to 2 folders that I have in the old profile :
-places.sqlite-wal
-places.sqlite-shm

These 2 folders are empty. When they are not in the profile, everything is normal and quick !!

So... I was going in my new profile folder and create 2 empty folder with the same name. And... Now it freeze now !

Could you make the test on your thunderbird to be sure it's not only with my Thunderbird. 
What are these 2 files ? Why do they slow down only the 12 releases of thunderbird, and not the TB 11 ?

Thank you for your time you spent on my problem !
places.sqlite is a sqlite database we use to keep track of the urls you've visited. It's mainly a firefox thing, but we have it as well because we use some pieces of code shared with Firefox that require it. the sqlite-wal and sqlite-shm files are helper files that sqlite uses while Thunderbird is running - the main database is just places.sqlite. When you shut down Thunderbird, do those files go away?

I really have no idea why they would make TB slow for you, per se but we can try to figure it out.
I think there is an understanding. I was not clear in my explanation. 
Yes I have 2 FILES places.sqlite-wal and places.sqlite-shm but they are not the problem. 
The problem was due to 2 empty FOLDERS called places.sqlite-wal and places.sqlite-shm !

When I deleted this 2 FOLDERS, freezes have disappeared. They are not be re-created now, so the problem is corrected, but I'm curious to know why I'm the only person to have this problem !

-Is it possible for you to make the test to create in your profile folder 2 empty folders called places.sqlite-wal and places.sqlite-shm and verify if thunderbird is slow down for you too ?
-In your opinion, why this 2 folders was create in my profile ?

I have no extension installed. (Maybe in the past, I have installed Lightin, but I'm not sure on this computer I did it)
Oh, folders! Where exactly were these folders in your user profile dir?
The 2 folders are at the root of my profile. 

So on my windows XP : C:\Documents and Settings\Administrateur\Application Data\Thunderbird\Profiles\myprofilename.default
OK, I'm a bit confused - that's the directory where those files should be, and they should be files, not folders. By folders, do you mean those two files are directories, containing other folders? If that were the case, I don't see how you could have both files and folders with the same name. Do they show up as folders in the Thunderbird UI?
David, maybe that is the problem? He has folders with those 2 names and when TB tries to show a message it accesses places and sqlite tries to create those 2 files for the places databese and can't as the names are taken by those folders. So it hangs for a long time?
Ok, so your last post help me to understand the problem.

In fact, I had the 2 folders with this name. 
Normally, when Thunderbird is launched, it creates 2 files with theses names. But like I had this 2 folders, it didn't create the 2 files (as you implied !). I suppose (But I know nothing at all !) that when a mail is open, TB try to search some information in theses files (maybe for each character in the mail like the freeze was depending of the mail size ?).

About your last question : "Do they show up as folders in the Thunderbird UI?" I'm not sure to understand. The TB tree structure  don't show them because folders are not in Mail folder but in the root of the profile folder.

So, why I had 2 folders here ? What happened to cause this problem ? Why Only TB 12 was slow down with this 2 folder (and not with TB 11 !)
aceman and David, can you made this simple test :
1)Close your thunderbird. 

2)Cerate 2 folders at the root of your profile with this name :
-places.sqlite-wal
-places.sqlite-shm

3)Re-launch Thunderbird and try to open mails and address book

Is it slower for you too ?
(In reply to Aurélien from comment #16)
> In fact, I had the 2 folders with this name. 
> Normally, when Thunderbird is launched, it creates 2 files with theses
> names. But like I had this 2 folders, it didn't create the 2 files (as you
> implied !). I suppose (But I know nothing at all !) that when a mail is
> open, TB try to search some information in theses files (maybe for each
> character in the mail like the freeze was depending of the mail size ?).
The places.sqlite database (to which those 2 files belong too) is used to store your browsing history (even in TB). If your emails have links in them (HTML) the database can be queried if you already accessed those links in the past so they can be marked 'visited'. If the database does not respond (because it can't create those 2 files) then TB could hang.
That is what I think could be a very imaginable explanation. But can't test it right now.
(In reply to Aurélien from comment #17)
> aceman and David, can you made this simple test :
> 1)Close your thunderbird. 
> 
> 2)Cerate 2 folders at the root of your profile with this name :
> -places.sqlite-wal
> -places.sqlite-shm

Do the folders really have "-" as the leading character? That's odd, but it explains why you could have the files and the folders, because the folder names are slightly different.
(In reply to Aurélien from comment #17)
> Is it slower for you too ?

I can reproduce. It's horrible. Lots of "not responding" doing almost anything. Not always on every click, but a high percentage.  I haven't attempted to determine frequency.


Aurélien, do you ever see "not responding" in windows title bar?
(In reply to David :Bienvenu from comment #19)
> Do the folders really have "-" as the leading character?
> That's odd, but it explains why you could have the files and the folders,
> because the folder names are slightly different.

No, no, without the "-" as the leading character. They are named "places.sqlite-wal" and "places.sqlite-shm" like files

(In reply to Wayne Mery (:wsmwk) from comment #20)
> Aurélien, do you ever see "not responding" in windows
> title bar?
Not ever, but sometime yes !
I put in attachment a printscreen of all folders at the root of my profile directory. 

I highlight each "strange" folders I have in my old profile...
Those marked folders are realz strange. The folders with .sbd extension are created automatically if you have a mail folder with that name and also some subfolder inside it. It looks like your profile and the Mail folder somehow mixed themselves. OR you have pointed your Local directory (in the account settings -> server settings) to your profile folder (which you should never do).
Thanks Wayne, So I was not alone in this case ^^

About you said aceman, it's possible that long times ago, I did a mishandling which cause the folder bug... Difficult to remember about that !
Here's a stack trace that we continually get stuck in once the profile has been somehow horked in this way. We could add code somewhere to check if these places.sqlite-wal and places.sqlite-shm directories (as opposed to files) exist before opening the db, and clear them out - I'm not sure where that code should go...but perhaps the sqlite gurus would have a better idea.

 	mozsqlite3.dll!retryIoerr(int * pnRetry, unsigned long * pError)  Line 33307	C
>	mozsqlite3.dll!winOpen(sqlite3_vfs * pVfs, const char * zName, sqlite3_file * id, int flags, int * pOutFlags)  Line 35263 + 0x4f bytes	C
 	mozsqlite3.dll!winOpen(sqlite3_vfs * pVfs, const char * zName, sqlite3_file * id, int flags, int * pOutFlags)  Line 35292 + 0x1f bytes	C
 	xul.dll!`anonymous namespace'::xOpen(sqlite3_vfs * vfs, const char * zName, sqlite3_file * pFile, int flags, int * pOutFlags)  Line 326 + 0x1f bytes	C++
 	mozsqlite3.dll!sqlite3OsOpen(sqlite3_vfs * pVfs, const char * zPath, sqlite3_file * pFile, int flags, int * pFlagsOut)  Line 14829 + 0x22 bytes	C
 	mozsqlite3.dll!sqlite3WalOpen(sqlite3_vfs * pVfs, sqlite3_file * pDbFd, const char * zWalName, int bNoShm, __int64 mxWalSize, Wal * * ppWal)  Line 46629 + 0x1c bytes	C
 	mozsqlite3.dll!pagerOpenWal(Pager * pPager)  Line 45203 + 0x3e bytes	C
 	mozsqlite3.dll!sqlite3PagerOpenWal(Pager * pPager, int * pbOpen)  Line 45243 + 0x9 bytes	C
 	mozsqlite3.dll!lockBtree(BtShared * pBt)  Line 51719 + 0xf bytes	C
 	mozsqlite3.dll!sqlite3BtreeBeginTrans(Btree * p, int wrflag)  Line 51972 + 0x12 bytes	C
 	mozsqlite3.dll!sqlite3InitOne(sqlite3 * db, int iDb, char * * pzErrMsg)  Line 28324 + 0xe bytes	C
 	mozsqlite3.dll!sqlite3Init(sqlite3 * db, char * * pzErrMsg)  Line 28496 + 0x11 bytes	C
 	mozsqlite3.dll!sqlite3ReadSchema(Parse * pParse)  Line 28534 + 0x10 bytes	C
 	mozsqlite3.dll!sqlite3Pragma(Parse * pParse, Token * pId1, Token * pId2, Token * pValue, int minusFlag)  Line 27258 + 0x9 bytes	C
 	mozsqlite3.dll!yy_reduce(yyParser * yypParser, int yyruleno)  Line 45238 + 0x20 bytes	C
 	mozsqlite3.dll!sqlite3Parser(void * yyp, int yymajor, Token yyminor, Parse * pParse)  Line 45622 + 0x12 bytes	C
 	mozsqlite3.dll!sqlite3RunParser(Parse * pParse, const char * zSql, char * * pzErrMsg)  Line 46458 + 0x20 bytes	C
 	mozsqlite3.dll!sqlite3Prepare(sqlite3 * db, const char * zSql, int nBytes, int saveSqlFlag, Vdbe * pReprepare, sqlite3_stmt * * ppStmt, const char * * pzTail)  Line 28710 + 0x11 bytes	C
 	mozsqlite3.dll!sqlite3LockAndPrepare(sqlite3 * db, const char * zSql, int nBytes, int saveSqlFlag, Vdbe * pOld, sqlite3_stmt * * ppStmt, const char * * pzTail)  Line 28802 + 0x21 bytes	C
 	mozsqlite3.dll!sqlite3_prepare(sqlite3 * db, const char * zSql, int nBytes, sqlite3_stmt * * ppStmt, const char * * pzTail)  Line 28866 + 0x1d bytes	C
 	mozsqlite3.dll!sqlite3_exec(sqlite3 * db, const char * zSql, int (void *, int, char * *, char * *)* xCallback, void * pArg, char * * pzErrMsg)  Line 25324 + 0x17 bytes	C
 	xul.dll!mozilla::storage::Connection::executeSql(const char * aSqlString)  Line 983 + 0x16 bytes	C++
 	xul.dll!mozilla::storage::Connection::initialize(nsIFile * aDatabaseFile, const char * aVFSName)  Line 680 + 0x17 bytes	C++
 	xul.dll!mozilla::storage::Service::OpenUnsharedDatabase(nsIFile * aDatabaseFile, mozIStorageConnection * * _connection)  Line 805 + 0x15 bytes	C++
 	xul.dll!mozilla::places::Database::InitDatabaseFile(nsCOMPtr<mozIStorageService> & aStorage, bool * aNewDatabaseCreated)  Line 524 + 0x3d bytes	C++
 	xul.dll!mozilla::places::Database::Init()  Line 427 + 0x10 bytes	C++
 	xul.dll!mozilla::places::Database::GetSingleton()  Line 377 + 0x87 bytes	C++
 	xul.dll!mozilla::places::Database::GetDatabase()  Line 125 + 0x5 bytes	C++
 	xul.dll!nsNavHistory::Init()  Line 340 + 0x9 bytes	C++
 	xul.dll!nsNavHistory::GetSingleton()  Line 299 + 0x87 bytes	C++
 	xul.dll!nsNavHistoryConstructor(nsISupports * aOuter, const nsID & aIID, void * * aResult)  Line 20 + 0x26 bytes	C++
 	xul.dll!mozilla::GenericFactory::CreateInstance(nsISupports * aOuter, const nsID & aIID, void * * aResult)  Line 48 + 0x14 bytes	C++
 	xul.dll!nsComponentManagerImpl::CreateInstanceByContractID(const char * aContractID, nsISupports * aDelegate, const nsID & aIID, void * * aResult)  Line 1064 + 0x25 bytes	C++
 	xul.dll!nsComponentManagerImpl::GetServiceByContractID(const char * aContractID, const nsID & aIID, void * * result)  Line 1466 + 0x34 bytes	C++
 	xul.dll!CallGetService(const char * aContractID, const nsID & aIID, void * * aResult)  Line 95	C++
 	xul.dll!nsGetServiceByContractID::operator()(const nsID & aIID, void * * aInstancePtr)  Line 278 + 0x13 bytes	C++
 	xul.dll!nsCOMPtr<nsINavHistoryService>::assign_from_gs_contractid(nsGetServiceByContractID gs, const nsID & aIID)  Line 1212 + 0xf bytes	C++
 	xul.dll!nsCOMPtr<nsINavHistoryService>::nsCOMPtr<nsINavHistoryService>(nsGetServiceByContractID gs)  Line 616	C++
 	xul.dll!nsNavHistory::GetHistoryService()  Line 149	C++
 	xul.dll!mozilla::places::`anonymous namespace'::VisitedQuery::Start(nsIURI * aURI, mozIVisitedStatusCallback * aCallback)  Line 351 + 0x5 bytes	C++
 	xul.dll!mozilla::places::History::RegisterVisitedCallback(nsIURI * aURI, mozilla::dom::Link * aLink)  Line 1960 + 0xb bytes	C++
 	xul.dll!mozilla::dom::Link::LinkState()  Line 127 + 0x29 bytes	C++
 	xul.dll!EnumeratePendingLinkUpdates(nsPtrHashKey<mozilla::dom::Link> * aEntry, void * aData)  Line 8092 + 0x13 bytes	C++
 	xul.dll!nsTHashtable<nsPtrHashKey<mozilla::dom::Link> >::s_EnumStub(PLDHashTable * table, PLDHashEntryHdr * entry, unsigned int number, void * arg)  Line 500 + 0x12 bytes	C++
 	xul.dll!PL_DHashTableEnumerate(PLDHashTable * table, PLDHashOperator (PLDHashTable *, PLDHashEntryHdr *, unsigned int, void *)* etor, void * arg)  Line 750 + 0x19 bytes	C++
 	xul.dll!nsTHashtable<nsPtrHashKey<mozilla::dom::Link> >::EnumerateEntries(PLDHashOperator (nsPtrHashKey<mozilla::dom::Link> *, void *)* enumFunc, void * userArg)  Line 251 + 0x12 bytes	C++
 	xul.dll!nsIDocument::FlushPendingLinkUpdates()  Line 8104	C++
 	xul.dll!nsCSSFrameConstructor::ResolveStyleContext(nsStyleContext * aParentStyleContext, nsIContent * aContent, nsFrameConstructorState * aState)  Line 4553	C++
 	xul.dll!nsCSSFrameConstructor::ResolveStyleContext(nsIFrame * aParentFrame, nsIContent * aContent, nsFrameConstructorState * aState)  Line 4542 + 0x18 bytes	C++
 	xul.dll!nsCSSFrameConstructor::AddFrameConstructionItems(nsFrameConstructorState & aState, nsIContent * aContent, bool aSuppressWhiteSpaceOptimizations, nsIFrame * aParentFrame, nsCSSFrameConstructor::FrameConstructionItemList & aItems)  Line 4995 + 0x18 bytes	C++
 	xul.dll!nsCSSFrameConstructor::ContentAppended(nsIContent * aContainer, nsIContent * aFirstNewContent, bool aAllowLazyConstruction)  Line 6582	C++
I have to say, this really does make the app unusable...
Status: UNCONFIRMED → NEW
Ever confirmed: true
Severity: normal → major
Summary: Thunderbird 12 veryyyyy slow when opening Message on Win XP → Thunderbird 12 veryyyyy slow when opening Message. Sometimes shows "Not Responding"
Maybe the same problem would be visible in Firefox too. Can anybody try it?
Shouldn't we call some Places/sqlite people to it?

Maybe all the other sqlite databases have the same problem if their names are hijacked.

Maybe we should focus on finding out why those folders appeared.
I have just make the test to create manually this 2 empty folders in the firefox profile, and yes, now it's veryyyyyyyy slow too.
Thanks for testing that.
Let's CC some people that seem to be involved in sqlite and Places.
Ugh, I thought I added sdwilsh earlier. FWIW, I doubt that Thunderbird-specific code created the directory places.sqlite-wal or -shm - there's no particular code that turns files into directories, and no reason it would just touch these two files.
(summary for drh: if the WAL/SHM files are directories on windows, SQLite seems to freak out and make mozStorage-using apps unhappy, it's not clear how the files would end up directories.)

From my reading of the SQLite source, there are no calls to CreateDirectory, and the Windows API docs seem to explicitly state that CreateFile is incapable of creating a directory on its own (http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858%28v=vs.85%29.aspx).  Only our Telemetry VFS should apply to the places database, and it does not implement any directory creation, nor should VFS structure disagreements lead to misbehavior.  The SQLite locking code does do some directory magic, but I think that is only the unix-y implementations; at an off-site right now where the hotel internet is being destroyed by a bunch of mozillians and fossil is dying during the clone ("fossil: unknown command: [4cd23fa2fd38f49090d6eeb9b1e521d51d5]"), so it's hard to tell.

It seems most likely that either the virus checker was buggy at one point, an extension that tries to optimize firefox/thunderbird by doing something with places did something dumb, or some type of aggressive trojan/DLL call hooking program glitched its thunks quite badly, and left folders as a side-effect of file accesses gone wrong.

drh, any ideas on what could cause the WAL/SHM files to be created as directories?  More applicably, is this the type of thing SQLite wants to bulletproof itself against?  (Specifically, recovering when the files are directories, as opposed to Byzantine failure of file API calls.)
(In reply to Andrew Sutherland (:asuth) from comment #33)
> (summary for drh: if the WAL/SHM files are directories on windows, SQLite
> seems to freak out and make mozStorage-using apps unhappy, it's not clear
> how the files would end up directories.)
> 
> From my reading of the SQLite source, there are no calls to CreateDirectory,
> and the Windows API docs seem to explicitly state that CreateFile is
> incapable of creating a directory on its own
> (http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858%28v=vs.
> 85%29.aspx).  Only our Telemetry VFS should apply to the places database,
> and it does not implement any directory creation, nor should VFS structure
> disagreements lead to misbehavior.  The SQLite locking code does do some
> directory magic, but I think that is only the unix-y implementations; at an
> off-site right now where the hotel internet is being destroyed by a bunch of
> mozillians and fossil is dying during the clone ("fossil: unknown command:
> [4cd23fa2fd38f49090d6eeb9b1e521d51d5]"), so it's hard to tell.
> 
> It seems most likely that either the virus checker was buggy at one point,
> an extension that tries to optimize firefox/thunderbird by doing something
> with places did something dumb, or some type of aggressive trojan/DLL call
> hooking program glitched its thunks quite badly, and left folders as a
> side-effect of file accesses gone wrong.
> 
> drh, any ideas on what could cause the WAL/SHM files to be created as
> directories?  More applicably, is this the type of thing SQLite wants to
> bulletproof itself against?  (Specifically, recovering when the files are
> directories, as opposed to Byzantine failure of file API calls.)

SQLite never creates directories (except on unix when you select the unix-dotfile VFS that uses dot-file locking instead of traditional file locking, in which case the lock files are really lock directories, but even then the names of the directories never correspond to WAL/SHM filenames.)

In our tests, manually creating directories with the same names as WAL/SHM files causes SQLite to fail to open with an SQLITE_CANTOPEN error.  Though, this failure is delayed by our anti-virus defense mechanism.  To try to work around AV software, if an CreateFile() system call fails, it is retried multiple times with the idea that some AV software might have the file open exclusively during the first try.  By default, these retries cease and the sqlite3_open_v2() request fails with SQLITE_CANTOPEN after about 1200 milliseconds.  But that delay can be adjusted using compile-time options SQLITE_WIN32_IOERR_RETRY and SQLITE_WIN32_IOERR_RETRY_DELAY or by run-time option SQLITE_FCNTL_WIN32_AV_RETRY with the sqlite3_config() interface.  Is Thunderbird increasing the AV timeout?  Might that be part of the problem?  What does Thunderbird do if it gets an SQLITE_CANTOPEN error on places?

We are looking into enhancing the AV-retry logic in SQLite so that it detects when the target file is really a directory and fails immediately rather than retrying for 1200 milliseconds.  Dunno if that will solve this problem, but is something that needs to be done nevertheless.
I have changes in an SQLite branch which cause SQLite open attempts to fail immediately with an SQLITE_CANTOPEN error if the file it is trying to open is really a directory, rather than retrying the open request on the chance that it is AV software that is preventing the open.  The diffs can be seen at http://www.sqlite.org/src/info/03875633f4 and amalgamated source code file can be downloaded from http://www.sqlite.org/tmp/sqlite3-03875633f465e.zip (file may disappear after a few weeks).  Can somebody try out this new version of SQLite and see if it clears the problem?
(In reply to D. Richard Hipp from comment #35)
> I have changes in an SQLite branch which cause SQLite open attempts to fail
> immediately with an SQLITE_CANTOPEN error if the file it is trying to open
> is really a directory, rather than retrying the open request on the chance
> that it is AV software that is preventing the open.  The diffs can be seen
> at http://www.sqlite.org/src/info/03875633f4 and amalgamated source code
> file can be downloaded from
> http://www.sqlite.org/tmp/sqlite3-03875633f465e.zip (file may disappear
> after a few weeks).  Can somebody try out this new version of SQLite and see
> if it clears the problem?

Thx, yes, I can try it here.
Yes, the patch gets us back to reasonable performance in this situation, thx! asuth, sdwilsh, should we try to incorporate this patch a la carte into mozilla/db/sqlite, or take a new code drop somehow? And/Or try to clean up the directories ourselves?
(In reply to Wayne Mery (:wsmwk) from comment #38)
> random reports still coming in. I have ~8 inquiries still open. These two
> confirm finding directory
> https://getsatisfaction.com/mozilla_messaging/topics/not_responding-12utr9
> https://getsatisfaction.com/mozilla_messaging/topics/thunderbird_problems-
> 1glh5e

it would be fascinating if they all had the same anti-virus program installed.
(In reply to David :Bienvenu from comment #37)
> Yes, the patch gets us back to reasonable performance in this situation,
> thx! asuth, sdwilsh, should we try to incorporate this patch a la carte into
> mozilla/db/sqlite, or take a new code drop somehow? And/Or try to clean up
> the directories ourselves?

We usually only ship released versions of SQLite in the tree.

I think what we should do cleanup-wise depends on whether this has been observed for other databases too.  If it has happened to other databases too, mozStorage should check and fix things.  If it has only happened to Places (possibly because only places uses the write-ahead-log) but happened in FF and TB, then either places should fix the problem or mozStorage.  If it's just places in TB, then just TB.  A lot of the decision factor is that Taras will yell at us if we add additional file-stats to the startup path of Firefox :)

If SQLite decides to automatically try and cleanup the situation, we can leave it to SQLite, but this does seem like an application-level problem.  It's on us to provide the directory and pick filenames that don't collide, etc.
Richard, by any chance, was this a known issue with sqlite and other apps?

Asuth, cleanup could be hooked into the db open error handling code for Places (or Mozstorage), couldn't it? So we wouldn't be adding extra fstats for the non-error case.
We have never before heard of anyone having trouble with *-wal and *-shm files being mysteriously changed into directories.  We are still pondering what, if anything, SQLite can and should do about it.

We will be committing changes so that the anti-virus retries are disabled when trying to open a directory as a file - the patch that fixes the delay.  SQLite version 3.7.12 will have these changes and should be out in a few weeks.  But apart from that, we are not sure what SQLite can or should do.  If an external process is messing with database files, what can SQLite do about that, really?

If anybody has any ideas as to why places.sqlite-wal and places.sqlite-shm were changed into directories, we would sure like to hear your theories.
(In reply to D. Richard Hipp from comment #43)
> 

> If anybody has any ideas as to why places.sqlite-wal and places.sqlite-shm
> were changed into directories, we would sure like to hear your theories.

The favorite theory is that a virus checker went a bit crazy - https://bugzilla.mozilla.org/show_bug.cgi?id=750781#c33 - I was just curious if you had heard other reports. We know that users who create a new profile don't have the problem re-appear (so far, at least), so it seems to have been something transitory.
(In reply to David :Bienvenu from comment #42)
> Asuth, cleanup could be hooked into the db open error handling code for
> Places (or Mozstorage), couldn't it? So we wouldn't be adding extra fstats
> for the non-error case.

The stack you provided suggests that yes, mozStorage could indeed handle this in the slow-path failure case.  Specifically, although OpenUnsharedDatabase wouldn't actually know whether the database wants to be in WAL mode, it would not be unreasonable for us to probe for directories with the appropriately mangled names if the error code lines up with that type of failure.  We could then unlink and retry the open exactly once (rather than an infinite recursion of death.)

A realistic test would likely need to create a new database, invoke "PRAGMA journal_mode=WAL", close the database, create the bogus directories, then re-open the database at which point the error cascade should be triggered and we verify success by a successful open and the directories disappearing.  The fix would want to land now, but the test would want to hang out on the tracker bug for planning to land when we take SQLite 3.7.12 or higher.

bienvenu, since I'm going to be begging for some of your review time to provide feedback on my B2G IMAP hackery, I'm willing to do a criss-cross where I provide the patch.  This is a B2G work week, but I can get to it this weekend.  I'll file the bug now.
(In reply to Andrew Sutherland (:asuth) from comment #45)

> bienvenu, since I'm going to be begging for some of your review time to
> provide feedback on my B2G IMAP hackery

Cool, yes, I'm happy to do that. I keep meaning to find time to see if I can run b2g on windows in firefox...

> I'm willing to do a criss-cross
> where I provide the patch.  This is a B2G work week, but I can get to it
> this weekend.  I'll file the bug now.
Great, thx!
(Thanks very much for your immediate and thorough support, D. Richard Hipp!  Since I think we all agree there is nothing more SQLite can do to address this issue, I'm un-CC'ing you to save you from further bug-spam.  Feel free to re-add yourself if you really do want to keep up with the bug.)
(In reply to Wayne Mery (:wsmwk) from comment #48)
> ESET NOD32, currently version 4.2.71.2. -
> http://forums.mozillazine.org/viewtopic.
> php?f=39&t=2466065&sid=bc756f6881de1a797d8b7f07214e33f4&p=11970521#p11970521

So two known occurences:
1. Aurelien using Avira Antivir in this bug, comment 7 & 11: https://bugzilla.mozilla.org/show_bug.cgi?id=750781#c11 https://bugzilla.mozilla.org/show_bug.cgi?id=750781#c7
2. Alex Moss using ESET NOD32, currently version 4.2.71.2. http://forums.mozillazine.org/viewtopic.php?f=39&t=2466065&sid=bc756f6881de1a797d8b7f07214e33f4&p=11970521#p11970521
I've created a Company Update on Get Satisfaction get input from users to see if it really is just Avira and ESET NOD32 (perhaps it's all anti-virus or perhaps AV is just a red herring!):

https://getsatisfaction.com/mozilla_messaging/topics/tb12_very_slow_when_opening_a_message_due_to_tb_places_sqlite_wal_and_shm_files_and_possibly_other_tb
(In reply to Wayne Mery (:wsmwk) from comment #51)
> Trend Micro Titanium Internet Security
> http://forums.mozillazine.org/viewtopic.
> php?f=39&t=2466065&p=11972135#p11972135

belay that - user doesn't have the offending directories - just otherwise crappy performance caused by AV
Hello,

Hmmm... For information, Tonight, I have start my computer and my thunderbird... And... It's slow once again...
I take a look in my thunderbird application data : the two folders places.sqlite-shm and places.sqlite-wal have been re-created anymore.

It seems files have been re-created this morning. When I have closed my thunderbird. (Because the time created folder correspond to the time that I have closed my Thunderbird).

Some other files have been modified at this time (7h03 french time) :
-cert8.db
-folderTree.json
-key3.db
-prefs.js
-virtualFolders.dat

Earlier in the day (6h21) when I have start my Thunderbird, a file cookies.sqlite.bak has been created.
The night before,

3 files have been modified

-blocklist.xml
-permissions.sqlite
-history.mab
At 6h20 my windows live messenger have crashed. Si my windows "Event viewer" says : 
"La description pour l'ID d'événement ( 1000 ) dans la source (Windows Live Messenger) est introuvable. L'ordinateur local n'a peut-être pas les informations de Registre ou les librairies requises pour afficher les messages émanant d'un ordinateur distant. Vous pourrez peut-être utiliser l'option /AUXSOURCE= pour récupérer cette description. Reportez-vous aux rubriques Aide et support pour plus de détails. Les informations suivantes font partie de l'événement : msnmsgr.exe; 14.0.8117.416; 4bc935af; msnmsgr.exe; 14.0.8117.416; 4bc935af; 0; 0019c72f."

Sorry it's in french but I supprose that the english traduction is probably :


"
The description for Event ID ( 1000 ) in Source ( Windows Live Messenger ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: msnmsgr.exe; 14.0.8117.416; 4bc935af; msnmsgr.exe; 14.0.8117.416; 4bc935af; 0; 0019c72f."

It's the only Error message I have in my Event Viewer since 2 days
I think I find how the 2 directories are created !

I have made this test : 
-Create two empty FILE called places.sqlite-wal and places.sqlite-shm
-Launch Thunderbird
-Close Thunderbird

After closing, the 2 files have been deleted and replace by 2 folder with the same name !

If the 2 FILES places.sqlite-shm and places.sqlite-wal exists before the Thunderbird launch, when you will close your thunderbird, the 2 FILES will be transform to 2 FOLDERS !

Can anyone can confirm this on his computer ?

So I suppose that if thunderbird does'nt close properly, the problem can be occured.

Maybe it can help you to find the problem in the sourcecode. 

For additionnal information, I have made the same test in my firefox profile directory, and the problem is not occured. I have created the 2 files manually before the firefox launch, they are correctly deleted after the closing of firefox and not replace by 2 folders unlike that with thunderbird.
(In reply to Aurélien from comment #57)
> I have made this test : 
> -Create two empty FILE called places.sqlite-wal and places.sqlite-shm
> -Launch Thunderbird
> -Close Thunderbird
> After closing, the 2 files have been deleted and replace by 2 folder with
> the same name !
> If the 2 FILES places.sqlite-shm and places.sqlite-wal exists before the
> Thunderbird launch, when you will close your thunderbird, the 2 FILES will
> be transform to 2 FOLDERS !

I can't see "Transform to two FOLDERs(directory) of the 2 FILEs" with Tb 12 on MS Win-XP SP3.
Can you get Process Monitor log for "access to places.sqlite-wal & places.sqlite-shm"?
> http://technet.microsoft.com/en-us/sysinternals/bb545021
> http://technet.microsoft.com/en-us/sysinternals/bb896645
> Monitor of access to pleces relevant files/directories
>   Filter : If path contains "\places."(no quote), Include
> To limit sqlite-wal/sqlite-shm for places only.
>   Filter : If path contains "places.sqlite-wal"(no quote), Include
>            If path contains "places.sqlite-shm"(no quote), Include
Hello everyone,

Thanks WADA for your answer ! I have followed your recommandations.

So, I launched Process Monitor with the filter If path contains "\places."(without quotes). I have created manually the 2 FILES "places.sqlite-wal" AND "places.sqlite-shm".

I have launched Thunderbird, selected a message, closed my thunderbird. I have executed a "dir" command on DOS for listing my profile directory that it confirm me that the 2 files have been replaced by directory, and immediately after, I have done an Export of the Process Monitor results. 

I put the .CSV log export of Process Monitor in attached file.

On this log, I understand it's that it's really the Thunderbird.exe process which replace the 2 files by 2 folders, isn't it ?

So... We can ask ourselves why I have the problem and not you...


In fact, I have made complementary test and it seems everything is due to one file : pref.js. When I delete this file, and replace by a new one, the problem disappear.

I think I have even found the line in my pref.js which cause the problem : 
user_pref("mail.server.server2.directory-rel", "[ProfD]");
When I deleted this line, the 2 files are deleted after closing thunderbird and NEVER replace by the 2 folders.

Does anyone have an idea ?
(In reply to Aurélien from comment #59)
> Created attachment 623402 [details]
> Process Monitor Log File

(A) Following log is seen. (files are searched as local mail folder file)
> "10:52:37,8104645","thunderbird.exe","436","QueryOpen","C: ... \places.sqlite.msf","SUCCESS","CreationTime: 12/05/2012 01:33:26, LastAccessTime: 12/05/2012 01:33:26, LastWriteTime: 12/05/2012 01:33:26, ChangeTime: 12/05/2012 01:33:26, AllocationSize: 0, EndOfFile: 0, FileAttributes: A"
> "10:52:37,8105470","thunderbird.exe","436","QueryOpen","C: ... \places.sqlite","SUCCESS","CreationTime: 12/05/2012 01:27:21, LastAccessTime: 12/05/2012 10:49:16, LastWriteTime: 12/05/2012 01:32:23, ChangeTime: 12/05/2012 01:32:23, AllocationSize: 10 485 760, EndOfFile: 10 485 760, FileAttributes: A"
> "10:52:37,8106704","thunderbird.exe","436","QueryOpen","C: ... \places.sqlite-shm.msf","SUCCESS","CreationTime: 12/05/2012 01:33:26, LastAccessTime: 12/05/2012 01:33:26, LastWriteTime: 12/05/2012 01:33:26, ChangeTime: 12/05/2012 01:33:26, AllocationSize: 0, EndOfFile: 0, FileAttributes: A"
> "10:52:37,8107526","thunderbird.exe","436","QueryOpen","C: ... \places.sqlite-shm","SUCCESS","CreationTime: 12/05/2012 10:52:12, LastAccessTime: 12/05/2012 10:52:12, LastWriteTime: 12/05/2012 01:13:14, ChangeTime: 12/05/2012 01:13:15, AllocationSize: 0, EndOfFile: 0, FileAttributes: A"
> "10:52:37,8108861","thunderbird.exe","436","QueryOpen","C: ... \places.sqlite-wal.msf","SUCCESS","CreationTime: 12/05/2012 01:33:26, LastAccessTime: 12/05/2012 01:33:26, LastWriteTime: 12/05/2012 01:33:26, ChangeTime: 12/05/2012 01:33:26, AllocationSize: 0, EndOfFile: 0, FileAttributes: A"
> "10:52:37,8109688","thunderbird.exe","436","QueryOpen","C: ... \places.sqlite-wal","SUCCESS","CreationTime: 12/05/2012 10:52:12, LastAccessTime: 12/05/2012 10:52:12, LastWriteTime: 12/05/2012 01:13:19, ChangeTime: 12/05/2012 01:13:23, AllocationSize: 0, EndOfFile: 0, FileAttributes: A"
(B) And following log is also seen. (searched as .sbd for subfolders of local mail folder)
> "10:52:43,4398255","thunderbird.exe","436","QueryOpen","C: ... \places.sqlite","SUCCESS","CreationTime: 12/05/2012 01:27:21, LastAccessTime: 12/05/2012 10:52:43, LastWriteTime: 12/05/2012 01:32:23, ChangeTime: 12/05/2012 01:32:23, AllocationSize: 10 485 760, EndOfFile: 10 485 760, FileAttributes: A"
> "10:52:43,4398823","thunderbird.exe","436","QueryOpen","C: ... \places.sqlite.sbd","NAME NOT FOUND",""
(C) "Creation as directory" is requested after (B).  
> "10:52:43,4403999","thunderbird.exe","436","CreateFile","C: ... \places.sqlite-shm","SUCCESS","Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Attributes: N, ShareMode: Read, Write, AllocationSize: 0, OpenResult: Created"
> "10:52:43,4421789","thunderbird.exe","436","CreateFile","C: ... \places.sqlite-wal","SUCCESS","Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Attributes: N, ShareMode: Read, Write, AllocationSize: 0, OpenResult: Created"
 
Why file of places.sqlite/places.sqlite-wal/places.sqlite-shm is still accessed as "file/directory for local mail folder"(.msf, .sbd, and no extension) in your environment?
It may be "server directory creation"(corresponds to path of Local Directory: of Server Settings) due to "mail directory not found".
Oh, you wrote following too.
> I think I have even found the line in my pref.js which cause the problem : 
user_pref("mail.server.server2.directory-rel", "[ProfD]");
> When I deleted this line, the 2 files are deleted after closing thunderbird and NEVER replace by the 2 folders.

You perhaps set server2's "Local Directory: of Server Settings" to "Tb's profile directory" in the past, and continued to keep it.
What was set in mail.server.server2.directory-rel after delete of the line and restart of Tb?
Hello Everyone.

WADA, forgot what I say in my previous post. Now I have found a way to easily reproduct the bug with a new account...

So, for more easily explain, I have created a new email address specially for it. You can use it for test !
Email address : "tbbug750781@gmail.com" (without quotes)
password : "placesproblem" (without quotes)

If you want to test try to do EXACTLY this, IN THE SAME ORDER (only one click different can be occured a no-reproducibility of the bug Maybe somes steps are unnecessary, but I didn't try to simplify the test case) :

1)Close your Thunderbird.
2)Launch thunderbird with the command line -profilemanager command line
3)Click on create a new profile and call it "placesproblem" (without quotes)
4)Select your "placesproblem" profile, and click on "Start Thunderbird"
5) Thunderbird ask to Set-up the mail account. So in "Your Name" put : "places" (without quotes), in Email address : "tbbug750781@gmail.com" (without quotes) and in password : "placesproblem" (without quotes). Click on Continue
6) Select the POP3 radio button, and click on "Create Account"
7) Click on the "Write" button to write en email, and send it to "tbbug750781@gmail.com" fill the subject and a body message. And click on "Send."
8) Click on "Get Mail", to recieve the mail. and double click on it to open the mail.
9) Close your Thunderbird.
10) Edit with a text editor (like notepad++) the prefs.js File which is in the directory of this profile.
11) Search the "mail.server.server2.directory-rel" character string in the file. Put "[ProfD]" in the value. (So now you have a line like this :  user_pref("mail.server.server2.directory-rel", "[ProfD]"); )
12) Search the "mail.server.server2.directory" character string in the file. Delete at the end of the value this character string : "\\Mail\\Local Folders". (So now you have a line like this :  user_pref("mail.server.server2.directory", "C:\\Documents and Settings\\Administrateur\\Application Data\\Thunderbird\\Profiles\\26ta72w2.placesproblem");  )
13) Add at the end of this file this 5 lines :
user_pref("places.database.lastMaintenance", 1336674260);
user_pref("places.history.expiration.transient_current_max_pages", 87215);
user_pref("places.history.expiration.transient_optimal_database_size", 139543346);
user_pref("storage.vacuum.last.index", 1);
user_pref("storage.vacuum.last.places.sqlite", 1334205512);

14) Save and Close the prefs.js file.
15) Launch thunderbird with the -profilemanager command line and select once again your "placesproblem" profile, and click on "Start Thunderbird"
16) Close the e-mail you previously open, if is always open, double click anymore on it to reopen it.
17) Open your Task Manager, and KILL the process Thunderbird.exe (If you take a look to your profile directory, you will see that somes files and some directory are created)
18) Re-Launch thunderbird with the -profilemanager command line and select once again your "placesproblem" profile, and click on "Start Thunderbird"
19) In the tree view of thunderbird, go to your Inbox folder, and doubleclick on your email to reopen it.
20) Open your Task Manager, and KILL the process Thunderbird.exe 
21) Re-Re-Launch thunderbird with the -profilemanager command line and select once again your "placesproblem" profile, and click on "Start Thunderbird"
22) Close the e-mail you previously open, if is always open, double click anymore on it to reopen it.
23) Close properly Thunderbird.

Normally at this step, you will have 4 additionnals folders which have been created :
cookies.sqlite-shm
cookies.sqlite-wal
places.sqlite-shm
places.sqlite-wal

The consequences we know them since 1 week, it make Thunderbird veryyyyyy slow.


Please, does anyone can do ALL this step and confirm that he reproduces the problem on his computer ?


I suppose that aceman have been reason here : https://bugzilla.mozilla.org/show_bug.cgi?id=750781#c24
Maybe, I have choosen the bad folder there is 2 years after a Windows system reinstallation. And maybe that it was not a severe problem for previous Thunderbird releaes than the 12.0.

So probably the problem is due to a human error, but maybe it's possible to find a way to catch it to avoid that end-user (like me !) to damage his thunderbird profile ?
(In reply to Aurélien from comment #60)
> In fact, I have made complementary test and it seems everything is due to
> one file : pref.js. When I delete this file, and replace by a new one, the
> problem disappear.
> 
> I think I have even found the line in my pref.js which cause the problem : 
> user_pref("mail.server.server2.directory-rel", "[ProfD]");
> When I deleted this line, the 2 files are deleted after closing thunderbird
> and NEVER replace by the 2 folders.
> 
> Does anyone have an idea ?

This is exactly what I was hinting at in comment 24 :)
(In reply to Aurélien from comment #63)
> I suppose that aceman have been reason here :
> https://bugzilla.mozilla.org/show_bug.cgi?id=750781#c24
> Maybe, I have choosen the bad folder there is 2 years after a Windows system
> reinstallation. And maybe that it was not a severe problem for previous
> Thunderbird releaes than the 12.0.
> 
> So probably the problem is due to a human error, but maybe it's possible to
> find a way to catch it to avoid that end-user (like me !) to damage his
> thunderbird profile ?

Yes, I think I could prevent the user to choose the profile folder as the Local Directory for mail in the Account settings. Maybe some other "bad" folders could be rejected (but the problem is with different platforms).
Bienvenu, standard8, would you like that?
(In reply to :aceman from comment #65)

> 
> Yes, I think I could prevent the user to choose the profile folder as the
> Local Directory for mail in the Account settings. Maybe some other "bad"
> folders could be rejected (but the problem is with different platforms).
> Bienvenu, standard8, would you like that?

yes, that shouldn't be allowed. I'm surprised things work at all with that setup...
In any case, we should be able to somehow detect this situation and deal with it better.
Summary: Thunderbird 12 veryyyyy slow when opening Message. Sometimes shows "Not Responding" → Thunderbird 12 veryyyyy slow when opening Message. Sometimes shows "Not Responding". (because places.sqlite-wal/-shm is created as mail directory by Tb if Tb's profile directory is specified by user as Local Directory: of an account)
(In reply to David :Bienvenu from comment #67)
> In any case, we should be able to somehow detect this situation and deal
> with it better.

It sounds like we won't need the mozStorage fix then, but I'll wait on closing that bug until additional data points suggest that all of the problems are of this variety.

In terms of fix complexity and potential eternally messed-up profiles, it seems like detecting the broken profile and sending the user to a support page that provides info on how to create a new profile and migrate existing data to that new profile is best.
I'd think we can try to disallow in in the account manager, but also the proposed fix for sqlite should be implemented. It could just throw an error into the error console if it is unable to create the files it needs. Just not hang for long times.
Depends on: 577775
No longer depends on: 577775
Depends on: 577775
Based on the fact this is caused by a non-default user configuration setting for something that isn't likely to be changed much, I'm taking this off the tracking list. It'd be good to get bug 577775 fixed anyway, but I don't think we need to track these specifically.
Can you decide on the sqlite-side fix and where is the bug for it?

Then I could take this bug and make the AM-side fix.
(In reply to :aceman from comment #72)
> Can you decide on the sqlite-side fix and where is the bug for it?

The bug is 752732, but I think the fix isn't high priority, as this is a not common user setting.

> Then I could take this bug and make the AM-side fix.

I think that's bug 577775 right?
(In reply to Mark Banner (:standard8) from comment #73)
> > Can you decide on the sqlite-side fix and where is the bug for it?
> The bug is 752732, but I think the fix isn't high priority, as 
Right, thanks.

> > Then I could take this bug and make the AM-side fix.
> 
> I think that's bug 577775 right?
No. See bug 577775 comment 5. That one is about really invalid (non-existent) directories.

In the bug here I could try to prevent existing valid directories that we choose to reject due to potential problems (like the profile dir).
(In reply to :aceman from comment #74)
> In the bug here I could try to prevent existing valid directories that we
> choose to reject due to potential problems (like the profile dir).

Yeah, that would be fine.
Component: General → Account Manager
OS: Windows XP → All
Product: Thunderbird → MailNews Core
QA Contact: general → account-manager
Hardware: x86 → All
Version: 12 → Trunk
Assignee: nobody → acelists
might it be possible to add something to troubleshooting page?
Path to the Local Directory. But that may be private info, like the profile path. But there already is a toggle for that.
Attached patch patch (obsolete) — Splinter Review
This patch must be applied on top of patch in bug 577775. It plugs a new set of checks into the Local directory checking infrastructure from that bug.

Even though the directory does exist, this patch prevents the user to use it if it is among some problematic dirs. I have chosen the profile dir and Thunderbird install dir to be bad and some others. I also implemented checks according to comment here and bug 577775 (e.g. that under profile only Mail/ImapMail/News are allowed, etc.

Please see if this is going in a good direction. And if you can come up with any other restrictions.
Attachment #638219 - Flags: ui-review?(bwinton)
Attachment #638219 - Flags: feedback?(dbienvenu)
Attachment #638219 - Flags: feedback?(iann_bugzilla)
Attachment #638219 - Flags: feedback?(vseerror)
Comment on attachment 638219 [details] [diff] [review]
patch

I haven't run the code, but it looks reasonable. 

+localDirectoryNotAllowed=The directory specified in Local Directory setting is unsuitable for mail storage. Move your messages to another directory to avoid loosing them.

"losing", not "loosing"

and perhaps, instead of saying "move your messages...", you should say to pick a different directory.
Attachment #638219 - Flags: feedback?(dbienvenu) → feedback+
The patch should pick up already used directories that are wrong so they may already contain messages that need to be moved out. But I have no problem using whatever wording bwinton decides on.
Comment on attachment 638219 [details] [diff] [review]
patch

(UI-Reviewing by inspection, cause I have no idea how to recreate this…)

> localDirectoryNotAllowed=The directory specified in Local Directory setting is unsuitable for mail storage. Move your messages to another directory to avoid loosing them.

Yeah, I think "The directory specified in Local Directory setting is unsuitable for mail storage. To avoid losing your messages, please choose another directory." would be better.

But ui-r=me with that new wording.

Thanks,
Blake.
Attachment #638219 - Flags: ui-review?(bwinton) → ui-review+
Bwinton, try to click Browse on the Local directory and choose your profile folder.
The message should pop up.
Comment on attachment 638219 [details] [diff] [review]
patch

>+++ b/mail/locales/en-US/chrome/messenger/prefs.properties
>+localDirectoryNotAllowed=The directory specified in Local Directory setting is unsuitable for mail storage. Move your messages to another directory to avoid loosing them.
Is it actually "mail storage" or would "message storage" be better as it sounds more general?

>+++ b/mailnews/base/prefs/content/AccountManager.js

>+function checkDirectoryIsAllowed(aLocalPath) {
>+  /**
>+    * Check if the directory contains/is/is a subfolder of the given special
>+    * directory
>+    *
>+    * @param aDirToCheck  array of { dir, below } objects to check.
>+    *                     'dir' is a property to retrieve from Directory service
>+    *                     'below' is a boolean saying if the mail directory
>+    *                     is allowed to be a subfolder of 'dir'.
Couldn't 'below' be an array of allowed subfolders and help avoid some code duplication for "ProfD"?

>+  let forbiddenDirs = [
Nit: not strictly forbiddenDirs, could do with a better variable name.
>+    // profile folder
>+    { dir: "ProfD", below: true },
{ dir: "ProfD", below: [ "Mail", "ImapMail", "News" ] },
>+    // GRE install folder
>+    { dir: "GreD", below: false },
{ dir: "GreD", below: [ "" ] },
>+    // Application install folder
>+    { dir: "CurProcD", below: false },
{ dir: "CurProcD", below: [ "" ] },
>+    // system temporary folder
>+    { dir: "TmpD", below: false } ];
{ dir: "TmpD", below: [ "" ] } ]

f- as I'd like to see the new patch.
Attachment #638219 - Flags: feedback?(iann_bugzilla) → feedback-
Comment on attachment 638219 [details] [diff] [review]
patch

looks good in theory - which is the extent of my expertise.
Attachment #638219 - Flags: feedback?(vseerror) → feedback+
Yes, I'll continue on this.
I'd really would like to see how the folder pane looks for these people...
aceman, ready to rip?
Need to finish bug 577775 first.
Status: NEW → ASSIGNED
Ian, should the hardcoded subdirs be taken from these prefs:
user_pref("mail.root.imap-rel", "[ProfD]ImapMail");
user_pref("mail.root.nntp-rel", "[ProfD]News");
user_pref("mail.root.none-rel", "[ProfD]Mail");
user_pref("mail.root.pop3-rel", "[ProfD]Mail");
?

What is their purpose?
I am experiencing what looks like the same problem being that it is sometimes extremely slow to open email and it looks like the longer the email is, the slower it is to open.  Sometimes it never opens.

I am on Ubuntu 11.04 using thunderbird 15.  I am using ESET (NOD32) but I have disabled this and the behaviour persists.  I have also tried stopping tb, deleting places.sqlite* and cookies.sqlite* and then restarting tb without any improvement in behaviour.  I am not experiencing the issue whereby places.sqlite-wal and places.sqlite-shm are being created as a directory instead of as files.
Then it must be a different problem. See e.g. bug 794401.
Hi :aceman,

Thanks, it looks like I have resolved it by upgrading to 15.0.1.
Yeah, there were problems like that fixed in TB15.0.1, I just expected you already have that.
Attached patch patch v2 (obsolete) — Splinter Review
We can now move this forward.
Attachment #638219 - Attachment is obsolete: true
Attachment #665591 - Flags: review?(iann_bugzilla)
Comment on attachment 665591 [details] [diff] [review]
patch v2

>+  // Do not allow some special Linux top directories.
>+  if (Services.appinfo.OS == "Linux") {
>+    let topDir = aLocalPath.clone();
>+    while (topDir.parent.path != "/")
>+      topDir = topDir.parent;
>+    const kForbiddenNames = [ "/dev", "/proc", "/sys" ];
>+    if (kForbiddenNames.indexOf(topDir.path) != -1)
>+      return false;
>+  }
Just wondering if any other OS also have similar special top directories? Worth checking OSX?

r=me with that investigated.
Attachment #665591 - Flags: review?(iann_bugzilla) → review+
I can't check it. bwinton, I think you have a Mac, can you please investigate?
I seem to recall on windows "C:\" will cause massive problems. I think we fixed the "remove" case but anyway, shouldn't be allowed to begin with.
Yeah, could be done. But I think you can renumber the disks in Windows so that the system disk may not be on C:. But we can add "WinD" and derive the drive letter from that one. Also "ProgF" could be disallowed.
For Mac, not sure what "SysD" returns.

http://mxr.mozilla.org/mozilla-central/source/xpcom/io/nsDirectoryServiceDefs.h
OS X has "/dev", but not the other two.  On the other hand, it also has "/System".  But on the third hand, I'm not entirely sure that it's Thunderbird's job to not write to or delete those directories.  That really seems like more of an OS's job…

How would I find out what "SysD" returns?

Thanks,
Blake.
Open Firefox Error console and in the "Code" field paste this:
Components.classes["@mozilla.org/file/directory_service;1"].getService(Components.interfaces.nsIProperties).get("SysD", Components.interfaces.nsIFile).path;

Click "Evaluate".

On Windows, this returns "C:\WINDOWS\system32".
"WinD" returns "C:\WINDOWS". So I can use this to derive the system drive letter.
It gives me:

NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIProperties.get]
Perhaps it should be, but I only ever get the error I reported.  ;)
Does any of the other values work?
Attached patch patch v3 (obsolete) — Splinter Review
So this adds some OS specific dirs into the kDangerousDirs structure.
This disallows e.g. "WinD" on Windows and subsequently that automatically disallows any path that contains it. So if WinD is C:\WINDOWS, then C:\ is not allowed too. At least that is the intention. Magnus, can you please try this on Windows?
Bwinton, please try Mac again. I disallowed /dev for it, so let's see if it works.
Attachment #665591 - Attachment is obsolete: true
Attachment #678866 - Flags: review?(mkmelin+mozilla)
Attachment #678866 - Flags: feedback?(bwinton)
Attached patch patch v4 (obsolete) — Splinter Review
Attachment #678866 - Attachment is obsolete: true
Attachment #678866 - Flags: review?(mkmelin+mozilla)
Attachment #678866 - Flags: feedback?(bwinton)
Attachment #678873 - Flags: review?(mkmelin+mozilla)
Attachment #678873 - Flags: feedback?(bwinton)
Attached patch patch v5 (obsolete) — Splinter Review
Sorry, forgot the /System on Mac.
Attachment #678873 - Attachment is obsolete: true
Attachment #678873 - Flags: review?(mkmelin+mozilla)
Attachment #678873 - Flags: feedback?(bwinton)
Attachment #678875 - Flags: review?(mkmelin+mozilla)
Attachment #678875 - Flags: feedback?(bwinton)
Comment on attachment 678875 [details] [diff] [review]
patch v5

Review of attachment 678875 [details] [diff] [review]:
-----------------------------------------------------------------

I don't do really do windows builds, only on occasion...

::: mail/locales/en-US/chrome/messenger/prefs.properties
@@ +15,5 @@
>  localDirectoryChanged=%1$S needs to restart now to apply the change to the Local directory setting.
>  localDirectoryRestart=Restart
>  serverNameEmpty=Neither the server name nor the user name can be empty.
>  localDirectoryInvalid=The directory specified in the Local Directory setting is invalid. Please pick a different directory.
> +localDirectoryNotAllowed=The directory specified in Local Directory setting is unsuitable for message storage. To avoid losing your messages, please choose another directory.

My suggestion would be:
%1$S is not suitable to use as Local Directory. Please choose another directory.

Better not specify what the problems might be as loosing messages is just one of the issues you'd have.

::: mailnews/base/prefs/content/AccountManager.js
@@ +264,5 @@
>  
>    return true;
>  }
>  
> +/* Even if the local path is usable, there are some special folders we do not

/**
 *

@@ +291,5 @@
> +    if ("dirsvc" in aDirToCheck)
> +      try {
> +        testDir = Services.dirsvc.get(aDirToCheck.dirsvc, Components.interfaces.nsIFile);
> +      } catch (e) {
> +        Components.utils.reportError("The special folder " + aDirToCheck.dirsvc + " is cannot be retrieved on this platform.");

add ": + e" to the message so it's easier to find out what the problem was, in case we ever get here

@@ +302,5 @@
> +        return true;
> +    }
> +
> +    if (!testDir)
> +      return true;

This only applies to the if clause above, so move it there.

@@ +317,5 @@
> +      for each (let subDir in aDirToCheck.safeSubdirs) {
> +        let checkDir = testDir.clone();
> +        checkDir.append(subDir);
> +        if (checkDir.contains(aLocalPath, true))
> +          isInSubdir = true;

Shouldn't his break if finding one unsafe?
(In reply to Magnus Melin from comment #109)
> > +localDirectoryNotAllowed=The directory specified in Local Directory setting is unsuitable for message storage. To avoid losing your messages, please choose another directory.

> My suggestion would be:
> %1$S is not suitable to use as Local Directory. Please choose another
> directory.
> 
> Better not specify what the problems might be as loosing messages is just
> one of the issues you'd have.

This already got some iterations over bwinton and Ian, would they be OK with such shortening? On the other hand mentioning losing messages may scare the user into fixing it quickly :)
Flags: needinfo?(iann_bugzilla)
Attached patch patch v6 [bitrotted] (obsolete) — Splinter Review
Attachment #678875 - Attachment is obsolete: true
Attachment #678875 - Flags: review?(mkmelin+mozilla)
Attachment #678875 - Flags: feedback?(bwinton)
Attachment #680459 - Flags: ui-review?(bwinton)
Attachment #680459 - Flags: review?(mkmelin+mozilla)
I'll go with whatever bwinton is happiest with.
Flags: needinfo?(iann_bugzilla) needinfo?(iann_bugzilla) → needinfo+
Comment on attachment 680459 [details] [diff] [review]
patch v6 [bitrotted]

Yeah, I could live with "%1$S is not suitable to use as Local Directory. Please choose another directory.".  UI-r=me if you want to go with that.

Thanks,
Blake.
Attachment #680459 - Flags: ui-review?(bwinton) → ui-review+
Ah, I didn't see you had changed the wording in this patch.  ui-r=me for this version, too.
Magnus, can you finish this?
Flags: needinfo?(mkmelin+mozilla)
Like i said, i don't build windows much, i did now but it turns out the patch has bitrotten.
Flags: needinfo?(mkmelin+mozilla)
Attachment #680459 - Attachment description: patch v6 → patch v6 [bitrotted]
Attachment #680459 - Attachment is obsolete: true
Attachment #680459 - Flags: review?(mkmelin+mozilla)
Attached patch patch v7Splinter Review
Attachment #689322 - Flags: review?(mkmelin+mozilla)
Comment on attachment 689322 [details] [diff] [review]
patch v7

Review of attachment 689322 [details] [diff] [review]:
-----------------------------------------------------------------

Yup, seems to work fine! Sorry for the very long review delay.
Attachment #689322 - Flags: review?(mkmelin+mozilla) → review+
Keywords: checkin-needed
https://hg.mozilla.org/comm-central/rev/85c7ae82d3f6
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 21.0
This is breaking account management for new account types. I have not traced it out precisely, but I suspect that the issue is coming from this:

{ dirsvc: "ProfD",    OS: null,           safeSubdirs: [ "Mail", "ImapMail", "News" ] },

This is a hidden restriction that only allows standard server types. A new account type extension should create a new subfolder of ProfD to use for storage, for example "ExQuilla".

I suppose that I can work around it in my AccountManager.js overlay (and for that reason I probably won't file a new bug), but I would really request that coders and reviewers be sensitive to the concept that you cannot make lists of core account types, even implictly as here, and restrict mailnews operations to only use those types.
Depends on: 848477
Unfortunately the logical variable to override in an extension (kDangerousDirs) is a private variable in a function, so I went ahead and filed the regression, as the overlay for an extension is more complex than it needs.
Keywords: addon-compat
I propose mentioning this in the release notes of TB 24 as suddenly some users may be notified their mail data is in improper location and need to move the directories. Preferably the relnotes will contain a link to an article how to the migration. Can anybody do that?
Keywords: relnote
Depends on: 921371
Depends on: 946876
Keywords: relnote
I have same problem with TB 24.2 for Mac. Emails take a long to open. If any fixes for Mac, please let me know, thank you.
(In reply to Katie Romanchuk from comment #123)
> I have same problem with TB 24.2 for Mac. Emails take a long to open. If any
> fixes for Mac, please let me know, thank you.

The specific issue related to this bug was fixed long ago and is not likely to be your problem. You should like for a bug report that is still open (not resolved) or file a new bug report. But before you do that, 1) please update to Thunderbird 31. 2) make sure you can reproduce the issue in safe mode  https://support.mozilla.org/en-US/kb/safe-mode
Whiteboard: [gs] → [gs][
Changing severity and adding dataloss keyword for feature reference.
Severity: major → critical
Keywords: dataloss
You need to log in before you can comment on or make changes to this bug.