422687 - encfs makes Firefox crash at start [@ libsqlite3.so@0x30fad][@ sqlite3_exec]

Status: RESOLVED FIXED

(Toolkit :: Storage, defect)

Description

[Ciprian Popovici]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.12) Gecko/20080201 Powerpanda/2.0.0.6 (Firefox/2.0.0.6)
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b4) Gecko/2008030318 Firefox/3.0b4

I have a directory encrypted with EncFS, where I keep certain sensitive files. I link to those files from the normal places where they're expected to be. Example: from ~/.mozilla to ~/.encfs/mozilla, where ~/.encfs is an EncFS mounted dir (via Linux FUSE).

Firefox 3 beta 4 crashes instantly on startup if the profile dir I attempt to use is under EncFS. I can run the profile manager fine, it just won't start. FWIW, I see many places.sqlite-*.corrupt files that seem to appear after every crash.

It dumps a core but the backtrace is useless (one liner).

I can work around this issue by moving the profile I want to use with FF3 b4 outside of the EncFS encrypted dir and symlinking to the new location. So now the symlinks look like this:
~/.mozilla -> ~/.encfs/mozilla
~/.encfs/mozilla/firefox/foobar.ff3b4 -> ~/.mozilla.ff3/firefox/foobar.ff3b4

FF3 beta 2 works perfectly under EncFS. Problems started in beta 3, who would start but fail to work properly (the address bar for instance was inactive, I would enter an URL and press enter or click go and it would do nothing).

Reproducible: Always

Steps to Reproduce:
1. Attempt to use a profile dir under a directory mounted with FUSE and encrypted with EncFS.
2. Crash.
Actual Results:  
Crashes.

Expected Results:  
Not crashing. Putting sensitive files under EncFS protection is very important to me. Not being able to do so with FF3 will basically stop me from using it, and resume to using FF2 or FF3 up to and including beta 2.

I use Debian unstable. FF3 is the binary official tarball published by Mozilla. Both the EncFS and FUSE support are those that come from Debian unstable (ie. I didn't compile them myself).

Comment 1

[timeless]

does about:crashes load?

Comment 2

[Ciprian Popovici]

Sure (well, when I move the profile out of EncFS. :P ).

Here's an entry:
http://crash-stats.mozilla.com/report/index/c9da711e-f1ae-11dc-aeb2-001a4bd43ed6

Comment 3

[timeless]

thanks. so I guessed right about sqlite. but there are no symbols. Can you use a debugger (./run-mozilla.sh -g -d gdb ./firefox-bin; r) and indicate the path listed for libsqlite3.so (gdb should mention it at least once as the process runs). I'm wondering if we're getting a system library instead of our own. because if we aren't, then i really don't understand why we don't have symbols for it :(. obviously if the sqlite isn't ours, it'd be appreciated if you could cause that one not to be found and give ours a chance to crash (and then report another incident), if it is ours, if you could try to setup a build env that would be appreciated. thanks for your prompt reply

Signature	libsqlite3.so@0x30fad
UUID	c9da711e-f1ae-11dc-aeb2-001a4bd43ed6
Time	2008-03-14 03:09:25-07:00
Uptime	0
Product	Firefox
Version	3.0b4
Build ID	2008030318
OS	Linux
OS Version	0.0.0 Linux 2.6.22.1-080204 #1 PREEMPT Tue Feb 5 00:06:49 EET 2008 i686 GNU/Linux
CPU	x86
CPU Info	AuthenticAMD family 1 model 8 stepping 1
Crash Reason	SIGSEGV
Crash Address	0xb682dfad
Comments	Profile stored under EncFS makes FF crash.
Crashing Thread
Frame 	Signature 	Source
0 	libsqlite3.so@0x30fad 	
1 	mozStorageConnection::CreateTable(char const*, char const*) 	mozilla/storage/src/mozStorageConnection.cpp:460
2 	xptiInterfaceEntry::EnsureResolved(xptiWorkingSet*) 	
3 	XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode) 	mozilla/js/src/xpconnect/src/xpcwrappednative.cpp:2369
4 	XPC_WN_CallMethod(JSContext*, JSObject*, unsigned int, long*, long*) 	mozilla/js/src/xpconnect/src/xpcwrappednativejsops.cpp:1470
5 	js_Invoke 	mozilla/js/src/jsinterp.c:1444
6 	js_Interpret 	mozilla/js/src/jsinterp.c:4805
7 	js_Invoke 	mozilla/js/src/jsinterp.c:1460
8 	js_InvokeConstructor 	mozilla/js/src/jsinterp.c:2034
9 	js_Interpret 	mozilla/js/src/jsinterp.c:3769
10 	js_Invoke 	mozilla/js/src/jsinterp.c:1460
11 	nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS*, unsigned short, XPTMethodDescriptor const*, nsXPTCMiniVariant*) 	mozilla/js/src/xpconnect/src/xpcwrappedjsclass.cpp:1475
12 	nsXPCWrappedJS::CallMethod(unsigned short, XPTMethodDescriptor const*, nsXPTCMiniVariant*) 	mozilla/js/src/xpconnect/src/xpcwrappedjs.cpp:556
13 	PrepareAndDispatch 	mozilla/xpcom/reflect/xptcall/src/md/unix/xptcstubs_gcc_x86_unix.cpp:95
14 	nsComponentManagerImpl::CreateInstance(nsID const&, nsISupports*, nsID const&, void**) 	mozilla/xpcom/components/nsComponentManager.cpp:1670
15 	nsComponentManagerImpl::GetService(nsID const&, nsID const&, void**) 	mozilla/xpcom/components/nsComponentManager.cpp:1882
16 	nsJSCID::GetService(nsISupports**) 	mozilla/js/src/xpconnect/src/xpcjsid.cpp:894
17 	xptiInterfaceEntry::EnsureResolved(xptiWorkingSet*) 	
18 	XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode) 	mozilla/js/src/xpconnect/src/xpcwrappednative.cpp:2369
19 	XPC_WN_CallMethod(JSContext*, JSObject*, unsigned int, long*, long*) 	mozilla/js/src/xpconnect/src/xpcwrappednativejsops.cpp:1470
20 	js_Invoke 	mozilla/js/src/jsinterp.c:1444
21 	js_Interpret 	mozilla/js/src/jsinterp.c:4805
22 	js_Invoke 	mozilla/js/src/jsinterp.c:1460
23 	js_InternalInvoke 	mozilla/js/src/jsinterp.c:1516
24 	js_InternalGetOrSet 	mozilla/js/src/jsinterp.c:1574
25 	js_NativeGet 	mozilla/js/src/jsobj.c:3523
26 	js_GetPropertyHelper 	mozilla/js/src/jsobj.c:3672
27 	js_Interpret 	mozilla/js/src/jsinterp.c:4137
28 	js_Invoke 	mozilla/js/src/jsinterp.c:1460
29 	js_InternalInvoke 	mozilla/js/src/jsinterp.c:1516
30 	js_InternalGetOrSet 	mozilla/js/src/jsinterp.c:1574
31 	js_NativeGet 	mozilla/js/src/jsobj.c:3523
32 	js_GetPropertyHelper 	mozilla/js/src/jsobj.c:3672
33 	js_Interpret 	mozilla/js/src/jsinterp.c:4137
34 	js_Invoke 	mozilla/js/src/jsinterp.c:1460
35 	js_InternalInvoke 	mozilla/js/src/jsinterp.c:1516
36 	JS_CallFunctionValue 	mozilla/js/src/jsapi.c:4982
37 	nsJSContext::CallEventHandler(nsISupports*, void*, void*, nsIArray*, nsIVariant**) 	mozilla/dom/src/base/nsJSEnvironment.cpp:1961
38 	nsGlobalWindow::RunTimeout(nsTimeout*) 	mozilla/dom/src/base/nsGlobalWindow.cpp:7739
39 	nsGlobalWindow::TimerCallback(nsITimer*, void*) 	mozilla/dom/src/base/nsGlobalWindow.cpp:8070
40 	nsTimerImpl::Fire() 	mozilla/xpcom/threads/nsTimerImpl.cpp:400
41 	nsTimerEvent::Run() 	mozilla/xpcom/threads/nsTimerImpl.cpp:488
42 	nsThread::ProcessNextEvent(int, int*) 	mozilla/xpcom/threads/nsThread.cpp:510
43 	NS_ProcessNextEvent_P(nsIThread*, int) 	nsThreadUtils.cpp:227
44 	nsBaseAppShell::Run() 	mozilla/widget/src/xpwidgets/nsBaseAppShell.cpp:151
45 	nsAppStartup::Run() 	mozilla/toolkit/components/startup/src/nsAppStartup.cpp:181
46 	XRE_main 	mozilla/toolkit/xre/nsAppRunner.cpp:3149
47 	main 	mozilla/browser/app/nsBrowserApp.cpp:158
48 	libc-2.7.so@0x16455 	

Comment 4

[(not currently active) Ted Mielczarek]

As discussed on IRC, our sqlite is missing symbols because sqlite3.c has >64k lines, and we're using stabs, which uses a short int for line number.

Comment 5

[Ciprian Popovici]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb64dab80 (LWP 11776)]
0xb6896fad in sqlite3_exec () from ./libsqlite3.so

(gdb) bt
#0  0xb6896fad in sqlite3_exec () from ./libsqlite3.so
#1  0xb7ac9918 in ?? () from ./libxul.so
#2  0x00000000 in ?? ()

Please advise about what you mean by "build env" and how I can help further. I presume you mean downloading a certain revision of the source code and doing stuff to it. :)

Comment 6

[Ciprian Popovici]

FWIW, I did it the other way around too. I replaced the FF libsqlite3.so with the system lib, to which I've also added the debug package. I got a more verbose message:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb64b0ac0 (LWP 22258)]
sqlite3_exec (db=0x0, 
    zSql=0x88a5350 "CREATE TABLE engine_data (id INTEGER PRIMARY KEY, engineid STRING, name STRING, value STRING)", xCallback=0, pArg=0x0, pzErrMsg=0x0) at ./src/legacy.c:50
50      ./src/legacy.c: No such file or directory.
        in ./src/legacy.c

Also, here's another clue that may or may not help. I've also had a bit of a trouble with the local Apache, convincing it to serve files from a dir also under my EncFS. It refused to do so (claimed it couldn't find them) until I told it to run as my user (User ciprian) AND my main group (Group users). If I used my user but another group (root or nobody) it wouldn't work. It still complains there is no such DocumentRoot when it starts, but at some point I suppose it changes credentials and can serve files just fine.

Comment 7

[timeless]

Attachment: wow it was obvious

thanks that helped a lot. i'm still chasing the other angles. it turns out we could have figured this out on our own w/ just your original information, but boy it was so much easier not having to worry about what we had.

Comment 8

[timeless]

anyway, since you have symbols, can you debug the other half of the problem? you'll need to file a new bug, but it will probably be a bug against sqlite or encfs and not mozilla. offhand my guess is that it has issues establishing a locking system [findLockInfo seems to be the best candidate].

./run-mozilla.sh -g -d gdb ./firefox-bin
b sqlite3_open
b openDirectory
b fillInUnixFile
b sqlite3DetectLockingStyle
b findLockInfo
r

you're going to need to trace through sqlite3_open and figure out where it fails and why. ideally you should be able to ask your distribution for the sources for sqlite3, that would of course help debugging, but is not technically necessary if you don't mind playing blind in a debugger.

Comment 9

[Shawn Wilsher :sdwilsh]

Comment on attachment 309755 [details] [diff] [review]
wow it was obvious

Dang - we missed one when we audited the code for places that could crash like that :/

Drivers this is a low risk patch that just does a wee bit more error checking.

r=sdwilsh

Comment 10

[timeless]

Attachment: testcase

js> sql=Components.classes["@mozilla.org/storage/service;1"].getService(Components.interfaces.mozIStorageService);
js> LocalFile=Components.Constructor("@mozilla.org/file/local;1","nsILocalFile","initWithPath");
js> db=sql.openDatabase(LocalFile("c:\\a"));
js> db.createTable("a","");

just for reference, you can trivially trigger this on windows :)
  sqlite3_mutex_enter(db->mutex);
00 sqlite3!sqlite3_exec(struct sqlite3 * db = 0x00000000, char * zSql = 0x01a2d260 "CREATE TABLE a ()", <function> * xCallback = 0x00000000, void * pArg = 0x00000000, char ** pzErrMsg = 0x00000000)+0x32 [mozilla\db\sqlite3\src\sqlite3.c @ 55850]
01 strgcmps!mozStorageConnection::CreateTable(char * aTableName = 0x01a07580 "a", char * aTableSchema = 0x0199e7f8 "")
02 xpcom_core!NS_InvokeByIndex_P

Comment 11

[Shawn Wilsher :sdwilsh]

Of course, this also means that this crash is occurring because someone is not using the api correctly....
http://bonsai.mozilla.org/cvsblame.cgi?file=/mozilla/browser/components/search/nsSearchService.js&rev=1.110#2927

Comment 12

[Shawn Wilsher :sdwilsh]

Comment on attachment 309766 [details] [diff] [review]
testcase

>+function test_createTable(){
nit: could you please follow the format in the rest of this file for declaring the test function - and call it test_createTable_bug422687?

>+  var ds = Components.classes["@mozilla.org/file/directory_service;1"].getService(Components.interfaces.nsIProperties);
nit: we have Cc and Ci here for line wrapping purposes.  Please use them

>+  var temp = ds.get("TmpD",Components.interfaces.nsIFile);
ditto

>+  temp.append("db-table");
>+  try {
>+    var con = getService().openDatabase(temp);
>+    con.createTable("a","");
>+  } catch (e) {
do_check_eq(e.result, Cr.NS_ERROR_NOT_INITIALIZES); please

>+    if (temp.exists()) {
>+      temp.remove(true);
why recursive delete?  also, braces not needed

>+var tests = [
>+  test_connectionReady_open,
>+  test_connectionReady_closed,
>+  test_databaseFile,
>+  test_tableExists_not_created,
>+  test_indexExists_not_created,
>+  test_createTable_not_created,
>+  test_indexExists_created,
>+  test_createTable_already_created,
>+  test_lastInsertRowID,
>+  test_transactionInProgress_no,
>+  test_transactionInProgress_yes,
>+  test_commitTransaction_no_transaction,
>+  test_rollbackTransaction_no_transaction,
>+  test_get_schemaVersion_not_set,
>+  test_set_schemaVersion,
>+  test_set_schemaVersion_same,
>+  test_set_schemaVersion_negative,
>+  test_backup_not_new_filename,
>+  test_backup_new_filename,
>+  test_backup_new_folder,
>+  test_createTable,
I like the idea of the lack of blame-loss this introduces

Comment 13

[Shawn Wilsher :sdwilsh]

Comment on attachment 309755 [details] [diff] [review]
wow it was obvious

It's easier to get approval when there are tests with the patch :)

Comment 14

[Shawn Wilsher :sdwilsh]

I have to admit though, I don't understand who the connection is failing to be ready in these cases...

Comment 15

[timeless]

Attachment: combined

oh, the create stuff makes enough sense, the locking code clearly can fail if it can't figure out how to handle the file system, and clearly encfs is relatively foreign. whether it's sqlite's fault or encfs remains to be seen (arguably both should be fixed).

Comment 16

[Shawn Wilsher :sdwilsh]

Comment on attachment 309777 [details] [diff] [review]
combined

r=sdwilsh

low risk crash fix with a test!

Comment 17

[Shawn Wilsher :sdwilsh]

(In reply to comment #15)
> oh, the create stuff makes enough sense, the locking code clearly can fail if
> it can't figure out how to handle the file system, and clearly encfs is
> relatively foreign. whether it's sqlite's fault or encfs remains to be seen
> (arguably both should be fixed).
Would you be willing to either file a ticket with the sqlite folks on this issue or e-mail the discuss list?  I'll admit that I won't have time to look into this and understand the issue to file a good report for about a month.

Comment 18

[timeless]

I'm hoping the reporter will do that once he gets the details i asked for, he's so far been very responsive and helpful.

Comment 19

[Sylvain Pasche]

I tried running a profile on Encfs with b4 or trunk on my Ubuntu 7.10 amd64 box, but I didn't crash.
encfs: 1.3.2-1-1, kernel 2.6.24-11-generic

Comment 20

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 309777 [details] [diff] [review]
combined

Yay, tests! a=beltzner

Comment 21

[Ciprian Popovici]

Attachment: gdb breakpoints output

Comment 22

[Ciprian Popovici]

I've added a file "firefox.breakpoints" with the gdb output using it like timeless instructed.

BTW, I haven't really used gdb before, so it involves more than monkeying around with the commands ie. actually knowing what I'm doing, I may not be able to help.

Is there another tool that can give some insight into what's going on but not require special skills from my part? Perhaps strace or valgrind can help?

Comment 23

[timeless]

Comment on attachment 309777 [details] [diff] [review]
combined

mozilla/storage/src/mozStorageConnection.cpp 	1.32

Comment 24

[Shawn Wilsher :sdwilsh]

(In reply to comment #23)
> (From update of attachment 309777 [details] [diff] [review])
> mozilla/storage/src/mozStorageConnection.cpp    1.32
what about that testcase?