629296 - journal_mode = WAL doesn't work on NFS on Solaris

Status: RESOLVED INACTIVE

(Core :: SQLite and Embedded Database Bindings, defect)

Description

[Ginn Chen]

Maybe I should file this bug for Cookie and Places because they use WAL now.

This issue caused Solaris users cannot get any bookmark, history, cookie stored with Firefox 4.0 beta 9 if their profile is on NFS.

I'm using default mount nfs options.
nfs server is Solaris.

Tried with sqlite3 command line.
sqlite3 /net/...../places.sqlite
sqlite> PRAGMA journal_mode;
Error: disk I/O error

The problem is mmap() failed to map the -wal file.

Because setting journal_mode from delete to wal is successful, we have no chance to fallback.

Tried with an empty db.
sqlite3 /net/.../test.sqlite
sqlite> PRAGMA journal_mode;
delete
sqlite> .tables
sqlite> PRAGMA journal_mode = wal;
wal
sqlite> PRAGMA journal_mode;
wal
sqlite> .tables
Error: disk I/O error
Ctrl+D
sqlite3 /net/.../test.sqlite
sqlite> PRAGMA journal_mode;
Error: disk I/O error

However, I tried to use Mac OS X and Linux with the same NFS server, it works fine.

The manual of sqlite3 says WAL should not be used over NFS.
But for the Firefox case, only one client is active, so I don't know if it is allowed.

Comment 1

[Marco Bonardo [:mak] (Away 10-21 August)]

If applying wal fails we fallback, but in this case looks like setting wal works, it's just that any operation later fails. Most likely should be asked upstream.

Comment 2

[Shawn Wilsher :sdwilsh]

(In reply to comment #0)
> The manual of sqlite3 says WAL should not be used over NFS.
It is fine to use if you only have one client connected to it, which is the case with Firefox.

Comment 3

[Ginn Chen]

(In reply to comment #0)
> The problem is mmap() failed to map the -wal file.
I meant -shm file.

Comment 4

[Shawn Wilsher :sdwilsh]

I'm told by drh that they have something that should work for Solaris now.  Can you e-mail him to get the sqlite3.c file to drop in so you can test it?  You'll also need to set the VFS to "unix-excl" when calling sqlite3_open.

Comment 5

[D. Richard Hipp]

The sqlite3.c file contained in the "amalgamation" ZIP archive at http://www.sqlite.org/draft/download.html contains the necessary enhancements.  Please give it a try and let me know if it helps.

Note that to set the "unix-excl" VFS you have to use sqlite3_open_v2() instead of sqlite3_open() and pass in the string "unix-excl" as the 4th argument.  Docs on the sqlite3_open_v2() interface are here:  http://www.sqlite.org/c3ref/open.html

Phone me at +1.704.948.4565 or skype drichardhipp or direct email to [e-mail] if I can help.

Comment 6

[Ginn Chen]

Good!

I've tried with that sqlite3.c and it works fine!

sdwilsh, where is the best place for "unix-excl"?
Should I make it default in mozStorageConnection.h?
Or use it only for NavHistory and Cookie?

Hipp, will the fix get into bi-monthly update of sqlite3?

Thanks!

Comment 7

[D. Richard Hipp]

The "unix-excl" enhancement will be in SQLite version 3.7.6, which is scheduled for release on or about 2011-04-14.

Comment 8

[Shawn Wilsher :sdwilsh]

(In reply to comment #6)
> sdwilsh, where is the best place for "unix-excl"?
I'm not sure exactly what the differences are for non-WAL databases, and my answer would depend on that.

Comment 9

[D. Richard Hipp]

The unix-excl VFS arranges things so that only a single process at a time can access the database file, but that any number of separate database connections and/or threads within that one process can access the database file.  For connections and threads within the same process, none of the access rules have changed.  The only thing that does changes is that other processes cannot access the file.  This is true for both WAL and rollback-journal modes.

Comment 10

[Ginn Chen]

(In reply to comment #8)
> (In reply to comment #6)
> > sdwilsh, where is the best place for "unix-excl"?
> I'm not sure exactly what the differences are for non-WAL databases, and my
> answer would depend on that.

What is your answer now?

Comment 11

[Marco Bonardo [:mak] (Away 10-21 August)]

I think that based on current situation where we don't plan accessing the same SQLite database from multiple processes we may well move to unix-excl, will also improve our protection from other processes. Indeed I'd also like to have a win32-excl vfs :)
My only doubt is whether this may break Sync on mobile, now that it is a separate service, though we may directly ask the Sync team!

Comment 12

[D. Richard Hipp]

If mobile Sync is running in a separate process, then using unix-excl will definitely prevent mobile Sync from getting access to the database.  On the other hand, you probably are not using an NFS filesystem on mobile, so you can probably use the plain old "unix" VFS there, right?

We'll get to work on win32-excl for you...

Comment 13

[Marco Bonardo [:mak] (Away 10-21 August)]

(In reply to D. Richard Hipp from comment #12)
> On the
> other hand, you probably are not using an NFS filesystem on mobile, so you
> can probably use the plain old "unix" VFS there, right?

Correct, we may #ifdef android for that, though -excl sounds like a good idea generally, not just for nfs.
The other concern may be if third party apps may want multi-process access, though at that point would be better to add an optional flags param to openConnection API, also for future options.

Sdwilsh?

Comment 14

[D. Richard Hipp]

The unix-excl VFS allows multiple connections from the same process and only restricts connections from other processes.  Achieving the same thing will be quite difficult on windows.  Is having win-excl important, or is it just a "nice to have"?

Comment 15

[Marco Bonardo [:mak] (Away 10-21 August)]

just a nice to have.

Comment 16

[Richard Newman [:rnewman]]

Aww, you're going to take away my ability to attach sqlite3 to my places.sqlite so I can debug while I debug? :P

I believe that Sync sometimes or always runs inside Fennec's process, but in a different thread. I'd have to check.

But we delegate database access to Fennec's ContentProvider. That might keep control in Fennec's process regardless.

I'll try to remember to post back in this bug next time I have a log with pids.

Comment 17

[Shawn Wilsher :sdwilsh]

(In reply to Marco Bonardo [:mak] from comment #13)
> Correct, we may #ifdef android for that, though -excl sounds like a good
> idea generally, not just for nfs.
> The other concern may be if third party apps may want multi-process access,
> though at that point would be better to add an optional flags param to
> openConnection API, also for future options.
We've tended to not do that in the past because it means we break all C++ callers.  Maybe we care less about that in the new release world.

Comment 18

[Marco Bonardo [:mak] (Away 10-21 August)]

(In reply to Shawn Wilsher :sdwilsh from comment #17)
> We've tended to not do that in the past because it means we break all C++
> callers.  Maybe we care less about that in the new release world.

Well, actually all add-ons are marked compatible, but all add-ons with binary components, that are always marked not compatible. So their authors have to go through them regardless a change. It's likely they will recompile regardless.

Btw, I think a bitfield flag would also make openUnsharedConnection useless, we may have excl, unshared, dotfile and similar options just in openConnection.

Comment 19

[BMO Automation]

Per policy at https://wiki.mozilla.org/Bug_Triage/Projects/Bug_Handling/Bug_Husbandry#Inactive_Bugs. If this bug is not an enhancement request or a bug not present in a supported release of Firefox, then it may be reopened.