Closed Bug 402615 Opened 17 years ago Closed 17 years ago

mozStorageService not only ACI, not Durable

Categories

(Core :: SQLite and Embedded Database Bindings, defect)

Product:
Core
Component:
SQLite and Embedded Database Bindings
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: dmosedale, Unassigned)

References

(
URL
)

Details

Back in bug 326334, mozStorage traded away durability for performance; write calls return immediately, but the actual writes happen on a background thread in their own time. <http://mxr.mozilla.org/mozilla/source/storage/src/mozStorageAsyncIO.cpp#40> discusses the design in detail. If there were some way to get durability back and still get the performance we need, it would be Really Spiffy (TM).
Even if we can't initially win back all the performance, it might well be worthwhile to make it possible for the embedding to initialize mozStorage without the async I/O layer. I suspect calendar and mail embeddings would want this.
Blocks: 398912, 408072
Version: unspecified → Trunk
Ondrej - if no one is going to pick this up, can you take a look? The first step would be to analyze what the performance hit is of turning off async I/O. Also, do you think that the async I/O layer might lead to database corruption if writing is interrupted somehow. There have been reports of this happening on network drives.
Maybe Bug 404695? It's unclear what has caused that one...
Also - can we look into using a worker thread + SQLite 3.5.4 to get equivalent read/write perf without async i/o? Ondrej you up for that :-)?
I would like to help with bug 385245 first, this spends a long time in database when showing history sidebar. When this is done, then I would first try what Matt suggested, turning off async I/O and seeing where are the problems are and if they cannot be fixed in other way, if not, then we should try what Mike suggests.
(In reply to comment #5) > Also - can we look into using a worker thread + SQLite 3.5.4 to get equivalent > read/write perf without async i/o? Ondrej you up for that :-)? I don't think that's a good general solution, but we could do that in hot spots where we see perf issues.
(In reply to comment #7) > (In reply to comment #5) > > Also - can we look into using a worker thread + SQLite 3.5.4 to get equivalent > > read/write perf without async i/o? Ondrej you up for that :-)? > I don't think that's a good general solution, but we could do that in hot spots > where we see perf issues. > Why not?
Because automagically doing things on a worker thread still cans durable. Calls that write to the database are expected to block until the write is successful, and if it wasn't, the consumer should know immediately. If we have a worker thread doing all the writes, we haven't actually changed anything from our current situation. We could try and create some complicated callback system, but I really think the right way to fix this is to have things that use mozStorage and are perf sensitive to have a worker thread where they can commit things to the database. If there's a situation where they want to make sure something made it to the DB, then they can still use blocking calls - otherwise they can pass it off to some worker thread. Now, with that said, we may be able to make this easier by providing some class that can proxy these calls to another thread. This all seems like some risky work this late in 1.9 though.
(In reply to comment #9) > Because automagically doing things on a worker thread still cans durable. > Calls that write to the database are expected to block until the write is > successful, and if it wasn't, the consumer should know immediately. If we have > a worker thread doing all the writes, we haven't actually changed anything from > our current situation. We could try and create some complicated callback > system, but I really think the right way to fix this is to have things that use > mozStorage and are perf sensitive to have a worker thread where they can commit > things to the database. If there's a situation where they want to make sure > something made it to the DB, then they can still use blocking calls - otherwise > they can pass it off to some worker thread. > > Now, with that said, we may be able to make this easier by providing some class > that can proxy these calls to another thread. This all seems like some risky > work this late in 1.9 though. > So we might be saying the same thing. My understanding of the order of ops right now: App-> sqllite-> asynicIO e.g. sqllite thinks a transaction has committed but it is not actually written to disk. I was suggesting-> App-> AppWorkerThread-> sqllite-> syncIO So any areas in the app where perf is senstive we do explicit threading/async rather than having sqllite magically return quickly even tho things are not done. In teh meeting I also suggested we make writes sync (to prevent corruption) and leave read's async..
I'm just not sure we want to automagically do that anymore. People tend to not expect that that with a database API. We can certainly provide helper methods/class that can dispatch it to a worker thread though.
Fixed by the backout of async stuff.
Status: NEW → RESOLVED
Closed: 17 years ago
Resolution: --- → FIXED
Fixed by the backout of async stuff.
Depends on: 408914
Product: Toolkit → Core
You need to log in before you can comment on or make changes to this bug.