245745 - use sqlite instead of mork for history back end

Status: RESOLVED FIXED

(Firefox :: Bookmarks & History, enhancement)

Description

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

In addition to indexing, one doesn't have to reimplement SQL in open coding
(f.e., groupby), and there are a variety of tools available for poking at
sqlite.  It's also multi-process/thread safe, when used correctly, and doesn't
have to be maintained by us.  I think it's a win, and Ben prodded me to hack up
something appropriate.

Interim patch coming up

- autocomplete works, though doesn't have quite the right semantics WRT 
  explicitly-typed prefixes yet
- history pane works, in all the views I could see
- RDF interface mostly works, except
  - GetSources
  - notifications
  I'm going to do the latter with triggers, and the former is just a matter of 
  spending the time.

You'll need to have sqlite-devel installed, and be building on Unix or else make
appropriate arrangements in build/Makefile.in.

The code got a bit simpler, too:
 3 files changed, 893 insertions(+), 2586 deletions(-)

Comment 1

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Attachment: trunk diff for firefox

I guess I should move this to the aviary branch, too...

Comment 2

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Attachment: aviary patch for firefox

This catches up to the aviary branch, uses a precompiled query for checking if
a URL is already in the history, fixes some very silly TRIGGER bugs, and
abandons the fragile-and-not-clever-enough update-or-insert logic in
AddPageToDatabase.

With either of these patches, by the by, you will want to blow away your
history file first.  I'll put some migration in later, as well as a facility
for schema alteration.

Comment 3

[Ron Baldwin]

I don't suppose it's possible to get this into 1.0?  If not, any idea of the
priority the devs put on this?  

I ask because it currently takes over 30 sec (!) to open the Go menu with the
size of my history file and I have a really don't want to have to wipe out my
history.

Comment 4

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

It's certainly not 1.0 material, especially since this patch needs to be
reworked to use sqlite3 for i18n-friendliness, and then the RDF notification
pieces need to be wired up more effectively.

In terms of developer priority, there is one very active Firefox developer who
is basically full-time on issues related to this and other efficient, shared
store issues for the post-1.0 roadmap.  So your pain, with which I do indeed
sympathize, should have an end in sight.

Comment 5

[Worcester12345]

(In reply to comment #4)
> It's certainly not 1.0 material, especially since this patch needs to be
> reworked to use sqlite3 for i18n-friendliness, and then the RDF notification
> pieces need to be wired up more effectively.
> 
> In terms of developer priority, there is one very active Firefox developer who
> is basically full-time on issues related to this and other efficient, shared
> store issues for the post-1.0 roadmap.  So your pain, with which I do indeed
> sympathize, should have an end in sight.

Could you point to this roadmap which references these store issues? Thanks.

Comment 6

[Aleksey Nogin]

Is this going to happen for FF 1.1? Things are _so_ painful right now, when you
have a large history file...

Comment 7

[Worcester12345]

(In reply to comment #6)
> Is this going to happen for FF 1.1? Things are _so_ painful right now, when you
> have a large history file...

It was marked to block 1.1 and then unmarked, so I'd say no. :-(

Comment 8

[Mike Goodspeed]

taking

I am going to be busy with moving to chicago, but would like to go ahead with
the migration when I get the chance.

Comment 9

[Robert Relyea]

My I request that the sqlite we build has a standard name (like libsqlite3.so or
sqlite3.dll) rather than be subsumed in a mozilla storage package.

The reason for this is basically one of sharing code. NSS would like to
eventually use sqlite3 for it's database. I would like to not have to build
sqlite3.dll in NSS itself, however NSS needs to be build standalone (without
mozilla), as it has many non-mozilla customers (mostly servers). As part of this
goal, I'd like to build NSS so it can use the system sqlite if it exists on the
target platform, but still be able to build sqlite for platforms that do not
already supply sqlite.

Thanks,

bob

Comment 10

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

We could support using a system sqlite, if desired, and if the system sqlite had
all the fixes and such that we need (including future encryption patches, etc.).
 Please file another bug on that, because it has nothing to do with porting
history over.

Comment 11

[Daniel Mota Leite]

what is the status of this bug?
what is bloking its inclusion?

current history database makes FF block for some seconds several times, so we should change this as fast as possible and solve several bugs (dups?) all at once.

Comment 12

[Worcester12345]

How does "places" being out affect (or not) this bug?

Comment 13

[Worcester12345]

How does "places" coming out affect (or not) this bug?

Comment 14

[Henrik Gemal]

cant this one be closed? Places took over?

Comment 15

[Brett Wilson]

Fixed by bug 266174.