345055 - Feature-Request: put history into some RDBMS or use some external program

Status: RESOLVED DUPLICATE of bug 382187

(SeaMonkey :: General, enhancement)

Description

[Enrico Weigelt]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20050920
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20050920

I'd like to have the possibility to add the history into some relational (SQL) database (ie. vie libpq or unixodbc) or let an external program (ie. some perl script) do the actual access task.

The SQL approach could look like that:

+ the user specifies the datasource credentials (ie. unixodbc dsn, perl::dbi style or pear::db url) and some table/view name.
+ now the history is expected to be in some table/view of an structure like: ctime,mtime,atime,url.
+ all operations are done via trivial SQL queries.

On the other hand the external app could work like this:

+ an command given by the user is executed on an pipe/socket.
+ we have a small set of trivial query commands: ie. 
++ "LIMIT (ctime|atime|mtime|host|prefix)"
++ "RESET", "LIST [<start> [<max>]]"
++ "TOUCH {(ctime|atime|mtime)} <url> [<timestam>]]"
++ "DELETE (URL|PREFIX|OID)", ...)
+ the LIST command replies simply a list of URLs and their additional attributes  (containing an unique OID), maybe as CSV or something similar, and some terminator tag.
+ the other commands reply either "OK" or some error tag

The implementation behind is completely left to the user - he can decide by his own what to put there.



Reproducible: Always

Comment 1

[Enrico Weigelt]

Maybe the history could be put into the same database as the history, as suggested in bug 345058.

Comment 2

[Brian Polidoro]

The mozilla platform is already moving in this direction. SQLite is the technology  being used for storage.  A starting place for information would be:
http://wiki.mozilla.org/Mozilla2:Unified_Storage

Comment 3

[Anders Conbere]

This will be mostly done with places in firefox 3 where they've implimented an sql lite backend.

Comment 4

[Enrico Weigelt]

Please have a look at my comments on the bottom of the wiki article:
I don't think such an grand unified storage engine would be good, neither I like to be bound to SQLite.

Comment 5

[Anders Conbere]

I don't think that there is any way that you are going to convince people that adopting a free form backend for unified storage is a good idea.  Not only would it be overly complicated for users but it would require a signifigantly larger amount of work on the part of developers to make a db agnostic access system.

As laid out on the wiki SQL Lite was chosen for several reasons
1) it has sql syntax already
2) it has a small footprint

You can always write a perl application to access said DB pull info and dump to postgress if that's something that you want to do.  For the average user having an agnostic backend is simply not something that will be beneficial.

Comment 6

[Enrico Weigelt]

(In reply to comment #5)
> I don't think that there is any way that you are going to convince people that
> adopting a free form backend for unified storage is a good idea.  Not only
> would it be overly complicated for users but it would require a signifigantly
> larger amount of work on the part of developers to make a db agnostic access
> system.

I don't see the real problem. 
We could easily start w/ things like history: 

Introduce an simple API for accessing tabular data: simply open an datasource by its name and get back an handle. With this descriptor you can set up an query with an minimalistic query / filtering language (ie. "date='today' host='www.metux.de') and get back an cursor handle. Once you have it, you can fetch data, scan or even reload the whole thing (aka. re-execute query).
If you're done with it, close the descriptor. 

What actually sits behind, is defined in the global configuration. Standard installations will be configured to use the traditional datasources (ie. history.dat), but tweaking several datasources to somewhere else would be easy.

> As laid out on the wiki SQL Lite was chosen for several reasons
> 1) it has sql syntax already
> 2) it has a small footprint
> 
> You can always write a perl application to access said DB pull info and dump to
> postgress if that's something that you want to do.  

You're joking ?!
This does *not* solve *any* of my problems. I need to have the bookmark and history data *within* the postgresql database, so several mozilla instances from different hosts will see each other's changes immediately. You don't want to suggest polling every second, don't you ?

*If* we already move to SQL, why not iODBC or unixODBC and use SQLite as the default backend ?

Comment 7

[Enrico Weigelt]

Attachment: example how the askethic database API could look like.

Comment 8

[Enrico Weigelt]

Another approach could be simply storing evrything in an very simple filesystem hierachy. The actual filesystem access is done via libmixp, connecting to an 
small 9P server (which may either be spawned automatically via an pipe/socket or connected via tcp). Now the 9P servlet is responsible for actual storage (ie. direct local filesystem, traditional history.db file, sqllite, ldap, ...).

Comment 9

[Tony Mechelynck [:tonymec]]

I think nothing is going to stop history (and bookmarks) from being kept as <profile>/places.sqlite -- and it looks like http://developer.mozilla.org/en/docs/Places might be a good "starting point" for info about that. Does this "resolve" this bug or does it need some additional software?

Changing from Version=unspecified to Trunk, and from PC/Linux to All/All.