Open Bug 1572000 Opened 1 year ago Updated 1 month ago

[meta] database backed global message index

Categories

(MailNews Core :: Database, task, P3)

Tracking

(Not tracked)

People

(Reporter: mkmelin, Unassigned)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

(Keywords: meta)

Currently folder indexing uses Mork, and message index is per folder. Mork removal has been wanted for a long time (bug 453975), but when working on removing it from usage for the folder index we should also take the opportunity to design it correctly to use a global message index.

This would enable a conversation view of messages (which currently requires gloda, and gloda is not meant for that).

It would make issues like bug 43278 go away.

I belive for Gmail, we're downloading the same messages multiple times, because we don't know it's already in All Mail (duh!).

We need to figure out how storage of the actual message data should be handled: put it in the database, or keep it on the file system, or a combination where normalized/decoded content would be put into the database for quick searching and indexing and the raw data would be kept only for backup.

Primarily I think we should target IndexedDB for database, since that is the web thing to do.

Blocks: 453975
Priority: -- → P3

Any reading material on the IndexedDB?

https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API is one. If you want the spec, see http://w3c.github.io/IndexedDB/

In a nutshell, if you need a database and want to be using web technologies, this is what you need to use. Amongst the more important features, it can be used from Web Workers, so for instance you can have background workers fetching your mail from the server and putting them into the database, with no need to steal processing time from the UI thread. For all other solutions (like sqlite) you'd have to jump through multiple hoops to get that working in any kind of hacky way (if at all).

Do you plan to do a prototype to check the performance of the new solution? Mork is old and ugly, but it can handle huge folders in a reasonable time.

Yes we would have to ensure the performance is acceptable.
A complete one-to-one comparison to current state will be more difficult since current state isn't dealing with global, but if you keep it one folder only for comparisons, that should work.

Depends on: 11050

This is at the same time a good idea and a totally wrong idea.

Indeed, we do need to be able to index all messages.
And we do need to be using a Database.

But - the entire separation between "the messages", "an index for the messages" and "a backing database" with Thunderbird marrying them together or using one for the other - is myopic.

Messages should simply go in a message database. That's it. Enough said, full stop.

No more saying, "Oh, we need search ability X" or "indexing ability Y" - we just need a DBMS, a software system which has all that stuff. You put your messages in, and you're done (up to having effective access to its contents and facilities). I'm not saying existing document-oriented DBMSes have all the features we need - maybe they're missing some - but we definitely need lots and lots of what they already have.

You need to log in before you can comment on or make changes to this bug.