1170578 - RFC: Buggy NSS dbm code ought to be updated

Status: NEW

(NSS :: Libraries, defect, P5)

Description

[ISHIKAWA, Chiaki]

(This blocks bug 1170564: [META] Failure to deal with short read )

For details of short read, and how to fix, please read the bug
1170564.

Here in this bugzilla entry, it is pointed out that NSS DBM
code (Berkeley db 1.85) ought to be updated to a new one.

This code is used by both C-C TB and FF since it is under M-C
tree.  This code cannot cope with short read, and cause
errors. It should be updated with later versions with more
robustness and nice features such as shared DB access.  (It is
way too old: I think it is from the first half of 1990's).

This was initially posted in a newsgroup posting by me [2].

I found that if short read operation occurs against one of the
following files under my MOZOBJ directory during
|make mozmill| test, TB causes errors reported in the test log.

    "_tests/mozmill/mozmillprofile/cert8.db",
    "_tests/mozmill/mozmillprofile/key3.db",
    "_tests/mozmill/mozmillprofile/secmod.db"

And I found these files are managed by NSS dbm code and the
code inspection reveals that it is not safe from short read at
all.

In response to the posting, Philip Chee responded in the
newsgroup posting [3],
(line break positions have been changed somewhat.)

--- begin quote ---

According to Google, current versions of NSS can use SQLite
for its backend. Butt the file names are different:

key3.db -> key4.db
cert8.db -> cert9.db
secmod.db -> pkcs11.txt

https://wiki.mozilla.org/NSS_Shared_DB
https://blogs.oracle.com/meena/entry/what_s_new_in_nss1

Advantage is that Firefox/Thunderbird/SeaMonkey can all share
the same files
........

> However, now that Sleepycat has been bought by Oracle, I am
> not sure what is the good option left.

I think v5 of Berkeley DB is still under the Sleepycat
licence.
--- end quote ---

From my own testing, I can confirm SQLite seems to be solid in
handling short read/write among many advantages it has. My
emulation of short read/write was done against SQLite
databases during tests, and I have not seen problems caused by
them if I am not mistaken. SQLite library handles the
necessary additional reads automagically. The casual code
browsing I did many months ago also support this informally.

> Advantage is that Firefox/Thunderbird/SeaMonkey can all
> share the same files

Frankly, I have no background to comment on this. But maybe if
one needs to add local certificates due to organizational
issues, there will be one less additional database to take
care of.

Re version v5 vs. v6.

Oracle web page today has a link to v6 code.

http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/downloads/index.html

If I am not mistaken, v6 even claims to have SQL interface
based on SQLite (!).  I don't know if it is done via talking
to a separate engine or even incorporate the SQLite source
code inside.

Anyway, it looks replacing the current code with the new NSS
code/backend seems a BIG WIN in the long run.

What do people think?

In the meantime, the bug caused by the short read is for real
and acute.

TIA

[1] Bug 1170564 - [META] Failure to deal with short read

[2] The second posting "DBM code too old (Re: mozmill test:
    Hightail ???)", in the newsgroup thread:
    https://groups.google.com/forum/#!topic/mozilla.dev.apps.thunderbird/G5UJxVa17SY
    
[3] The third posting in the above newsgroup thread.

PS: I have created a small patch for DBM code before not knowing the extent of the problem, but I think now it should be ditched and replaced with a new version.

Comment 1

[Wayne Mery (:wsmwk)]

Is there not already a bug about doing sqlite?
I had a brief look around but didn't find one that's open.

Comment 2

[Jeremy Andrews]

Is there any chance y'all would be interested in replacing dbm with something like LMDB (or possibly Kyoto Cabinet, LevelDB, etc) rather than simply removing it? The problem with SQLite being the only option is that applications integrating NSS that are designed around using dbm might find it prohibitively difficult to migrate to SQLite, and there are a few problems with it in situations that aren't SQL-friendly. Key value databases and relational databases are somewhat different animals.

I completely understand if y'all don't find it feasible to maintain support for two types of databases, but LMDB in particular scales very well, and is what most projects that have moved on from dbm seem to have chosen as a replacement. A high-performance, somewhat dbm-compatible replacement for key-value storage applications would be very welcome.

We might be able to produce some patches for it for our own internal use, but we'd really rather work with upstream on this if possible given that we're concerned there may be security risks involved in trying to configure NSS to use an alternate database downstream.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

In the process of migrating remaining bugs to the new severity system, the severity for this bug cannot be automatically determined. Please retriage this bug using the new severity system.

Comment 4

[ISHIKAWA, Chiaki]

(In reply to Jeremy Andrews from comment #2)

Is there any chance y'all would be interested in replacing dbm with something like LMDB (or possibly Kyoto Cabinet, LevelDB, etc) rather than simply removing it? The problem with SQLite being the only option is that applications integrating NSS that are designed around using dbm might find it prohibitively difficult to migrate to SQLite, and there are a few problems with it in situations that aren't SQL-friendly. Key value databases and relational databases are somewhat different animals.

I completely understand if y'all don't find it feasible to maintain support for two types of databases, but LMDB in particular scales very well, and is what most projects that have moved on from dbm seem to have chosen as a replacement. A high-performance, somewhat dbm-compatible replacement for key-value storage applications would be very welcome.

We might be able to produce some patches for it for our own internal use, but we'd really rather work with upstream on this if possible given that we're concerned there may be security risks involved in trying to configure NSS to use an alternate database downstream.

Obviously I am not the right person to ponder on these topics. Someone in the know re DB ought to make a policy decision for this.