476167 - Expose the cookies database connection: Now extensions cannot access to the db "cookies.sqlite"

Status: RESOLVED WONTFIX

(Core :: Networking: Cookies, defect)

Description

[YUKI "Piro" Hiroshi]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.1b3pre) Gecko/20090129 Shiretoko/3.1b3pre
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.1b3pre) Gecko/20090129 Shiretoko/3.1b3pre

Cookies database is locked by nsCookieService so we cannot create new connections to the database. The places database has same problem, but now, a new interface "nsPIPlacesDatabase" introduced by the bug 449506 (Expose the places db connection) exposes the existing connection for our codes. For other databases, downloads.sqlite, content-prefs.sqlite and formhistory.sqlite, connections are already exposed (nsIDownloadManager, nsIContentPrefService, nsIFormHistory2). We need same solution for cookies too.


Reproducible: Always

Comment 1

[YUKI "Piro" Hiroshi]

s/The places database has same problem/The places database had same problem/

Comment 2

[Shawn Wilsher :sdwilsh]

This is likely WONTFIX, but it goes to cookies

Comment 3

[[e-mail]]

this is WONTFIX unless there's a really compelling reason...

hiroshi, you realize the cookies.sqlite database doesn't include session cookies, right? so to access the entire cookie store you need to go through nsICookieManager{2}'s APIs. can you explain what use case you have to access the database directly, rather than going through these APIs?

Comment 4

[YUKI "Piro" Hiroshi]

I'm developing an addon "XUL/Migemo" which provides a feature to find entries by regular expressions from the huge Places database and replaces the awesomebar result. It were very very slow if I implemented the feature via Places APIs.

Like this, we can find cookies by regexp without delay if the database is accessible via mozIStorageService APIs. Moreover, we can also provide find-ability based on their expiration date. For example we'll get ability to clear the far distant future cookies.

I also think developers of such addons have to solve session cookies problem, even if cookies.sqlite becomes readable. But it seems different problem.

To be honest, I'm not planning to develop addons around cookies in a few days. I filed this bug because an author of SQLite Manger project noticed about cookies.sqlite. (Issue 219: Getting error opening places.sql database in Firefox 3.1b2. http://code.google.com/p/sqlite-manager/issues/detail?id=219 ) But I think that APIs should keep accessibilities if doing it doesn't increase the cost. In other words, I don't think that connecting to cookies.sqlite must be forbidden.

Comment 5

[Shawn Wilsher :sdwilsh]

...except that there is a performance cost by opening it up.  This is why it's opened with an exclusive lock on an unshared connection.

Comment 6

[Domenic Denicola]

I am pretty sure this bug is related to the fact that, since Firefox 3.5, I have been unable to open the cookie database in the code for my third-party program.

That is, with Firefox 3.0, my .NET 3.5SP1 application could use the System.Data.SQLite library to open, read from, and write to cookies.sqlite. But as of Firefox 3.5 (well, tested only on Firefox 3.5.2), I get an error that the database is locked whenever I try to open it, if Firefox is open at the same time.

This is unfortunate for my application. I'll describe why my application does this, and maybe you guys will think it's legit and possibly un-lock the database, or maybe you'll (somewhat justifiably) say that Firefox is the only program that should have access to the cookies database.

My application is a screen-scraper, that automates certain tasks on a certain website. This website, of course, requires a login. Before the locking started happening, I was able to read the user's cookies for this website, and thus log them in automatically. I was also able to update the Firefox cookie database with any new cookies that the website gave me, so that the user's state with regards to this website was synchronized across Firefox and my application. So this was very convenient. Unfortunately now I can't even read the user's cookies, if the user has Firefox open---and, let's face it, most people have Firefox open 100% of the time their computer is on :).

Thanks for your consideration.

Comment 7

[Daniel Mitchell]

You can work around not being able to _read_ cookies by making a copy of cookies.sqlite and reading from that file instead. (http://osdir.com/ml/svn-commits-list/2009-07/msg00277.html appears to be another project using this solution).

Sadly, my app also needs to be able to delete cookies after reading them, which appears to now be impossible, because while I can delete from my copy of cookies.sqlite, they aren't removed from the original one.

Comment 8

[[e-mail]]

Providing access to cookies.sqlite while the browser is running is unsound for several reasons.

1) The cookieservice maintains its own in-memory table of cookies, and cookies.sqlite is used as an on-disk backing store for that table. Changing cookies.sqlite will break the state of consistency with the in-memory table, and will not have the desired effect of changing the in-memory table.

2) Reading from cookies.sqlite will not give you any information about session cookies, which is probably a bad thing, depending on application.

3) comment 4 indicates that access to the underlying DB is important for performance reasons. The fact that the cookieservice has an in-memory table means that access to it through the APIs is very fast. Even if you have to fetch the entire table with thousands of entries and iterate through it, such operations only take a few milliseconds. Performing a sqlite query would likely take just as long.

So I don't see the performance benefit here, but I do see downsides for the correct operation of the cookieservice. There are two solutions: a) change the cookieservice to perform all operations directly on the DB and make it a shared connection, or b) have everyone access the table via APIs. The former would be a huge performance regression, because database access is inherently slow. For the latter, if people have suggestions for improving the API, please make them. I'm all for making the API more useful and flexible. (Currently, you can ask for an enumerator of all cookies, or for cookies from a certain host.)

Comment 9

[Daniel Mitchell]

 Unless I'm missing something, the builtin nsICookieManager API won't help external applications; as with comment 6, our app runs _outside_ Firefox, so these APIs aren't any use to us.

Comment 10

[Shawn Wilsher :sdwilsh]

And as noted in comment 8, even if you had access to the cookie database, it wouldn't show you all the cookies.

Comment 11

[Daniel Mitchell]

 We don't need access to session cookies in our case, just normal ones, so if the builtin APIs were usable by external apps they would do what we need. 

 Otherwise, as comment 8 points out, the cookie file on disk doesn't accurately represent the cookies in memory, so even working with a copy of the file to avoid locking doesn't help.

 It's unfortunate that 3.5 breaks this, because solutions like this worked fine with all previous versions of FF/Mozilla/Netscape; IE and Safari provide actual APIs for external users to use, and Opera's cookie file is persisted, accurate, and documented, so currently FF 3.5 is the only browser with this problem. (unless Chrome has similar problems).