places.sqlite created when I click on a link in a message. wastes disk space

RESOLVED FIXED in Thunderbird 28.0

Status

Thunderbird
General
RESOLVED FIXED
7 years ago
4 years ago

People

(Reporter: Eric Moore, Assigned: rsx11m)

Tracking

(Blocks: 2 bugs)

Thunderbird 28.0
All
Other
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

7 years ago
User Agent: Mozilla/5.0 (Windows NT 6.0; rv:6.0) Gecko/20100101 Firefox/6.0
Build ID: 20110713171652

Steps to reproduce:

places.sqlite is used by Firefox to store bookmarks and browsing history. I have the same file in my Thunderbird profile and I noticed its size has increased from less than 500KB to over 10MB after upgrading to Thunderbird 5. This made me suspect Thunderbird uses the same file for browsing history. I don't want that.

I added a browser.history_expire_days_min setting in the config editor and set it to 0 to disable keeping the browsing history (just like I have with Firefox 6). I then exited Thunderbird, deleted the places.sqlite file and restarted. It created a replacement 10,240KB places.sqlite file when I restarted Thunderbird, without my even clicking on a link. I then exited and deleted places.sqlite, places.sqlite-shm and places.sqlite-wal and restarted. It did not create any places.* files. I then clicked on a link in a message and it recreated the files (same size).

I do not have global search/indexing or offline folders (synchronization & storage) enabled. Other than the quick filter toolbar I am basically trying to use Thunderbird 5.0 as if it was Thunderbird 2.0.0.24 with less bugs. I don't use any add-ons like ThunderBrowse or G-Hub Lite that do any sort of browsing.

https://bugzilla.mozilla.org/show_bug.cgi?id=608981 states:
"To move the Thunderbird build closer to Firefox, and to easily enable interesting things like favicons for content tabs, we should start building places. Extensions should also be able to use it if they wish to.

I can't see a reason at the moment for us to enable storing of history by default, so I think we should disable it by default but keep reasonable defaults in place. I believe that favicon storing/usage will still work even if the history is disabled (I think it relies on other prefs)." 

1. The file didn't exist until I clicked on a link in a mail message. How do I guarantee that history is not being recorded? It sure looks suspicious and its not reasonable to expect a normal user to install a SQL manager to investigate the contents of the file.

2. This might be an issue for roaming profiles on large networks. Its creating 10.5MB of wasted space per user (if its just there for potential future use).

3. Most of the add-ons I use don't support 5.0, I have to use the add-on compatibility reporter add-on in order to use them. I don't expect this to change since Thunderbird add-on developers in general had problems keeping up with the old schedule (the new rapid development schedule just makes things worse), and none of the add-ons I use store much data. So, why do I have to pay the cost for something I won't use? One of the reasons why I do not use global search is IMHO its not mature enough, I've seen too many threads where the size of the globl-message-db.sqlite file grows out of control and it has to be deleted. I'm concerned the same type of thing might happen some day with places.sqlite.  There should be a optional setting in the config editor to prevent that file from being created.
jcranmer@xochiquetzal ~/.thunderbird/3w40q3cd.default $ ls -l places.sqlite 
-rw-r--r-- 1 jcranmer jcranmer 10485760 Jun 17 10:13 places.sqlite
jcranmer@xochiquetzal ~/.thunderbird/3w40q3cd.default $ sqlite3 places.sqlite 
SQLite version 3.7.7 2011-06-23 19:49:22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .tables
moz_anno_attributes  moz_favicons         moz_keywords       
moz_annos            moz_historyvisits    moz_places         
moz_bookmarks        moz_inputhistory   
moz_bookmarks_roots  moz_items_annos    
sqlite> SELECT * FROM moz_anno_attributes;
sqlite> SELECT * FROM moz_annos;
sqlite> SELECT * FROM moz_bookmarks;
1|2||0|0||||1308330831249496|1308330831282878|cFotiO-sL9-F
2|2||1|0|Bookmarks Menu|||1308330831250583|1308330831280560|L_qidTrk5TU7
3|2||1|1|Bookmarks Toolbar|||1308330831281009|1308330831281688|TWeA_ErXnG6W
4|2||1|2|Tags|||1308330831281798|1308330831282467|_-xpCTXkScO0
5|2||1|3|Unsorted Bookmarks|||1308330831282878|1308330831283540|ze8qll1sPTVo
sqlite> SELECT * FROM moz_bookmarks_roots;
places|1
menu|2
toolbar|3
tags|4
unfiled|5
sqlite> SELECT * FROM moz_favicons;
sqlite> SELECT * FROM moz_historyvisits;
sqlite> SELECT * FROM moz_inputhistory;
sqlite> SELECT * FROM moz_items_annos;
sqlite> SELECT * FROM moz_keywords;
sqlite> SELECT * FROM moz_places;

So the places file is effectively completely empty.

Comment 2

7 years ago
yup. I did an external compact, but I don't know if it will stick.

Comment 3

7 years ago
Is this unconfirmed?

Comment 4

7 years ago
I've actually just stumbled upon this. It's a killer for roaming profiles because it creates an empty 10MB file that will have to be redistributed among the network(s) for every single user. We're talking about 2.5GB for a mere 250 users. Wayne, would you consider this relevant for Bug 564148?

Updated

7 years ago
See Also: → bug 676247

Comment 5

7 years ago
(In reply to Martin Jungowski from comment #4)
> I've actually just stumbled upon this. It's a killer for roaming profiles
> because it creates an empty 10MB file that will have to be redistributed
> among the network(s) for every single user. We're talking about 2.5GB for a
> mere 250 users. Wayne, would you consider this relevant for Bug 564148?

for a "smallish shop" not much space in the grand seem of things, but still a nuisance.  but at numbers approaching or over 10k users we're talking 100GB. And all the backup time and whatnot (if the chose to backup roaming profiles). so yeah
Blocks: 564148
Summary: places.sqlite created when I click on a link in a message → places.sqlite created when I click on a link in a message. wastes disk space

Comment 6

6 years ago
The size of places.sqlite in my thunderbird directory is 10MB and in firefox 41MB. There is nothing I want done that requires that amount of information about my usage to be stored without my requesting it explicitly. It causes backups to be bloated too. These huge files should not be created without explicitly asking the user's permission and explaining their purpose (compare the cache preferences, and cache-clearing options). I suspect ulterior motives to do with the needs of commercial organisations. I use thunderbird as an occasional alternative to pine, and firefox as my main browser both on linux desktop machine and my linux laptop. I shall now investigate less intrusive alternatives. I occasionally use seamonkey, but now I find it also has a large places.sqlite file that I did not give it permission to create.

Comment 7

6 years ago
Hey, the places.sqlite file contains your browsing history. It is not sent to/used by Mozilla.org in any way.
You will not find better alternatives, all browsers do this, as it is a standard part of browser operation.

If you still have problem with it, just disable it:

In Firefox, just go to Tools->Options->Privacy->History and disable what you want. You can even enable Private browsing mode.
In Thunderbird just don't browse (click on links in emails) and there will be no history. Or go into Tools->Options->Advanced->General->Config editor and search for history.

The bug here is that even if you don't browse the file sqlite.places still has a minimum of 10MB. But it is EMPTY (no data)! So do not jump to premature conclusions.
(Assignee)

Comment 8

6 years ago
(In reply to :aceman from comment #7)
> In Thunderbird just don't browse (click on links in emails) and there will
> be no history. Or go into Tools->Options->Advanced->General->Config editor
> and search for history.

The point of this bug report is that a 10MB almost empty places.sqlite is created even if it is *not* used, which are carried around unnecessarily in the profile.

I don't know if the originally described behavior for 5.0 still exists in the same way, but as stated, even modifying browser.history preferences didn't help.
(Assignee)

Comment 9

6 years ago
BTW: "places.history.enabled" is false already by default for Thunderbird, thus the underlying problem is that the sqlite file is created despite the feature which it is used for has been disabled.

Updated

6 years ago
Status: UNCONFIRMED → NEW
Ever confirmed: true
(Assignee)

Updated

5 years ago
Duplicate of this bug: 944833
(Assignee)

Comment 11

5 years ago
(Quoting Marco Castelluccio [:marco] from bug 944833)
> We've reduced the default places.sqlite size for webapps in bug 762083, we
> could do the same in Thunderbird (it's just a matter of defining a new pref).
(Assignee)

Comment 12

5 years ago
Created attachment 8341921 [details] [diff] [review]
Set database increment to zero

This does the job for me and seems to be the reasonable thing to do. With the increment set to zero, the initial size in a new profile goes down from the previous 10485760 to just 1146880 bytes (1120KB) on Linux x86_64, which may be the minimum size we can get with no content actually filled (but with the places database structure and tables established).

Note that this should affect only /new/ profiles, not any places.sqlite file in already established profiles (i.e., the 10MB size would be retained there until you delete the file prior to a restart). If migration code is wanted, I wouldn't know how to do that, but it seems tricky given that an extension may have actually used the places feature (in which case the file can't just be truncated or removed without verifying that it's actually empty).

On a side note, without that patch to set the preference, the default of 10MB should be maintained. However, in that case I only get 1179648 bytes, which is 32KB more (even) compared with the preference setting set to zero. Anyway, if that's reproducible in Firefox it should be a separate Toolkit bug.
Attachment #8341921 - Flags: review?(mkmelin+mozilla)

Comment 13

5 years ago
Comment on attachment 8341921 [details] [diff] [review]
Set database increment to zero

Review of attachment 8341921 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM, thx! r=mkmelin
Attachment #8341921 - Flags: review?(mkmelin+mozilla) → review+
(Assignee)

Comment 14

5 years ago
Thanks for the quick review, push for comm-central please.
Assignee: nobody → rsx11m.pub
Status: NEW → ASSIGNED
Depends on: 762083
Keywords: checkin-needed
https://hg.mozilla.org/comm-central/rev/54799131e042
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 28.0
(Assignee)

Updated

4 years ago
Blocks: 1038296
You need to log in before you can comment on or make changes to this bug.