Last Comment Bug 754671 - [Page Thumbnails] size of thumbnails directory (in profiles directory) keeps growing infinitely
: [Page Thumbnails] size of thumbnails directory (in profiles directory) keeps ...
Status: RESOLVED FIXED
: relnote
Product: Firefox
Classification: Client Software
Component: Tabbed Browser (show other bugs)
: Trunk
: All All
: -- normal with 14 votes (vote)
: Firefox 17
Assigned To: Tim Taubert [:ttaubert] (on PTO, back Aug 29th)
:
Mentors:
https://wiki.mozilla.org/Privacy/Revi...
: 756598 760633 773893 (view as bug list)
Depends on: 987551 752407 764436 779445 784558 784868 785436 790649 794804
Blocks: 704882 744388 753768 762094 775540
  Show dependency treegraph
 
Reported: 2012-05-13 07:33 PDT by Jo Hermans
Modified: 2014-03-25 01:51 PDT (History)
53 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
+
unaffected
-
fixed


Attachments
patch v1 (17.00 KB, patch)
2012-06-07 06:46 PDT, Tim Taubert [:ttaubert] (on PTO, back Aug 29th)
no flags Details | Diff | Splinter Review
patch v2 (19.08 KB, patch)
2012-06-07 08:19 PDT, Tim Taubert [:ttaubert] (on PTO, back Aug 29th)
no flags Details | Diff | Splinter Review
patch v3 (18.44 KB, patch)
2012-06-07 15:54 PDT, Tim Taubert [:ttaubert] (on PTO, back Aug 29th)
no flags Details | Diff | Splinter Review
patch v4 (19.43 KB, patch)
2012-06-18 07:34 PDT, Tim Taubert [:ttaubert] (on PTO, back Aug 29th)
dao+bmo: feedback+
Details | Diff | Splinter Review
patch v5 (19.92 KB, patch)
2012-07-19 07:31 PDT, Tim Taubert [:ttaubert] (on PTO, back Aug 29th)
no flags Details | Diff | Splinter Review
patch v6 (19.97 KB, patch)
2012-07-23 03:37 PDT, Tim Taubert [:ttaubert] (on PTO, back Aug 29th)
no flags Details | Diff | Splinter Review
patch v7 (20.06 KB, patch)
2012-07-25 06:14 PDT, Tim Taubert [:ttaubert] (on PTO, back Aug 29th)
no flags Details | Diff | Splinter Review
patch v8 (20.10 KB, patch)
2012-08-01 03:09 PDT, Tim Taubert [:ttaubert] (on PTO, back Aug 29th)
no flags Details | Diff | Splinter Review
patch v9 (20.38 KB, patch)
2012-08-17 17:06 PDT, Tim Taubert [:ttaubert] (on PTO, back Aug 29th)
felipc: review+
akeybl: approval‑mozilla‑aurora+
Details | Diff | Splinter Review
backport for Aurora (20.44 KB, patch)
2012-08-21 13:03 PDT, Tim Taubert [:ttaubert] (on PTO, back Aug 29th)
no flags Details | Diff | Splinter Review
follow-up to rename old thumbnail folder instead of removing it (1.55 KB, patch)
2012-08-21 16:46 PDT, Tim Taubert [:ttaubert] (on PTO, back Aug 29th)
felipc: review+
Details | Diff | Splinter Review
backport for Aurora (20.43 KB, patch)
2012-08-21 16:49 PDT, Tim Taubert [:ttaubert] (on PTO, back Aug 29th)
felipc: review+
akeybl: approval‑mozilla‑aurora-
Details | Diff | Splinter Review

Description Jo Hermans 2012-05-13 07:33:00 PDT
Since the thumbnails were moved from the cache to the file system, they have become much more visible. At this moment, I have 130MB of images in 1983 files. That's after *only* 10 days.

It's not a new issue, but previously they were hiding among tens of thousands of entries in the 1GB cache, and cleared when that cache got cleared (automatically or manually). They could also have been pushed out of that cache by other content, but that's not happening anymore. They could also have been deleted by the regular expiration algorithm, but since the current history can keep entries for more than many months, that's not an option either. Also note that they are questions about privacy (there's no reason to store images if they're never going to be displayed again).

I do understand that the thumbnails do not only contain the 9 images that have to present on the new-tab page, but that it also contains the images that are needed in panorama. You can also not write an algorithm that would only remember the top 9 sites, and discard the rest.

So what can be our options ?
- at startup, discard all except the top-N sites ? In case people are removing tabs from the new-tab page, we need N to be larger than 9.
- discard images that are older than X days ? Maybe skip their deletion if they're still in the top-N list.
- if a tab is deleted, we should also delete the thumbnail, except if they're in the top-N list
- there could be also reasons why some sites are not stored in the cache, like mentioned in bug 754608.
Comment 1 Girish Sharma [:Optimizer] 2012-05-15 01:15:07 PDT
From comment 54 of bug 744388 [https://bugzilla.mozilla.org/show_bug.cgi?id=744388#c53]

Due to this feature, the thumbnails folder in my profile directory is of the size range of 350 MB.

Why does each of the 9 folders in the thumbnails folder contain more 9 folders ? Please someone explain.

IMO, Firefox should delete old thumbnails to reduce the size of the folder.

Maybe on each Firefox shutdown, the thumbnails that are not being shown on the new tab page should be deleted. This way, there would be only 9 thumbnails when the browser is closed.

Images should be discarded on browser shutdown and not start-up (so as to prevent any start-up lag). So on next browser start, only 9 thumbnails would be present in the thumbnails folder under profile directory. In fact, any more thumbnails are not even required as if the user removes any tab from the new tab page grid, its thumbnail will be automatically generated when the user visits that page again.
Comment 2 Ben Bucksch (:BenB) 2012-05-15 13:50:12 PDT
In an extension, I was storing the images in places in a private field. I also only saved screenshots of pages that are in the top 20 most-visited list (top 20 and not top 9, so that I'm sure to have a screenshot at the moment when the page's rank rises above 10 and it enters the newtab page). This already seriously limits the number of thumbs to 20-25. I would refresh the thumbnail once a day, whenever the user visits the page anyway. To delete thumbs of pages that sink in rank, I was setting expireStorageAfter = Ci.nsIAnnotationService.EXPIRE_WEEKS; . Together with the refresh, that ensured that we only keep a small number of thumbs.
Comment 3 Virtual_ManPL [:Virtual] - (ni? me) 2012-05-16 00:37:57 PDT
We should only store 9 screenshots for thumbnails for privacy and security reasons.
Comment 4 Matthew Turnbull [Bluefang] 2012-05-19 21:35:56 PDT
(In reply to Jo Hermans from comment #0)
> So what can be our options ?

Here's my suggestion:

* Keep thumbnails for new-tab entries, and for some N additional potential new-tab sites (in case the user removes some entries).

* Keep thumbnails for open and recently navigated-away/closed tabs. When a tab is removed from the navigation/undo-close-tab cache, also remove the thumbnail (provided it isn't a reserved new-tab thumb). Of course, this could just be simplified to remove the thumb when the tab is closed.

* On shutdown, if the user has selected to clear the cache, also clear the entire thumbnail cache.

* On startup, after the session has been restored, asynchronously evict all eligible thumbs. Since the cache should be kept fairly clean while running, this should not amount to much disk I/O.


Another matter that I think needs to be addressed is the thumbnail size its self. They're being rendered at 640x400. Even with a very large screen (1920x1200), the about:newtab thumbs are only 480x270. The tab-drag thumbs only look to be ~350x250. The panorama tabs can be scaled ridiculously huge, but I'd much rather see those limited to a sane size (and maybe deal with some image up-scaling) rather than have enormous thumbs saved to disk.

Although, if the cache is change to only ever contain a limited number of entries (i.e. the set of open tabs and about:newtab entries), then the physical size issue becomes somewhat moot.
Comment 5 Gian-Carlo Pascutto [:gcp] 2012-05-20 10:08:53 PDT
*** Bug 756598 has been marked as a duplicate of this bug. ***
Comment 6 Antti Tervasmäki 2012-05-24 12:13:11 PDT
I am wondering why in this bug is mentioned 9 folders?
In my thumbnails folder is 16 folders each containing 16 folders...
"Total space occupied: 1 149 360 956 bytes in 7 322 file(s),
in 272 directories" says Total Commander...
Comment 7 Jo Hermans 2012-05-24 12:56:39 PDT
Not 9 folder, but 9 images. The about:newtab page has 9 thumbnails in a 3x3 grid.
Comment 8 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-06-01 12:44:35 PDT
*** Bug 760633 has been marked as a duplicate of this bug. ***
Comment 9 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-06-07 06:46:32 PDT
Created attachment 630950 [details] [diff] [review]
patch v1

Here's a working patch.

So the idea is the following: every 10 minutes (or whatever we want to set it to) the thumbnail service checks its storage directory for files that can be expired. It does this by sending a notification called "browser:cleanup-thumbnails". The subject is an array that expects to be filled with URLs we want to keep. This way the newtab page can specify the URLs it is currently showing. All Panorama does (if activated/used) is provide the URLs of all tabs that are currently opened.

The thumbnail service then iterates through all the files in the directory and compares their names and deletes them if they're not in the list and older than one day.

Other notes:

1) The storage version is bumped to "2" and wipes the whole thumbnails storage. This is because I removed the file-cache-like sub-directories. The files are just stored under $profile/thumbnails now. Not sure why I thought this was necessary but we clearly don't want to have lots of thumbnails in here.

2) Links.getLinks() in NewTabUtils provides a synchronous fallback to retrieve links. It is highly theoretical that a user hasn't opened a single blank tab (with about:newtab enabled) so the link cache *should* be populated. If it is not, then this is a fallback to get all the links currently shown in the newtab page so that their thumbnails won't get lost.

3) In a second iteration we can and should update this code to use the new asynchronous file API that does all this cleanup and removal in a worker. Yoric said he's willing to help us with that.
Comment 10 Dão Gottwald [:dao] 2012-06-07 06:53:13 PDT
(In reply to Tim Taubert [:ttaubert] from comment #9)
> All Panorama does (if activated/used) is
> provide the URLs of all tabs that are currently opened.

Can we do this regardless of panorama? There's a good chance that add-ons will utilize the service for open tabs. Requiring them all to observe browser:cleanup-thumbnails seems suboptimal.
Comment 11 Girish Sharma [:Optimizer] 2012-06-07 06:55:44 PDT
10 minutes seems to small of a time gap, and as thumbnails are deleted synchronously, deleting them every 10 minutes (or not) might be too heavy.
Every hour seems a good option.
Comment 12 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-06-07 07:12:33 PDT
(In reply to Dão Gottwald [:dao] from comment #10)
> (In reply to Tim Taubert [:ttaubert] from comment #9)
> > All Panorama does (if activated/used) is
> > provide the URLs of all tabs that are currently opened.
> 
> Can we do this regardless of panorama? There's a good chance that add-ons
> will utilize the service for open tabs. Requiring them all to observe
> browser:cleanup-thumbnails seems suboptimal.

Sure, I'll just move that to browser-thumbnails.js.

(In reply to Girish Sharma [:Optimizer] from comment #11)
> 10 minutes seems to small of a time gap, and as thumbnails are deleted
> synchronously, deleting them every 10 minutes (or not) might be too heavy.
> Every hour seems a good option.

I agree but we'd need some smaller initial delay, because not everyone might keep their browser open for an hour. Also deleting every 10 mins should take less time than deleting bigger chunks every hour. We can always change that afterwards if it proves to be annoying.
Comment 13 Dão Gottwald [:dao] 2012-06-07 07:19:01 PDT
(In reply to Tim Taubert [:ttaubert] from comment #12)
> we'd need some smaller initial delay, because not everyone might
> keep their browser open for an hour.

There are special timers that take care of this, like the one used for the periodic software update check.
Comment 14 David Rajchenbach-Teller [:Yoric] (please use "needinfo") 2012-06-07 07:20:22 PDT
(In reply to Girish Sharma [:Optimizer] from comment #11)
> 10 minutes seems to small of a time gap, and as thumbnails are deleted
> synchronously, deleting them every 10 minutes (or not) might be too heavy.
> Every hour seems a good option.

Note that removal should be asynchronous pretty soon, see bug 753768.
Comment 15 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-06-07 07:43:57 PDT
(In reply to Dão Gottwald [:dao] from comment #13)
> (In reply to Tim Taubert [:ttaubert] from comment #12)
> > we'd need some smaller initial delay, because not everyone might
> > keep their browser open for an hour.
> 
> There are special timers that take care of this, like the one used for the
> periodic software update check.

There is nsIUpdateTimerManager which does exactly this (implemented with plain nsITimers) but seems very specific to mozapps/update. Should we use that?

http://dxr.lanedo.com/mozilla-central/toolkit/mozapps/update/nsUpdateTimerManager.js.html
Comment 16 Virtual_ManPL [:Virtual] - (ni? me) 2012-06-07 08:17:24 PDT
ok, ok guys...

but why store more thumbnails than what's in use (9 or more according to future setting)?
and next this scheduler will be in some way heavy on hdds, because of making a new thread etc.

Shouldn't we store only used and visible thumbnails like Speed/Fast Dial add-ons?
This will be the lightest way on memory usage.

What do you think about it?
Comment 17 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-06-07 08:19:27 PDT
Created attachment 630988 [details] [diff] [review]
patch v2

We now have an interval of one hour. The initial delay is 10 minutes. I didn't use nsIUpdateTimerManager because that seems too tied to the update service and re-initializing the timer is no big deal.
Comment 18 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-06-07 08:21:44 PDT
(In reply to Virtual_ManPL from comment #16)
> but why store more thumbnails than what's in use (9 or more according to
> future setting)?
> and next this scheduler will be in some way heavy on hdds, because of making
> a new thread etc.

The thumbnail service is used by Panorama as well and we'd like to have more features use it instead of every component implementing its own thumbnailing mechanisms. So we can't just throw away every thumbnail that's not "visible" anymore.
Comment 19 :Gavin Sharp [email: gavin@gavinsharp.com] 2012-06-07 11:22:31 PDT
AFAIK the only thing update-specific about nsIUpdateTimerManager is where it lives in the source tree (and its name, I guess :/). Assuming its functionality is suitable for this, I don't see any reasons not to use it.
Comment 20 Dão Gottwald [:dao] 2012-06-07 13:34:03 PDT
What mechanism do we use for add-on or blocklist update checks? Maybe some places maintenance jobs use such a timer as well...
Comment 21 Justin Dolske [:Dolske] 2012-06-07 14:11:52 PDT
How many files are likely to stay in this directory with expiration involved?

I'm a bit concerned about having to hit the filesystem and walk through everything. That's not exactly cheap, especially with lots of files.
Comment 22 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-06-07 14:31:00 PDT
(In reply to Justin Dolske [:Dolske] from comment #21)
> How many files are likely to stay in this directory with expiration involved?

I have no idea, tbh. At least 34 + the number of open tabs and then all the pages we took screenshots for over the day.

> I'm a bit concerned about having to hit the filesystem and walk through
> everything. That's not exactly cheap, especially with lots of files.

Would it help to split this work into chunks, using setTimeout()? Until we have a worker thread that's able to help us with this?
Comment 23 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-06-07 15:15:51 PDT
(In reply to Dão Gottwald [:dao] from comment #20)
> What mechanism do we use for add-on or blocklist update checks? Maybe some
> places maintenance jobs use such a timer as well...

We do use nsIUpdateTimerManager, finding those uses was really hard as they are registered with the category in a manifest:

toolkit/components/search/toolkitsearch.manifest
toolkit/mozapps/extensions/extensions.manifest

I'll update the patch to use it as well.
Comment 24 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-06-07 15:54:41 PDT
Created attachment 631191 [details] [diff] [review]
patch v3

Using nsIUpdateTimerManager.
Comment 25 Dão Gottwald [:dao] 2012-06-12 01:41:41 PDT
Comment on attachment 631191 [details] [diff] [review]
patch v3

>+  observe: function Thumbnails_observe(aSubject, aTopic, aData) {
>+    if (aTopic == "browser:cleanup-thumbnails") {
>+      aSubject.QueryInterface(Ci.nsIMutableArray);
>+
>+      for (let tab of gBrowser.tabs) {

for (let browser of gBrowser.browsers) {

>+        let url = Cc["@mozilla.org/supports-string;1"]
>+                    .createInstance(Ci.nsISupportsString);
>+        url.data = tab.linkedBrowser.currentURI.spec;

url.data = browser.currentURI.spec;

>   migrateToNextVersion: function Migrator_migrateToNextVersion() {
>     let version = this.currentVersion;
> 
>     if (version == 0)
>       this.removeThumbnailsFromRoamingProfile();
>+    else if (version == 1)
>+      this.clearThumbnailsFolder();
>   },

I'd prefer if 'migrateToNextVersion' bumped the version or if 'migrate' did it in the loop.

>+  getLinksSynchronously: function PlacesProvider_getLinksSynchronously() {

This wouldn't be needed if PageThumbs.jsm wasn't using the observer service but had a list of functions that would call back asynchronously, would it?
Comment 26 :Gavin Sharp [email: gavin@gavinsharp.com] 2012-06-12 09:27:23 PDT
Ah, I didn't realize that this was using nsIFile APIs on the main thread for the iteration/clearing. I don't think that's going to be suitable for landing, even as a stop gap solution. Unfortunately I'm not sure what better options we have now that this is on Aurora...
Comment 27 Gian-Carlo Pascutto [:gcp] 2012-06-12 09:34:37 PDT
That's an easy one, back out bug 744388 ASAP as was asked (and promised...) in that bug several times already?  :-P
Comment 28 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-06-18 07:34:15 PDT
Created attachment 634026 [details] [diff] [review]
patch v4

This patch is now using the async file API that Yoric just landed (bug 747876). It lazily spawns its own worker (for now as we'd be the first client, later there should be a global worker) and removes files asynchronously.

Also, this patch uses the async directory traversal API from bug 764436. The only thing we don't have yet are async stat calls which means that this patch does not yet always keep files younger than one day.

The list of URLs to keep thumbnails for is now collected by calling filters that will call back asynchronously which removes the need for a 'getLinksSynchronously' method and we can omit some nsI* casting and XPCOM calls.

Stills needs a test.
Comment 29 Dão Gottwald [:dao] 2012-06-18 07:50:48 PDT
Comment on attachment 634026 [details] [diff] [review]
patch v4

>+  filter: function Thumbnails_filter(aCallback) {
>+    aCallback(Array.map(gBrowser.browsers, function (aBrowser) {
>+      return aBrowser.currentURI.spec;
>+    }));
>+  },

[browser.currentURI.spec for (browser of gBrowser.browsers)]

>+  filter: function ExpirationFilter_filter(aCallback) {
>+    if (!AllPages.enabled) {
>+      aCallback([]);
>+      return;
>+    }

You could probably return false here and let PageThumbs handle this instead of calling back with an empty array.
Comment 30 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-06-19 14:55:47 PDT
Marking Fx 15 as unaffected because bug 744388 has been backed out on Aurora.
Comment 31 Justin Dolske [:Dolske] 2012-06-22 17:01:03 PDT
(In reply to Tim Taubert [:ttaubert] from comment #22)

> > I'm a bit concerned about having to hit the filesystem and walk through
> > everything. That's not exactly cheap, especially with lots of files.
> 
> Would it help to split this work into chunks, using setTimeout()? Until we
> have a worker thread that's able to help us with this?

Even with a worker thread you may still want to process it in chunks, so that there's less impact on other disk IO that might want to happen during that time.
Comment 32 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-07-14 00:29:17 PDT
*** Bug 773893 has been marked as a duplicate of this bug. ***
Comment 33 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-07-19 07:31:45 PDT
Created attachment 643842 [details] [diff] [review]
patch v5

Addressed the feedback from comment #29 and added chunked processing.

I talked to Gavin and we're planning to merge this back to Aurora. We'll not wait for bug 766194 any further and will implement this part as a follow-up.

While with this patch we might possibly expire thumbnails that we *could* need (that are younger than a day) we will have a solid storage for all other thumbnails that we're sure we need.
Comment 35 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-07-20 06:41:02 PDT
(In reply to Ben Bucksch (:BenB) from comment #34)
> https://wiki.mozilla.org/Privacy/Reviews/New_Tab#Principle:_Limited_Data

Right, it's also a privacy goal. Thanks for adding that.

(In reply to Tim Taubert [:ttaubert] from comment #33)
> I talked to Gavin and we're planning to merge this back to Aurora. We'll not
> wait for bug 766194 any further and will implement this part as a follow-up.

It's not only that we'll not wait for bug 766194, it will most probably not be backported so it's not available in Aurora.
Comment 36 Dão Gottwald [:dao] 2012-07-23 03:02:59 PDT
Comment on attachment 643842 [details] [diff] [review]
patch v5

>+  filter: function Thumbnails_filter(aCallback) {

rename this to filterForThumbnailExpiration or something similarly self-explanatory?

>+let PageThumbsWorker = {

>+  postMessage: function Worker_postMessage(message, callback) {
>+    this._callbacks.push(callback);
>+    this._worker.postMessage(message);
>+  },

>+  handleEvent: function Worker_handleEvent(aEvent) {
>+    let callback = this._callbacks.shift();
>+    if (callback)
>+      callback(aEvent.data);
>   }

I don't see how this ensures that the correct callback is picked.

>+  filter: function ExpirationFilter_filter(aCallback) {
>+    if (!AllPages.enabled)
>+      return false;
>+
>+    Links.populateCache(function () {
>+      let urls = [];
>+
>+      // Add all URLs to the list that we want to keep thumbnails for.
>+      for (let link of Links.getLinks().slice(0, 25)) {
>+        if (link && link.url)
>+          urls.push(link.url);
>+      }
>+
>+      aCallback(urls);
>+    });
>+  }

"return true;" at the end of the function
Comment 37 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-07-23 03:37:08 PDT
Created attachment 644886 [details] [diff] [review]
patch v6

(In reply to Dão Gottwald [:dao] from comment #36)
> >+  filter: function Thumbnails_filter(aCallback) {
> 
> rename this to filterForThumbnailExpiration or something similarly
> self-explanatory?

Done.

> I don't see how this ensures that the correct callback is picked.

The OS.File API itself works synchronously because we have no cross-platform support for async file IO. The worker thread receives a message and blocks until the operation has finished. That's why we can use a FIFO pipe to store the callbacks.

> "return true;" at the end of the function

Not sure why I made it return a boolean value to indicate a filter is disabled. The code's easier if this just calls the callback with an empty array - so that's what the patch does now.
Comment 38 Dão Gottwald [:dao] 2012-07-24 15:26:04 PDT
Comment on attachment 644886 [details] [diff] [review]
patch v6

>   wipe: function Storage_wipe() {
>     try {
>-      FileUtils.getDir("ProfLD", [THUMBNAIL_DIRECTORY]).remove(true);
>+      this.getDirectory().remove(true);
>     } catch (e) {
>       /* The file might not exist or we're not permitted to remove it. */
>     }
>   },

This looks like it will now create the directory before removing it.

>+  clearThumbnailsFolder: function Migrator_clearThumbnailsFolder() {
>+    try {
>+      FileUtils.getDir("ProfLD", [THUMBNAIL_DIRECTORY]).remove(true);
>+    } catch (e) {
>+      // The directory might not exist or we're not permitted to remove it.
>+    }
>+  }

Does getDir itself throw when the directory doesn't exist?
Comment 39 Dão Gottwald [:dao] 2012-07-24 15:32:18 PDT
Comment on attachment 644886 [details] [diff] [review]
patch v6

>+    function filterCallback(aURLs) {
>+      urls = urls.concat(aURLs);
>+      if (--filtersToWaitFor == 0)
>+        expire();
>+    }
>+
>+    for (let filter of this._filters) {
>+      if (typeof filter == "function")
>+        filter(filterCallback)
>+      else if ("filterForThumbnailExpiration" in filter)
>+        filter.filterForThumbnailExpiration(filterCallback);
>+    }

You should either a) just not test for filterForThumbnailExpiration's existence or b) throw an exception if it doesn't exist or c) ensure that filtersToWaitFor is decremented and expire is called.
Comment 40 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-07-25 05:40:39 PDT
(In reply to Dão Gottwald [:dao] from comment #38)
> >   wipe: function Storage_wipe() {
> >     try {
> >-      FileUtils.getDir("ProfLD", [THUMBNAIL_DIRECTORY]).remove(true);
> >+      this.getDirectory().remove(true);
> >     } catch (e) {
> >       /* The file might not exist or we're not permitted to remove it. */
> >     }
> >   },
> 
> This looks like it will now create the directory before removing it.

Right, that's not what we want if we're about to wipe the directory. I'll add an argument to getDirectory() to disable directory creation.

> Does getDir itself throw when the directory doesn't exist?

No, it returns an nsIFile instance where .exists() returns false. remove() throws in case of permission errors or if the directory doesn't exist.

(In reply to Dão Gottwald [:dao] from comment #39)
> You should either a) just not test for filterForThumbnailExpiration's
> existence or b) throw an exception if it doesn't exist or c) ensure that
> filtersToWaitFor is decremented and expire is called.

Good catch, thanks. I think we should just go with (a) and this will report an error if the object does not have the expected method.
Comment 41 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-07-25 06:14:34 PDT
Created attachment 645740 [details] [diff] [review]
patch v7

All issues addressed.
Comment 42 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-07-25 07:47:21 PDT
I'm permanently hitting bug 737244 with debug builds on try :(

https://tbpl.mozilla.org/?tree=Try&rev=a366582689c4

Seems to be a crash/bug in the memory reporter for workers.
Comment 43 Dão Gottwald [:dao] 2012-07-25 09:48:56 PDT
> > Does getDir itself throw when the directory doesn't exist?
> 
> No, it returns an nsIFile instance where .exists() returns false. remove()
> throws in case of permission errors or if the directory doesn't exist.

Then I'd prefer if you moved the this.getDirectory calls out of the try/catch blocks, as we don't expect exceptions from them.
Comment 44 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-08-01 03:09:17 PDT
Created attachment 647898 [details] [diff] [review]
patch v8

All feedback addressed. Found the reason for crashing debug builds - will be fixed by bug 779445.

I *think* Dão is on vacation. Gavin, could you please take a look at this? We can of course ask someone else as well.
Comment 45 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-08-01 11:17:14 PDT
Try looks good:

https://tbpl.mozilla.org/?tree=Try&rev=d0ab054846d1
Comment 46 :Gavin Sharp [email: gavin@gavinsharp.com] 2012-08-08 17:24:39 PDT
Comment on attachment 647898 [details] [diff] [review]
patch v8

>+  clearThumbnailsFolder: function Migrator_clearThumbnailsFolder() {
>+    let dir = FileUtils.getDir("ProfLD", [THUMBNAIL_DIRECTORY]);
>+    try {
>+      dir.remove(true);

This seems like it will cause the problem as bug 767411. I guess if we ship it in 16, it won't actually affect anyone, for the same reasons I listed in bug 767411 comment 1? Even still, it would be nice to avoid this huge hit for Nightly users. Can we easily switch this to an async task using OS.File?

PageThumbsStorage.wipe has the same problem, and looks like it's called when history is cleared - I think we need to fix that too, in a tracking16+ bug.

bumping the review request to felipe for the rest of the review
Comment 47 :Felipe Gomes (needinfo me!) 2012-08-09 18:44:10 PDT
Comment on attachment 647898 [details] [diff] [review]
patch v8

Review of attachment 647898 [details] [diff] [review]:
-----------------------------------------------------------------

My understanding is with the chunked removal, this will only remove 50 thumbnails per hour. Is this the intended behavior? I don't know what's the expected number of thumbnails being saved to know if the removal can catch up with the thumbnail growth in this case (as this opens the chance to grow infinitely).

::: browser/components/thumbnails/PageThumbs.jsm
@@ +316,2 @@
>      try {
> +      dir.remove(true);

every nsIFile.remove() needs to have .followLinks = false set, otherwise it will follow symlinks/.lnk files that could have been dropped in that folder and remove their contents (instead of just removing the symlinks)

@@ +431,5 @@
> +        filter.filterForThumbnailExpiration(filterCallback);
> +    }
> +  },
> +
> +  expireThumbnails: function Expiration_expireThumbnails(aURLsToKeep) {

this function could live in the worker to avoid the roundtrip of getFilesInDirectory + 1 postMessage for each removeFile. But I believe you put it in this file to keep the expiration logic all here? If the postMessages are not expensive then that's fine

::: browser/components/thumbnails/PageThumbsWorker.js
@@ +26,5 @@
> +      self.postMessage({result: false, detail: "message not understood"});
> +      return;
> +    }
> +
> +    self.postMessage({result: result});

I think it would be clearer to write this function as a switch statement where the default case is the "message not understood" error, so that you only use 1 postMessage (although the result obj will be a bit more cumbersome)

@@ +34,5 @@
> +    let iter = new OS.File.DirectoryIterator(msg.path);
> +    let entries = [];
> +
> +    for (let entry in iter) {
> +      if (!entry.isDir && !entry.isSpecialDir && !entry.isLink)

where are these getters defined? I couldn't find them.
http://mxr.mozilla.org/mozilla-central/source/toolkit/components/osfile/osfile_win_front.jsm#563 defines isSymLink for example, and no isSpecialDir that I can see

::: browser/modules/NewTabUtils.jsm
@@ +621,5 @@
> +    if (!this._initialized) {
> +      this._initialized = true;
> +      ExpirationFilter.init();
> +      Telemetry.init();
> +    }

is it not necessary to uninit the expiration filter in this file? I'm guessing no because they have the same lifetime (i.e. until shutdown)
Comment 48 David Rajchenbach-Teller [:Yoric] (please use "needinfo") 2012-08-10 14:06:26 PDT
(In reply to :Felipe Gomes from comment #47)
> Comment on attachment 647898 [details] [diff] [review]
> patch v8
> 
> Review of attachment 647898 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> My understanding is with the chunked removal, this will only remove 50
> thumbnails per hour. Is this the intended behavior? I don't know what's the
> expected number of thumbnails being saved to know if the removal can catch
> up with the thumbnail growth in this case (as this opens the chance to grow
> infinitely).

I have not checked the code, but I have the intuition that we should rather maintain some data structure of least-recently modified thumbnails (map + circular linked list?) and cut it (by half?) whenever it passes some threshold.
Comment 49 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-08-17 17:06:04 PDT
Created attachment 652986 [details] [diff] [review]
patch v9

(In reply to :Gavin Sharp (use gavin@gavinsharp.com for email) from comment #46)
> >+  clearThumbnailsFolder: function Migrator_clearThumbnailsFolder() {
> >+    let dir = FileUtils.getDir("ProfLD", [THUMBNAIL_DIRECTORY]);
> >+    try {
> >+      dir.remove(true);
> 
> This seems like it will cause the problem as bug 767411. I guess if we ship
> it in 16, it won't actually affect anyone, for the same reasons I listed in
> bug 767411 comment 1? Even still, it would be nice to avoid this huge hit
> for Nightly users. Can we easily switch this to an async task using OS.File?

Right, that will only affect Nightly/Aurora users. The problem is that there's no way to remove a directory asynchronously, yet - bug 761138. OTOH migrating the storage directory might cause interesting effects if we're still in the transition and the thumbnail service captures thumbnails.

> PageThumbsStorage.wipe has the same problem, and looks like it's called when
> history is cleared - I think we need to fix that too, in a tracking16+ bug.

I also thought about doing this but until now, the process of clearing your history is synchronous. You click the button and after that the history got cleared. We could fix that as a follow-up if we want to change that.

(In reply to :Felipe Gomes from comment #47)
> My understanding is with the chunked removal, this will only remove 50
> thumbnails per hour. Is this the intended behavior? I don't know what's the
> expected number of thumbnails being saved to know if the removal can catch
> up with the thumbnail growth in this case (as this opens the chance to grow
> infinitely).

Yes, good point. I took Yoric's proposal and now cut the list of files to remove in half if that half is bigger than EXPIRATION_MIN_CHUNK_SIZE (=50 by default).

> ::: browser/components/thumbnails/PageThumbs.jsm
> @@ +316,2 @@
> >      try {
> > +      dir.remove(true);
> 
> every nsIFile.remove() needs to have .followLinks = false set, otherwise it
> will follow symlinks/.lnk files that could have been dropped in that folder
> and remove their contents (instead of just removing the symlinks)

Fixed.

> @@ +431,5 @@
> > +        filter.filterForThumbnailExpiration(filterCallback);
> > +    }
> > +  },
> > +
> > +  expireThumbnails: function Expiration_expireThumbnails(aURLsToKeep) {
> 
> this function could live in the worker to avoid the roundtrip of
> getFilesInDirectory + 1 postMessage for each removeFile. But I believe you
> put it in this file to keep the expiration logic all here? If the
> postMessages are not expensive then that's fine

That's right it could live in the worker but I kept it here to keep the logic together. I modified the code anyway to take a removeFiles() method. I don't know the overhead of postMessage() but we call it now only twice for every expiration round.

> ::: browser/components/thumbnails/PageThumbsWorker.js
> @@ +26,5 @@
> > +      self.postMessage({result: false, detail: "message not understood"});
> > +      return;
> > +    }
> > +
> > +    self.postMessage({result: result});
> 
> I think it would be clearer to write this function as a switch statement
> where the default case is the "message not understood" error, so that you
> only use 1 postMessage (although the result obj will be a bit more
> cumbersome)

Done.

> @@ +34,5 @@
> > +    let iter = new OS.File.DirectoryIterator(msg.path);
> > +    let entries = [];
> > +
> > +    for (let entry in iter) {
> > +      if (!entry.isDir && !entry.isSpecialDir && !entry.isLink)
> 
> where are these getters defined? I couldn't find them.
> http://mxr.mozilla.org/mozilla-central/source/toolkit/components/osfile/
> osfile_win_front.jsm#563 defines isSymLink for example, and no isSpecialDir
> that I can see

Yeah... The API changed in the meantime. Fixed.

> ::: browser/modules/NewTabUtils.jsm
> @@ +621,5 @@
> > +    if (!this._initialized) {
> > +      this._initialized = true;
> > +      ExpirationFilter.init();
> > +      Telemetry.init();
> > +    }
> 
> is it not necessary to uninit the expiration filter in this file? I'm
> guessing no because they have the same lifetime (i.e. until shutdown)

It's not necessary because both things are live until the browsers shuts down.
Comment 50 :Felipe Gomes (needinfo me!) 2012-08-20 17:07:09 PDT
Comment on attachment 652986 [details] [diff] [review]
patch v9

Review of attachment 652986 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/components/thumbnails/PageThumbsWorker.js
@@ +48,5 @@
> +  },
> +
> +  removeFiles: function Worker_removeFiles(msg) {
> +    for (let file of msg.paths) {
> +      OS.File.remove(file);

Is there any chance of this throwing? if so we should wrap a try catch to make sure we don't end up skipping postMessage with the return value, which would make the callback queue on the other thread to get out of sync
Comment 51 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-08-20 17:39:53 PDT
(In reply to :Felipe Gomes from comment #50)
> > +  removeFiles: function Worker_removeFiles(msg) {
> > +    for (let file of msg.paths) {
> > +      OS.File.remove(file);
> 
> Is there any chance of this throwing? if so we should wrap a try catch to
> make sure we don't end up skipping postMessage with the return value, which
> would make the callback queue on the other thread to get out of sync

Good point, this throws indeed. Will wrap it in a try/catch and continue with the next file on error.
Comment 52 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-08-21 11:19:11 PDT
https://hg.mozilla.org/integration/fx-team/rev/327883b4f2fe
Comment 53 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-08-21 13:03:58 PDT
Created attachment 653906 [details] [diff] [review]
backport for Aurora

Same as v9 with some slight modifications as the Aurora codebase is a little different.
Comment 54 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-08-21 16:37:38 PDT
Comment on attachment 653906 [details] [diff] [review]
backport for Aurora

Will update the patch.
Comment 55 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-08-21 16:46:24 PDT
Created attachment 654012 [details] [diff] [review]
follow-up to rename old thumbnail folder instead of removing it

I talked to felipe and before this patch gets into the next Nightly we decided to rename the old thumbnails directory instead of removing it synchronously (that would cause bug 767411).

We'll handle (or maybe not) removing the directory asynchronously in a follow-up bug. This will leave Aurora/Nightly users with the big old thumbnail directory for now but that's better than Firefox freezing at startup while removing gigs of images.
Comment 56 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-08-21 16:49:47 PDT
Created attachment 654016 [details] [diff] [review]
backport for Aurora

Updated the backport to rename the thumbnail folder as well.
Comment 57 :Felipe Gomes (needinfo me!) 2012-08-21 16:57:02 PDT
Comment on attachment 654012 [details] [diff] [review]
follow-up to rename old thumbnail folder instead of removing it

Review of attachment 654012 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/components/thumbnails/PageThumbs.jsm
@@ +394,2 @@
>      try {
> +      dir.moveTo(null, dir.leafName + "-1");

"-old" maybe? so it's easier for people to confidently manually remove the old folder if they want
Comment 58 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-08-21 17:03:19 PDT
(In reply to :Felipe Gomes from comment #57)
> >      try {
> > +      dir.moveTo(null, dir.leafName + "-1");
> 
> "-old" maybe? so it's easier for people to confidently manually remove the
> old folder if they want

Yeah, that's better, will do.
Comment 59 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-08-21 17:04:25 PDT
https://hg.mozilla.org/integration/fx-team/rev/8e30c456cb6e
Comment 60 :Felipe Gomes (needinfo me!) 2012-08-21 17:05:23 PDT
Comment on attachment 654016 [details] [diff] [review]
backport for Aurora

Review of attachment 654016 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/components/thumbnails/PageThumbs.jsm
@@ +14,5 @@
>  const PREF_STORAGE_VERSION = "browser.pagethumbnails.storage_version";
> +const LATEST_STORAGE_VERSION = 2;
> +
> +const EXPIRATION_MIN_CHUNK_SIZE = 50;
> +const EXPIRATION_INTERVAL_SECS = 10;//3600;

leftover change?
Comment 61 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-08-21 17:06:08 PDT
(In reply to :Felipe Gomes from comment #60)
> > +const EXPIRATION_MIN_CHUNK_SIZE = 50;
> > +const EXPIRATION_INTERVAL_SECS = 10;//3600;
> 
> leftover change?

Oops :/ Will fix and change to "-old" as well.
Comment 62 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-08-21 17:09:13 PDT
Comment on attachment 654016 [details] [diff] [review]
backport for Aurora

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 744388
User impact if declined: thumbnail storage keeps growing
Testing completed (on m-c, etc.): included in tomorrow's Nightly (probably)
Risk to taking this patch (and alternatives if risky): risk is middle I'd say but it would fix a long-standing issue with thumbnail disappearing from the new tab page because the alternative would be to back out bug 744388 from Aurora again.
String or UUID changes made by this patch: none
Comment 63 Dão Gottwald [:dao] 2012-08-22 00:31:45 PDT
(In reply to Tim Taubert [:ttaubert] from comment #55)
> We'll handle (or maybe not) removing the directory asynchronously in a
> follow-up bug. This will leave Aurora/Nightly users with the big old
> thumbnail directory for now but that's better than Firefox freezing at
> startup while removing gigs of images.

Please do handle it. Leaving gigabytes of garbage behind doesn't seem reasonable at all, as 1) that space can be very valuable depending on how much free space is left, 2) it makes backing up profiles more cumbersome (which nightly users will do more likely than others) and 3) it's a privacy leak.
Comment 64 David Rajchenbach-Teller [:Yoric] (please use "needinfo") 2012-08-22 05:45:57 PDT
As far as I understand, the issue of async removal is clearly not specific to this bug or even to thumbnails (e.g. cache, cookies).

Perhaps we should have a general-purpose API to handle "safe" asynchronous directory removal. Also, this could be useful on Android, as the temporary directory is something of a hack.
Comment 65 Alex Keybl [:akeybl] 2012-08-22 15:24:13 PDT
(In reply to Tim Taubert [:ttaubert] from comment #62)
> Risk to taking this patch (and alternatives if risky): risk is middle I'd
> say but it would fix a long-standing issue with thumbnail disappearing from
> the new tab page because the alternative would be to back out bug 744388
> from Aurora again.

I don't think the feedback about missing thumbnails has been negative enough to warrant taking a medium-risk forward fix. Let's back out bug 744388 instead.
Comment 66 Alex Keybl [:akeybl] 2012-08-22 15:26:00 PDT
Sorry, the feedback about large thumbnail directories*

Let's let things stand as they were in 15 for one more cycle.
Comment 67 Alex Keybl [:akeybl] 2012-08-22 15:34:14 PDT
Comment on attachment 654016 [details] [diff] [review]
backport for Aurora

I'm tripping over myself and need to review further. I'm going to discuss with ttaubert and leave this nominated.
Comment 68 Alex Keybl [:akeybl] 2012-08-22 15:45:59 PDT
Comment on attachment 654016 [details] [diff] [review]
backport for Aurora

Spoke with Tim, my comment 65 contains the correct justification. We'll back out instead given the medium risk of the new async IO lib and our proximity to release.
Comment 69 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-08-22 16:58:50 PDT
(In reply to Dão Gottwald [:dao] from comment #63)
> (In reply to Tim Taubert [:ttaubert] from comment #55)
> > We'll handle (or maybe not) removing the directory asynchronously in a
> > follow-up bug. This will leave Aurora/Nightly users with the big old
> > thumbnail directory for now but that's better than Firefox freezing at
> > startup while removing gigs of images.
> 
> Please do handle it. Leaving gigabytes of garbage behind doesn't seem
> reasonable at all, as 1) that space can be very valuable depending on how
> much free space is left, 2) it makes backing up profiles more cumbersome
> (which nightly users will do more likely than others) and 3) it's a privacy
> leak.

Yes, those are all very valid points. I'll handle removing the leftover thumbnails in bug 784868.
Comment 71 :Gavin Sharp [email: gavin@gavinsharp.com] 2012-08-23 10:08:12 PDT
Comment on attachment 652986 [details] [diff] [review]
patch v9

(In reply to Alex Keybl [:akeybl] from comment #65)
> I don't think the feedback about missing thumbnails has been negative enough
> to warrant taking a medium-risk forward fix. Let's back out bug 744388
> instead.

I don't think we can delay bug 744388 any further, the "thumbnails disappearing" issue has been a major problem for New Tab users (see e.g. bug 756881 comment 22, numerous posts on SUMO, etc.).

I don't think the risk assessment here is accurate - it's true that we're using a new and "unproven" API, but there's no reason to expect it will cause problems, and we can always re-evaluate this decision later on in the beta cycle if we discover serious issues (doing a backout then is straighforward).
Comment 72 Jo Hermans 2012-08-23 11:52:47 PDT
Funny. It never stopped Mozilla before from checking in bug 744388, despite that there was no expiration method, no size-limit, obvious security issues and it was using a roaming profile. And now when it's possible to fix this, there's a risk with an unproven API ?
Comment 73 Alex Keybl [:akeybl] 2012-08-23 14:49:51 PDT
(In reply to :Gavin Sharp (use gavin@gavinsharp.com for email) from comment #71)
> I don't think the risk assessment here is accurate - it's true that we're
> using a new and "unproven" API, but there's no reason to expect it will
> cause problems, and we can always re-evaluate this decision later on in the
> beta cycle if we discover serious issues (doing a backout then is
> straighforward).

Given Gavin's concerns, we discussed this as part of a larger group at today's Channel Meeting. The medium risk evaluation was originally made with an expectation that we would not necessarily find new regressions and the prevalence of missing thumbnails was not great. Gavin believes that the medium risk evaluation is correct initially, but that we'll be able to find regressions over the 6 week cycle and backout if necessary.

The fact that it's taken us a couple of cycles is not a matter of severity, but rather a fact of the difficulty in technical implementation. We checked with Support and they have heard enough data to warrant taking the forward fix. Given that, let's land this and re-land bug 744388 on Aurora 16.
Comment 74 Tim Taubert [:ttaubert] (on PTO, back Aug 29th) 2012-08-23 15:04:21 PDT
https://hg.mozilla.org/releases/mozilla-aurora/rev/4762f9ba5727

Note You need to log in before you can comment on or make changes to this bug.