1598371 - Auto generated random favicons are stored and never expired, flooding the favicons database

Status: VERIFIED FIXED

(Toolkit :: Places, defect, P1)

Description

[Vsevolod Velichko]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0

Steps to reproduce:

My mail provider uses favicon to display amount of messages unread.
To achieve it auto-generated urls are used, e.g.: https://mail.yandex.ru/u2709/favicon/png/main-favicon/16/2.png?nocacheIcon=9992105949301
These urls have no-cache header:
Cache-Control: private,no-cache,no-store,must-revalidate,max-age=0

I used Firefox for many months and got huge and slow favicons cache.

Actual results:

Each time after some months of Firefox usage I find that my favicons.sqlite storage had grown up to 1 GB+ of size and when I open new tab, Firefox freezes for some time. I found that during that time it performs some lookups in favicons.sqlite. I used the following sql to inspect it:

with domain(item) as (select replace(replace(icon_url, "https://" ,""), "http://", "") as item from moz_icons)
select count(*) as cnt, substr(item, 0, instr(item, '/')) as dom from domain group by dom order by cnt desc limit 5;

There I found the following distribution:
cnt dom

---

207516 mail.yandex.ru
120 yastatic.net
93 cdn.sstatic.net
86 img.imgsmail.ru
42 s.rbk.ru

The cache is flooded with the nocache-urls similar to the example above. It appears that Firefox doesn't respect no-cache header, nor does it prune old records after receiving new favicon.

Expected results:

The favicons db is clean and tidy, no non-relevant records remain after continuous usage.

Comment 1

[Vsevolod Velichko]

Bug reports that are may be relevant to this problem:

• Bug 1503809 — probably that's what causes freezes when cache is large
• Bug 699785 — unimplemented favicon cache expiration (for android, but may be desktop version shares the code, I do not know)

Comment 2

[Vsevolod Velichko]

Just for better problem understanding, this is the link to another favicon bug, because of which icons are not cached: https://bugzilla.mozilla.org/show_bug.cgi?id=1429766#c6

Comment 3

[Florencia Di Ciocco, NI to vbalducci]

Hi Vsevolod,

I wasn't able to reproduce the bug since I don't have an account with favicon but I've chosen a component for this bug in hope that someone with more expertise may look at it. We'll await their answer. If you consider that there's another component that's more proper for this case you may change it.

Regards, Flor.

Comment 4

[Kevin Brosnan [Ex-Mozilla]]

Favicons are not widgets (buttons, scrollbars, etc.). Guessing toolkit::places but that might not be the correct place.

Comment 5

[Marco Bonardo [:mak]]

Yes, Places is a good guess.
Favicons are expired after a week when a page is revisited, so this is a bit surprising, there's surely a bug.
Since you already looked into the database, can you tell me more about the associated data?
In particular, I'd like to get some of the expire_ms values for these icons, they are just timestamps in milliseconds so it should be easy to convert them and check if they are expired.
The other thing is which urls these are associated to, icons are associated through the moz_icons_to_pages table to a moz_pages_w_icons... so a join would do.
I can build a query for you, if you wish.

What I'd like to understand is whether these icons are all associated to the same url, or yandex is also generating random urls and icons get associated with them, so basically history gets flooded, not just icons.

Comment 6

[Marco Bonardo [:mak]]

And it may be a good idea to respect the "no-store" Cache-control, even if it could also cause some bookmarks to never have a favicon if it's misused...

Comment 7

[Marco Bonardo [:mak]]

Taking for investigation, it seems bad enough.

Comment 8

[Vsevolod Velichko]

Marco, while page urls are generated based on current open mail info, sometimes they match:

sqlite> select p.page_url, i.icon_url, datetime(i.expire_ms / 1000, 'unixepoch') as expire_ts from moz_icons i join moz_icons_to_pages ip on (i.id=ip.icon_id) join moz_pages_w_icons p on (ip.page_id=p.id) where icon_url like 'https://mail.yandex.ru/u2709/favicon/png/main-favicon/16/0.png?noCacheIcon=%' limit 5;
page_url                                                                           icon_url                                                                                  expire_ts          
---------------------------------------------------------------------------------  ----------------------------------------------------------------------------------------  -------------------
https://mail.yandex.ru/?uid=1120000000009144#folder/40/thread/t170573835886993666  https://mail.yandex.ru/u2709/favicon/png/main-favicon/16/0.png?nocacheIcon=8755854125501  2019-11-14 18:44:48
https://mail.yandex.ru/?uid=1120000000009144#folder/40/thread/t170573835886993666  https://mail.yandex.ru/u2709/favicon/png/main-favicon/16/0.png?nocacheIcon=3069186659077  2019-11-14 18:44:52
https://mail.yandex.ru/?uid=1120000000009144#folder/40/thread/t170573835886993666  https://mail.yandex.ru/u2709/favicon/png/main-favicon/16/0.png?nocacheIcon=2529999701121  2019-11-14 18:44:56
https://mail.yandex.ru/?uid=1120000000009144#folder/40/thread/t170573835886993666  https://mail.yandex.ru/u2709/favicon/png/main-favicon/16/0.png?nocacheIcon=4499835338761  2019-11-14 18:45:00
https://mail.yandex.ru/?uid=1120000000009144#folder/40/thread/t170573835886993666  https://mail.yandex.ru/u2709/favicon/png/main-favicon/16/0.png?nocacheIcon=1419540394202  2019-11-14 18:45:17

Comment 9

[Marco Bonardo [:mak]]

Could you count how many unique uris point to https://mail.yandex.ru icons?

Comment 10

[Vsevolod Velichko]

Sure.

sqlite> select count(distinct p.page_url) from moz_icons i join moz_icons_to_pages ip on (i.id=ip.icon_id) join moz_pages_w_icons p on (ip.page_id=p.id) where icon_url like 'https://mail.yandex.ru/%';
count(distinct p.page_url)
--------------------------
436     

Just to clarify: currently my favicons.sqlite has 34970 mail.yandex.ru icons, it was rotated again some days before my bugreport.

Comment 11

[Marco Bonardo [:mak]]

This should be fixed now in Firefox Nightly 73, I'll evaluate uplifting the fixes to one of the next Firefox Beta 72.
I filed bug 1601613 to evaluate expiring favicons more aggressively.

Please let me know if you still have problems after these fixes.

Comment 12

[Vsevolod Velichko]

That's great, looking forward to test it!

Comment 13

[Vsevolod Velichko]

I can finally confirm that while old cached favicons are not removed, the new ones are not being stored anymore, so my problem is generally solved in FF72.