1628011 - Weave directory is >30 GB in size

Status: RESOLVED DUPLICATE of bug 1627638

(Firefox :: Sync, defect)

Description

[:Vincent]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0

Steps to reproduce:

Just used Firefox normally. I've deleted the weave directory in my profile folder a couple of times, but it grows back to >30GB every time. This is quite an old install I think, so my Firefox Sync account has existed for quite a while, I think - it's possible the same one I was using back when it was still called Weave.

Actual results:

The weave directory in my profile folder takes up over 30GB in size, almost entirely due to the contents of weave/logs. That, in turn, contains >4000 files called error-sync-<number>.txt, whose contents appears to be mostly lines containing "INFO BulkKeyBundle being created for".

Expected results:

Most of those logs should probably not be created.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 2

[Lina Butler (ex-Mozilla)]

Hi Vincent, thank you for the report! That looks like the same bug as bug 1627638 (which is moco-confidential because it has a log attachment with some of the user's history), and likely bug 1624586, comment 9. I'll copy over some of the discussion here.

Me:

That logging comes from CollectionKeyManager#setContents, which is called by extension storage here, inside getKeyRing. That, in turn, is called by getKeys.

But this comment inside getKeys suggests we fetch the key ring record for every extension?

Mark:

Best I can tell there are 21 extensions. If we were only fetching the keyring etc per extension, then we'd be in a much better position - Setting collection keys contents appears 3915 times in that log file, which is very bad

Ethan:

• Don't provide the transformer when opening the collection outside of sync. We only use it during sync and we only call sync from within syncAll so dropping it in other places should be safe.
• Instead of passing the transformer a link to the crypto collection, get the keyring/salts or whatever from the crypto collection and pass those instead.

We drop the handle to the collection as soon as sync is over so we shouldn't have to invalidate this "cache" in any way.

Comment 3

[:Vincent]

Thanks Lina, that comment certainly looks the same. Is there anything I should do to e.g. +1 that confidential bug, or just close this one, or let it be? (/is there any additional data I could/should provide, e.g. such log attachments?)

Comment 4

[Lina Butler (ex-Mozilla)]

Let's leave this one be for now, since it's open—I think Ethan is going to attach a patch to the other bug, and, once we remove the log with the PII, we can open it up and mark yours as a dupe. Thanks!

Comment 5

[Lina Butler (ex-Mozilla)]

Aaaand...it's open! Let's dupe this out, and you can follow the other bug. Thanks Vincent!