643796 - about:support - indicate folders which may be near or exceed OS/Thunderbird limits. Or over N GB in size.

Status: NEW

(Thunderbird :: General, enhancement)

Description

[Wayne Mery (:wsmwk)]

about:support should indicate folders which may be near or exceeding OS/Thunderbird limits. Or over N GB in size (2GB?)

Such folders cause at least three problems:
1. large folders = large memory requirements (due to bug .msf summary files)
2. users having trouble deleting or moving messages
3. needless, excessive size of global search index

and identifying them may help expose these issues:
1. user not emptying trash or junk - or bugs which have the same result
2. user not doing compact of folder - or bugs which have the same result
3. user never deletes mail
4. corrupted folders


from https://github.com/sid0/tb-about-support/issues/9

Comment 1

[Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland]

:sid0 can you pull off any of this in time for 3.3 #hopingForYetAnotherSid0Miracle :-) ?

Comment 2

[Rain [:rain1] (inactive)]

Bryan: possible to get some UI feedback here? How should this be presented? Should we just have an entirely separate section listing large folders?

Roland, are there any privacy concerns with displaying folder names and paths?

Comment 3

[Bryan Clark (DevTools PM) [:clarkbw]]

(In reply to comment #2)
> Bryan: possible to get some UI feedback here? How should this be presented?
> Should we just have an entirely separate section listing large folders?

I would guess you'd have to show a separate list for "Possible Problem Folders".  This feels a little scary because unless we have the tools to help people fix these issues I wouldn't want to identify these folders as "problems"; unless that is we're certain they are problems.  But perhaps we should overlook user impression and possible support burden of labeling these folders this way so people can get their installations fixed.

Part of me also feels like fixing the folders issues might want to be a space or page.  Right now we have all sorts of "Troubleshooting Information" and no actions so it makes the page very simple to copy&paste.  I guess I'm wondering if we should list the possible problematic folders with a link that opens up the "Thunderbird Garage" for tuning things.  Maybe that's overboard.

Comment 4

[Rain [:rain1] (inactive)]

How about a more neutral term, e.g. "Large Folders"?

Comment 5

[Bryan Clark (DevTools PM) [:clarkbw]]

(In reply to comment #4)
> How about a more neutral term, e.g. "Large Folders"?

Yeah, that could work.

Comment 6

[Wayne Mery (:wsmwk)]

"Folders of Interest" ala Person of Interest?

just kidding. Large sounds good to me too.

(In reply to comment #3)
> Part of me also feels like fixing the folders issues might want to be a space
> or page.  Right now we have all sorts of "Troubleshooting Information" and no
> actions so it makes the page very simple to copy&paste.  I guess I'm wondering
> if we should list the possible problematic folders with a link that opens up
> the "Thunderbird Garage" for tuning things.  Maybe that's overboard.

I think we can be patient about pointing the user to help.

Comment 7

[Rain [:rain1] (inactive)]

Attachment: First cut

This needs feedback about:
- the threshold
- the fields displayed in both public and private mode
- the general UX.

I've started off try builds.

Comment 8

[Rain [:rain1] (inactive)]

Try builds: http://stage.mozilla.org/pub/mozilla.org/thunderbird/tryserver-builds/[e-mail]-944daa4c8553

Comment 9

[Wayne Mery (:wsmwk)]

Comment on attachment 524103 [details] [diff] [review]
First cut

on my fast i7 desktop, Tested local folders @ 1.8 & 2.1 GB, roughly 40k messages.

Some aspects of performance were not good. My feedback / thoughts on possible improvements, some of which go beyond scope of this bug:
1. Add msg count to the folder
2. Add msg count threshold as alternative size criteria
3. 2 gb seems reasonable. I think we don't want to go lower
4. Can we easily flag folders that are over the OS limit?
5. Is there a simple way to indicate whether it's a laptop environment? (which typically have slow drives)
6. add drive space and free sPace
7. To provide more context, Add to the folder name ... the full mail dir path, or name of parent folder?
8. List size(s) first, before acct and folder name?

6.

Comment 10

[Wayne Mery (:wsmwk)]

Comment on attachment 524103 [details] [diff] [review]
First cut

#1 only must have, assuming it is easy.
#8 would be nice for this pass
#2 likewise, but beyond scOpe?

Comment 11

[Wayne Mery (:wsmwk)]

p.s. kudos to sid0 for continuing to push forward with improvements for about:support.


> on my fast i7 desktop, Tested local folders @ 1.8 & 2.1 GB, roughly 40k
> messages.
> 
> Some aspects of performance were not good. My feedback ...

ick! That doesn't read well. My comment about bad performance of course relates to folders with 40k messages, not to about:support.


While I'm here, to elaborate on #6 (add drive space and free sPace) although it's out of scope for this bug ... for windows systems, defrag doesn't run great at reduce freespace (<20% iirc) and app performance can also tend to suffer, extremely, at some lower points.

Comment 12

[Bryan Clark (DevTools PM) [:clarkbw]]

Comment on attachment 524103 [details] [diff] [review]
First cut

I didn't have large enough folders to test this out with so my only concern was that it awkwardly says "None" when the list is empty.

My only suggestion is to take the largest 3 or so folders and list them as with their sizes and status as "good".  Though grabbing some random number of folders might need an explanation.  It would at least allow me to see what folders might become problematic and also know that I'm in the clear.

I'll switch the review over to Blake from now on.  Feel free to request feedback from me when there are try builds available.

Comment 13

[Blake Winton (:bwinton) (:☕️)]

Comment on attachment 524103 [details] [diff] [review]
First cut

Review of attachment 524103 [details] [diff] [review]:

wmswk said:
> 1. user not emptying trash or junk - or bugs which have the same result

I wonder if we should make the limit of trash or junk folders much smaller, like, 100k, or something?

When I changed the limit to 2MB (so that I could see my folders), I noticed that it showed a bunch of folders named after companies I consulted with, which I probably shouldn't be sending around, so the ability to either remove all the large folders, or selectively hide some of them, seems like it might be a good idea.

2 Gb sounds pretty big to me, but it might be the right number.  What's the OS limit that we're trying to avoid here?

And other than that, it seems to fit into the rest of the page pretty well, so ui-r=me.

Thanks,
Blake.

Comment 14

[Wayne Mery (:wsmwk)]

(In reply to Blake Winton (:bwinton - Thunderbird UX) from comment #13)
> Comment on attachment 524103 [details] [diff] [review]
> First cut
> 
> Review of attachment 524103 [details] [diff] [review]:
> 
> wmswk said:
> > 1. user not emptying trash or junk - or bugs which have the same result
> 
> I wonder if we should make the limit of trash or junk folders much smaller,
> like, 100k, or something?

not a bad idea. But indeed what value?  100k seems too low to me, given that someone can easily delete a single N mb message and thus exceed the threshold.  Is a (additional) metric of number of messages worth considering?  


> When I changed the limit to 2MB (so that I could see my folders), I noticed
> that it showed a bunch of folders named after companies I consulted with,
> which I probably shouldn't be sending around, so the ability to either
> remove all the large folders, or selectively hide some of them, seems like
> it might be a good idea.
> 
> 2 Gb sounds pretty big to me, but it might be the right number.  What's the
> OS limit that we're trying to avoid here?

bienvenu, what do you think?

Comment 15

[:aceman]

I have a test folder of 2,1GB (600 000 messages), so if there is any patch I can try it. The performance on it is not bad, I am doing some tests.

I didn't get a definitive answer what is the max size of mbox file that TB supports. I think it is 4GB on linux, but I've seen reports 2GB already has problems. I'd vote for showing folders exceeding 2GB.

Comment 16

[Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland]

:bienvenu in an email about another bug confirmed the max folder size on all platforms is 4gb, so i also vote for showing folders exceeding 2GB

Comment 17

[Mark Banner (:standard8)]

Comment on attachment 524103 [details] [diff] [review]
First cut

Removing no longer needed request.

Comment 18

[:aceman]

Sid0, are you working on this? Can you move it further? David Bienvenu pointed out that the 4GB limit is no longer valid (but probably not on all platforms/filesystems) so this can still be useful.

Comment 19

[Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland]

(In reply to :aceman from comment #18)
> Sid0, are you working on this? Can you move it further? David Bienvenu
> pointed out that the 4GB limit is no longer valid (but probably not on all
> platforms/filesystems) so this can still be useful.

:aceman which platforms/filesystems is the 4GB limit still valid on ?

Comment 20

[:aceman]

I am not sure about that. David Bienvenu would be the authoritative source. I am not sure how he meant it (if it is only lifted in some backend architecture and still needs more work in various places).

But I'd bet FAT32 filesystem would still have the limit.

Comment 21

[:aceman]

For reference I am talking about bug 721925.

Comment 22

[David :Bienvenu]

It's true that FAT file systems won't allow files > 4GB. This is something the berkeley mailbox pluggable store should special-case. I'm not sure how common FAT file systems are in this day and age. Macs have used HFS Plus for a long time now, which doesn't have a reachable limit, as I understand it.

Comment 23

[:aceman]

Internally, we have removed the 4GB limit for folder size. So it can only fail if the filesystem does not allow a larger file (mbox).

I think there is some way to get the filesystem type (https://searchfox.org/comm-central/source/mailnews/test/resources/mailTestUtils.js#136) so we could display when FAT is used.

We could do indicators like this one when an overall infrastructure for these (how to display them nicely) is created in bug 1517958.

Comment 24

[Magnus Melin [:mkmelin]]

Not sure this is worth doing as it would be the specific over 4G on FAT filesystems... which will be very rare by now.