Closed Bug 122391 Opened 23 years ago Closed 12 years ago

Selective download of message headers when opening folders

Categories

(MailNews Core :: Networking: IMAP, enhancement)

Product:
MailNews Core
Component:
Networking: IMAP
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: moz-bugzilla2, Assigned: Bienvenu)

References

Details

(Keywords: perf) 

This is both a performance bug and an enhancement. When opening very large 
folders, it takes forever to open them as it's downloading the header for all 
5000 or whatever number of messages. However, I believe this could be resolved 
by only requesting a subset of the header as I have seen in other mail programs 
such as if the header window is the size of 40 headers, only request 40 headers 
at a time. Even on our office is LAN, if I open up folders with many messages 
it is very slow to retrieve them all.
This is an enhancement. Not sure if it makes sense, though.

pi
Severity: normal → enhancement
Status: UNCONFIRMED → NEW
Ever confirmed: true
To clarify, when opening a messages folder, only a certain number of message 
header should be requrested at a time. Maybe if the screen is sized for showing 
30 messages, it should request 40 or 50 message headers. I've seen other mail 
clients do this. This would help when opening a folder with 1000 messages so 
Mozilla doesn't attempt to download ALL 1000 headers.
Summary: Opening folders should only request limited number of messages → Opening folder should only request limited number of message headers
QA Contact: huang → gchan
Trying to clarify this, can we only do a FETCH on a subset of all the messages
in a folder? I know Mulberry (by Cyrusoft) doesn't download all the headers in a
folder when opening it, but only the visible list (and maybe a few more). The
headers are requested incrementally to prevent large initial outlays of message
downloading which is very slow for large folders and users on dial-up in
Mozilla, but not in Mulberry due to not downloading the entire folder's headers
all at once.
I've done a little more research on this one. Here's an overview of what I'm
thinking from this thread:

http://groups.google.com/groups?hl=en&lr=lang_en&ie=UTF-8&safe=off&threadm=Pine.OSX.4.50L.0212020544300.806-100000%40localhost&rnum=8&prev=/groups%3Fq%3Dimap%2Bget%2Bheaders%26hl%3Den%26lr%3Dlang_en%26ie%3DUTF-8%26safe%3Doff%26selm%3DPine.OSX.4.50L.0212020544300.806-100000%2540localhost%26rnum%3D8

"Pine doesn't save headers and neither does Mulberry (if you use
it in online-only mode). But, both of these IMAP clients open a
mailbox -- even a jumbo mailbox -- pretty quickly. Pine does it
by only downloading the headers that it needs to display the
current page of the Pine Message Index and Mulberry does it by
only downloading as many message headers as you've configured in
your Mulberry Preferences."

I've been trying to convert some people at work to Mozilla from Outlook (with
the great spam support being the key) and these people often have over 1000,
2000, or more messages in a folder. So when they open the folder initially it is
very slow even on the office LAN it takes a little bit of time. So this option
to not download all the headers would be a great performance benefit. Of course,
it couldn't be used it the user wanted to read in a threaded messages view, but
in other situations it could save much time, especially on slow connections.
One additional excert from http://www.itd.umich.edu/imap/Mulberry.html :

"Mulberry allows user to set limit on number of message headers fetched at one
time. This means if you have 1000 messages in your INBOX, you can limit the
number of message headers fetched at once to, say, 50, allowing you to open your
mailbox quickly. Fetching headers for all 1000 would be very time-consuming.
It's relatively easy to "window" through a mailbox to view all your messages --
but might be somewhat unintuitive to uninitiated users. In this mode, when the
server receives new mail, the user selects "Display Next Block" to see new
messages."
The old owner is no longer with us.

Additionally, from imap.org:

" Finally, the ability of IMAP software to use "server-based searching and
selection to minimize data transfer" should not be underestimated. This is one
of the key advantages of an application-specific messaging protocol in
comparison to a general purpose file access protocol (or POP in pseudo-online
mode, for that matter). When the client-server protocol does not include
primitives to ask the server to do searching, then searching implies
transferring all of the data over the net to the client. Being able to ask the
server to find relevant messages by searching its local file is a huge win."

This is one of the two biggest problems I have in getting people to switch to
Mozilla Mail. The people have large mailboxes and Outlook will not download all
headers at once so it seems much quicker. Slower connections will greatly
appreciate this. Even on the same LAN as the office mail server, downloading a
few thousand headers takes too long.
Assignee: mscott → bienvenu
Keywords: mail1, perf
Summary: Opening folder should only request limited number of message headers → Selective download of message headers when opening folders
*** Bug 221249 has been marked as a duplicate of this bug. ***
I've implemented this before but it never made it into the product - it was
called imap headers on demand - the request to paint a header triggered a fetch
of the headers to be displayed. It's a fair amount of work to re-implement it.
Also, bear in mind about this quote "Finally, the ability of IMAP software to
use "server-based searching and selection to minimize data transfer" also
applies to message bodies, which we don't download but can still search on.

Also, I believe mscott is still alive and kicking :-)
Thanks for the response. mscott's comment next to his name said (gone) and with
him being netscape.com I assumed that he was no longer involved. As for what you
had previously implemented, would it work such that only a subset of headers
(some amount slightly larger than what needed to be currently displayed) would
be downloaded. Then if you scrolled it would chunk download more messages? I
used to use Cyrusoft's Mulberry and it behaved this way.

Yes, I agree that the comment from imap did go beyond the scope of this bug.

With that all being said, I would still consider this an important issue as
Outlook's interaction with IMAP or atleast with Exchange servers seems to
emulate this behavior to be friendly to slow connections and / or large folders.
Whether they should or not, people have folders with extremely large number of
messages and Mozilla acts like a dog on the first load. The appearance of this
could be greatly improved by selective header download.
Scott's now actively working on Thunderbird. What I did before worked as you
described. However, it's a lot of work to make it work again and I think/hope
there might be a simpler approach, i.e., allowing the user to read messages
while headers are getting downloaded, instead of making the user wait for all
the headers to get downloaded. Would that be acceptable to your users, do you
think? Or would it be confusing? Once we've downloaded the headers the first
time, they should be able to stay in sync without a lot of downloading. Also, in
1.6, as I said in a previous bug, downloading headers on a LAN is about twice as
fast as it was in 1.5, so that should help.
That might a start in allowing users to read messages while downloading headers.
However, this still seems to be an issue of downloading massive amounts of data
when it's not necessary and other similar applicatins (Outlook and Mulberry
atleast) take the more conservative approach. Downloading all of the headers
could flood the user's connection.

What are the issues associated with doing a selective download of headers? I
know sorting on fields that are not server sortable might be an issue and still
would require download of all headers.

How about something along these lines, enough headers are downloaded initially
to fill the display (maybe a few more), the newest messages perhaps. The
interface would then become functional and a user could read specific messages.
Now if we could somehow throttle down the downloading of headers perhaps then
connection flooding might not be an issue. I'm not sure how to do this. 

Here's how Mulberry works:

1) When you loaded the folder if 18 messages were visible, it would download
maybe 20 headers.
2) If you scrolled, it would download that chunk of say 20 more message headers
3) If the user clicked to sort on a column, it would basically give them the
option of downloading all headers or sorting the currently downloaded set of
messages.

The ability to have it download all headers as an option is not bad, but if
something similar to the above were implemented as the default it would make the
use of large folders and/or slow connections much more bareable.
To be clear, i filed my bug because i didn't want any emails
downloaded.

I have >180k emails in inbox and i was intentionally trying to avoid
having mozilla download any of them, but i needed access to the sent
and drafts folder. the last time i needed to do this i got various
errors from mozilla about 'already subscribed' when i did things
like save as draft or send. so this time i subscribed, but i was very
careful *not* to select the inbox.
 
Now i've read the original bug description and skimmed the rest of
this bug. I've reopened my bug because i don't think the direction of
this bug coincides with the intent of my bug.

For some reason mozilla mail decided to download my inbox's headers
anyway. this was disastrous, partly because it costs a few hundred
megabytes of ram, and partly because mozilla hung without any
explanation or causality. i presumed that mozilla hung randomly but
couldn't figure out why the disk was constantly spinning.

As for selective downloading. I try to have about 500 unread emails
in my inbox (it's more than that when I vacation from mozilla).
If I could have:
Download [All|Unread|New|v] messages.
Then I would probably select Unread or New (I'd argue that Unread
should be the default), but only if I remembered to do so. If I
didn't remember then I'd get hung and be right back to the bug which
I filed.
timeless: If the selective download of messages were say to start with only a
slightly smaller group of headers than the number of headers visible in the
display area (say 20 if the size of the window were to show 18 headers) and not
download any more headers until either:

1) you attempted to scroll (in which case download a few more headers)
2) you attempted to sort on a field other than message number (in which case you
should be prompted if you want to sort on existing downloaded headers or to
download all)

Would this solve your issue as no more than a few messages would get downloaded
unless you took action to download additional headers in one of the 2
possibilities mentioned above?

From my testing and previous use of Cyrusoft's Mulberry, this is how it behaves.
Would it satisify your needs in addressing your reopened issue?
I could be wrong of course but my guess is that you have the junk mail controls
turned on, and that all your unread message bodies (not just headers) are
getting downloaded in order to analyze them for spam. If you have the junk mail
controls turned on, try turning them off and let us know if the behaviour improves.
steve wardell: please reread comment 12, especially:
 so this time i subscribed, but i was very
 careful *not* to select the inbox.

I never selected the inbox, so your discussion of views into it
does not apply. bienvenu /might/ be right, although it could just as
easily have been biff doing me the 'favor' of checking for 'email'.

In the past I've had biff tell me that it found *all* of the emails
in my Inbox (not just the New or Unread ones). Personally, I'd
probably bet on biff, because while i did consider disabling junkmail
i don't think i considered disabling biff. but again the box is not
available. i'll probably return to it next spring at which time i'll
look into complaining about something else.

Bienvenu: i don't have the box, i didn't change many settings before
I just downloaded, extracted and ran mozilla, the only significant
thing I did was make sure imap knew what folders were subscribed.
History: setup a mozilla in spring 2001, uninstall it (leave
profile). come back in fall 2003, install mozilla, try to send an
email. get upset. Oh the only other thing I can offer is that there
was no 400mb mailbox file which I'd expect to have if junkmail
actually downloaded the emails.

note that sometimes i'm on dialup instead of dsl, which would make
the behavior even worse than it was (the box happened to be a p3/500
which is faster than raistlin, but it had a lot less ram).

And again, this entire thread is offtopic. it's just an explanation
of why I want my bug fixed independently and why i don't think the
fixes described in this bug will fix my problem+bug.

I don't care about junk mail or anything else, although I think I
might have considered disabling it. (i'm not sure if i did nor do i
know if i would have disabled it before or after the problem.)

it doesn't matter. the first time i use mozilla mail to send an email
i should not hang, and i really don't think the design decisions
here will fix my problem. if junk mail wants to download 100,000
emails, then it should ask me before it does so. and that's one way
that the bug i filed would address a possible cause.

again it's really not the focus of this bug.

All I want is a simple dialog box, and that's what the bug I filed
requests. I don't care which codepaths happen to tickle the dialog,
so long as I get it whenever mozilla mail decides it has nothing
better to do than download lots of mail.

bienvenu: I know it's nice to think of people as developers, but when
I file bugs based on random boxes, it's because I'm an end user who
wanted to use mozilla quickly for a simple task, and not spend hours
or days or months configuring it not to bite, choke, cough, crash,
hang, shred, or otherwise interfere with the relatively simple task
with which it was presented. (Task: write an email, save as draft,
send email.)

As an end user, I expect things to tolerate old profiles even if I
don't remember that there's a profile on the box (the profile's
existence surprised me). If mozilla doesn't want to deal with my old 
profiles, then mozilla should tell me that it found an old profile
but is concerned that using it might cause harm. This too is not the 
subject of this bug.
Timeless, to be blunt, my comments weren't addressed to you and I don't even
know what you're ranting about.
David: Any suggestions about trying to get this addressed before v1.0? This
created a very negative perception again yesterday when I was setting up
Thunderbird for someone. The user had 5000 emails in their inbox. Thunderbird
sat there for like 5-10 minutes downloading (on low end SDSL connection) while
Outlook opened right up and was almost immediately functional showing all
folders and showing messages in the INBOX.
I did start working on parts of this but got pulled off on other things. Did
Thunderbird spend all that time downloading flags and message hdrs? Or was some
of the time spent downloading message bodies for the spam filter (for the unread
messages in the folder)? I would think it would take a lot less time than 5-10
minutes to download 5000 headers over SDSL, but I can imagine that downloading
the message bodies for the spam filter would take some time...While that's going
on, you can read messages.

I can try to find some time to look at this once the Mozilla tree opens for
Seamonkey 1.8.
The setup was a new install of Thunderbird with Thunderbird's default settings.
The user's folder had some new unread messages in it, but I wouldn't say that
was an extensive amount of unread messages.  Would spam filters be enabled for a
fresh install of TB? The initial load is probably critical. This situation is a
first time user's setup of an account and should become functional as soon as
possible is my thinking to create the best experience. If there's a default
setting in TB that causes loading of full messages, perhaps additional changes
could be made to improve this experience.
*** Bug 216949 has been marked as a duplicate of this bug. ***
Thunderbird equivalent added: Bug 252644
Product: MailNews → Core
David: Did you happen to have any time to look at this? You had mentioned last
year that you were going to try to look at it after trunk opened for 1.8. Just
checking to see if you had made any progress on this.
no, lots of other stuff got in the way :-( I don't know when I'll get to this.
Blocks: 252644
Product: Core → MailNews Core
QA Contact: grylchan → networking.imap
I did implement chunked header fetching, so we fetch the newest XX headers, and then fetch the rest in chunks so the user can do stuff while the rest of the headers are getting downloaded.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.