1986399 - [ews] Download messages for offline use

Status: ASSIGNED

(MailNews Core :: Networking: Exchange, task, P1)

Description

[Ben Campbell]

EWS folders should automatically download messages in the background for offline use, according to offline-storage settings (folders which are marked for offline use, and whatever other policy controls there may be).

Background downloading should be preempted by user-initiated fetches - i.e. if a bunch of messages are downloading in the background, clicking on not-yet-downloaded message to view it should take priority.

Ideally such downloading should take place during idle times, and messages should be prioritised to anticipate the users needs. e.g. the user is more likely to view recent messages that extremely old ones, say.

Comment 1

[Ben Campbell]

Attachment: Bug 1986399 - Fix spurious assert when preempting writes to mbox. r=#thunderbird-back-end-reviewers

Comment 2

[Ben Campbell]

Attachment: Bug 1986399 - Add SyncCopyStreamN() to nsMsgUtils. r=#thunderbird-back-end-reviewers

Comment 3

[Ben Campbell]

Attachment: Bug 1986399 - Add EwsFetchMsgToOffline() to asynchronously download messages from server. r=#thunderbird-back-end-reviewers

Comment 4

[Ben Campbell]

Attachment: Bug 1986399 - Use EwsFetchMsgToOffline in EwsMessageChannel. r=#thunderbird-back-end-reviewers

Comment 5

[Ben Campbell]

try run with the first 3 patches looks OK
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=d8354c897ac7646c881c046fddd7eb64b5cfe247

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/97feb2a6ffcb
Fix spurious assert when preempting writes to mbox. r=jtracey
https://hg.mozilla.org/comm-central/rev/c57950cf5078
Add SyncCopyStreamN() to nsMsgUtils. r=jtracey
https://hg.mozilla.org/comm-central/rev/c025c069ee89
Add EwsFetchMsgToOffline() to asynchronously download messages from server. r=jtracey

Comment 7

[Ben Campbell]

Attachment: Bug 1986399 - Remove utility function nsIAutoSyncState.isSibling() from public interface. r=#thunderbird-back-end-reviewers

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/46bab7f041c3
Use EwsFetchMsgToOffline in EwsMessageChannel. r=babolivier

Comment 9

[Ben Campbell]

Attachment: Bug 1986399 - Remove unused local vars from nsImapMailFolder::DownloadAllForOffline(). r=#thunderbird-reviewers

Comment 10

[Ben Campbell]

Attachment: Bug 1986399 - Isolate IMAP-specific bits of nsAutoSyncState::DownloadMessagesForOffline() to prep for EWS support. r=#thunderbird-back-end-reviewers

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/e386a0172a64
Remove utility function nsIAutoSyncState.isSibling() from public interface. r=mkmelin
https://hg.mozilla.org/comm-central/rev/881d6df338b1
Remove unused local vars from nsImapMailFolder::DownloadAllForOffline(). r=mkmelin

Comment 12

[Ben Campbell]

Attachment: Bug 1986399 - Modify EwsFetchMsgToOffline() to handle multiple messages. r=#thunderbird-back-end-reviewers

Comment 13

[Ben Campbell]

Oh, I should explain where I'm going with this stuff.
The existing nsIAutoSyncManager already has most of the features we want. It's a bit IMAP-specific, but actually, it looks like it won't be too hard to add support for EWS folders, so that's what I'm doing.

The main barrier is understanding nsAutoSync code - it could do with a big refactor and deCOMtamination - but just searching through for IMAP-specific bits and adding an EWS equivalent seems to work a treat.
I'm trying to land small, digestible patches, which don't need a full understand of all autosync to review.

My ongoing autosync notes are here: https://github.com/bcampbell/tb-notes/blob/master/autosyncmanager.md
(eventual plan is to snip off the gory details and turn it into actual source documentation)

Comment 14

[Ben Campbell]

Attachment: Bug 1986399 - Generalise nsAutoSyncState ctor for use with non-imap folders. r=#thunderbird-back-end-reviewers

Comment 15

[Ben Campbell]

Attachment: Bug 1986399 - Move .autoSyncStateObj from nsIMsgImapMailFolder to nsIMsgFolder. r=#thunderbird-back-end-reviewers

Comment 16

[Ben Campbell]

Attachment: Bug 1986399 - Temporary workaround for Bug 1987826 (zero .messageSize in EWS) to get EWS autosync working. r=#thunderbird-back-end-reviewers

Comment 17

[Ben Campbell]

Attachment: Bug 1986399 - Add an nsAutoSyncState object to EwsFolder. r=#thunderbird-back-end-reviewers

Comment 18

[Ben Campbell]

Attachment: Bug 1986399 - Move autoSync server pref attrs from nsIImapIncomingServer to base nsIMsgIncomingServer. r=#thunderbird-back-end-reviewers

Comment 19

[Ben Campbell]

Attachment: Bug 1986399 - Set Offline flag where appropriate for EWS folders. r=#thunderbird-back-end-reviewers

Comment 20

[Ben Campbell]

Attachment: Bug 1986399 - Add AutoSyncManager queue-new-messages-for-download support to EwsFolder. r=#thunderbird-back-end-reviewers

Comment 21

[Ben Campbell]

With these patches, EWS folders should auto-download new messages when they arrive, using the autoSync manager.

To try it out, apply these patches, then run (the IMAPAutoSyncLog is good to have on):

MOZ_LOG="IMAPAutoSync:5" ./mach run

Easiest way to force a check for new messages is to click on a non-INBOX folder, send yourself some messages from another email account, then click on Inbox to force a new-message check.
You should see message downloads happening in the background in the IMAPAutoSyncLog.
If you use some messages with huge attachments, you can really feel the difference as you click on them.

Still to do:

1. The account settings GUI needs some work to support the various autosync prefs.
2. the activitymanager isn't correctly showing the downloading. I see output like: JavaScript error: resource:///modules/activity/autosync.sys.mjs, line 236: TypeError: can't access property "activity", syncItem is undefined, which is likely related.
3. It doesn't yet download non-offline messages already in the folder. There is code in autosyncmanager to do this, and I think it'll be pretty easy to get working with EWS.

I'll write up proper bugs for 1 and 2 once this stuff lands.
I'll continue working in this bug for Number 3.

Comment 22

[Ben Campbell]

Attachment: Bug 1986399 - Increase autosync download group size from 50KB to 2MB. r=#thunderbird-back-end-reviewers

50KB Limit seemed a little archaic. Even the smallest of modern emails are
so bloated that they'll be pushing 30KB, so the autosync would only ever
download one or two messages at a time.

Also removes the unused (and thus misleading) nsIAutoSyncManager.groupSize
attr from the public interface.

Comment 23

[Ben Campbell]

Attachment: Bug 1986399 - Fix IMAPism in autosync.sys.mjs preventing activity manager updates. r=#thunderbird-reviewers

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/0449bed406b7
Isolate IMAP-specific bits of nsAutoSyncState::DownloadMessagesForOffline() to prep for EWS support. r=mkmelin
https://hg.mozilla.org/comm-central/rev/a9315979811c
Modify EwsFetchMsgToOffline() to handle multiple messages. r=babolivier
https://hg.mozilla.org/comm-central/rev/c8e46f3b7ca2
Generalise nsAutoSyncState ctor for use with non-imap folders. r=mkmelin
https://hg.mozilla.org/comm-central/rev/0e0e76fb868e
Move .autoSyncStateObj from nsIMsgImapMailFolder to nsIMsgFolder. r=mkmelin
https://hg.mozilla.org/comm-central/rev/b5ae13585405
Add an nsAutoSyncState object to EwsFolder. r=babolivier
https://hg.mozilla.org/comm-central/rev/9f1681645ac0
Move autoSync server pref attrs from nsIImapIncomingServer to base nsIMsgIncomingServer. r=mkmelin
https://hg.mozilla.org/comm-central/rev/3f71e164f1bb
Set Offline flag where appropriate for EWS folders. r=mkmelin
https://hg.mozilla.org/comm-central/rev/48922717b998
Add AutoSyncManager queue-new-messages-for-download support to EwsFolder. r=babolivier
https://hg.mozilla.org/comm-central/rev/b5a540092ac8
Increase autosync download group size from 50KB to 2MB. r=mkmelin
https://hg.mozilla.org/comm-central/rev/3889f61f494c
Fix IMAPism in autosync.sys.mjs preventing activity manager updates. r=mkmelin

Comment 25

[Ben Campbell]

Attachment: Bug 1986399 - Add EWS support to AutoSync periodic timer. r=#thunderbird-back-end-reviewers

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/4fcff647ae10
Add EWS support to AutoSync periodic timer. r=mkmelin

Comment 27

[Ben Campbell]

Attachment: Bug 1986399 - Use common base implementation of nsIMsgFolder.getLocalMsgStream(). r=#thunderbird-back-end-reviewers

The only reason nsIMsgFolder.getLocalMsgStream() exists is to allow the IMAP
gmail messages-in-multiple folders hack.
So it makes sense for all other folder types to share the base implementation.
The IMAP folder is the only one that needs to override it.

Comment 28

[Ben Campbell]

Attachment: Bug 1986399 - Remove redundant nsIMsgFolder.getLocalMsgStream(). r=#thunderbird-back-end-reviewers

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/a3c6045eb389
Use common base implementation of nsIMsgFolder.getLocalMsgStream(). r=edicharry,babolivier
https://hg.mozilla.org/comm-central/rev/62b82057efb0
Remove redundant nsIMsgFolder.getLocalMsgStream(). r=mkmelin