823384 - [email/IMAP] support synchronizing less than a day's worth of messages via partial sync

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::E-Mail, defect, P3)

Description

[Andrew Sutherland [:asuth] (he/him)]

Currently, the smallest set of messages we can synchronize is one day's worth.  This was an initial design simplification motivated by IMAP time-based search queries being limited to day granularity and the understood target market.  This becomes problematic because syncs hold a mutex on the folder for their duration and is our current database commit granularity.

The proposed solution is to enhance our implementation so that we can synchronize a smaller portion of a day.  We will do this by detecting when our search is constrained to a single day and we have N new messages that exceeds our limit K, and in that case issuing a FETCH command to get the INTERNALDATE for the N messages so that we can order the N uids.  We will then take either the newest or oldest K messages, depending on the direction of our sync.

Explicit sync direction is a new thing being necessarily introduced by bug 822882 (~never block on network I/O to show messages).  Previously it was implicit and always past-wards as we were always synchronizing from the present into the past.  But when bug 822882 lands, we can end up in a situation where we are displaying messages from the past and must, out of necessity, grow towards 'now'/the future.  Since our concept of "growing" a slice already involved direction, this just means propagating the concept a little further.

We will, however, continue to assume that we have a continuously covered synchronization range with no gaps.  We will always grow 'outwards' from this range.  (Mainly because this is a logical consequence of our UI showing the most recent messages we have and not allowing for gaps; we could do gaps.)


Important design questions:

1) How does this affect deletion inference?  We definitely don't want to end up in a situation where we sync messages and then delete them.  And the answer is that we are modeling this implementation as a partial sync of a whole day rather than creating smaller time subdivisions than a day.

The cost tradeoff is that further time subdivisions increase server traffic and server work to synchronize a 'busy' day in its entirety.  Their upside is that they reduce the set of message headers we need to hold in memory during the sync.  Under the current and partial strategy, we load all the headers for the sync range.

However, the per-message cost is not particularly high.  Currently I estimate an upper bound of 430 bytes for the structured clone (assuming utf-8), and bloating that to 1k for in-memory JS rep, ~1000 message in a day is just a megabyte of data.  We could try and shrink things by maintaining a much more compact digest of just the mutable state for the messages, but there's a good chance we would end up loading those blocks and caching them anyways.

2) How does partial sync impact ImapFolderSyncer.growSync and avoid us erroneously thinking we have entirely synchronized a date range and therefore skipping to the next day in our sync direction?  A: We use accuracy ranges.  We have growSync find the accuracy range that covers the origin timestamp of the growth (which is the timestamp of the last header we have in that direction).  If a partial day sync occurred, then the "far" timestamp bound of the sync range will line up with that origin timestamp exactly[1] (and will be unquantized.)  The accuracy range will be this way because we will change our call to markSyncRange to use the "far" timestamp of the highest new message retrieved in the event of a partial sync.  It is possible there could exist already-known messages with a "further" timestamp, but we don't care because the accuracy range only covers time spans we have fully synchronized.

footnote 1: Grow requests always hand out all of the messages the database knows about in a batch and requires the grow requester to issue another grow request which will then be a purely network request.  This ensures that everything lines up and keeps us sane.

Comment 1

[Andreas Gal :gal]

Email is unusable without this.

Comment 2

[Andrew Overholt [:overholt]]

This should be tef+, right?

Comment 3

[Daniel Coloma:dcoloma]

Andreas, in which circumstances is email unusable?

Comment 4

[Andreas Gal :gal]

This has gotten significantly better since we only fetch headers and then snippets separately. We can probably survive 1.0.1 as is so TEF-.

Comment 5

[Daniel Coloma:dcoloma]

based iin comment 4 tef-

Comment 6

[Andrew Sutherland [:asuth] (he/him)]

We stopped working this given the happy results of all our other efforts.

Comment 7

[Ravi Dandu]

Andrew, what is the value this bug will bring, given all the other email changes ?

Comment 8

[Andrew Sutherland [:asuth] (he/him)]

The mutex issue I reference in comment 0 is not as much of a concern since we just want to address that problem via bug 888807.

In general this is only relevant for messages where there are a *lot* of messages on a single day.

There are two main benefits we could derive from variants on addressing this bug:

1) Faster initial synchronization in a folder with a ton of messages since we would inherently need to find out the timestamps of the messages and could then more confidently return the newest messages first.  This would slightly bloat our network usage and increase overall sync latency although we could choose the constants so it's usually always a perceived win to the user.

2) Reduced peak memory usage by generating smaller IndexedDB deltas and likewise holding less discardable database-state (for bodies) in memory at a given time.  This would mainly mean performing more database checkpoints during the sync process.  The bulk of the work for this is just in making sure we checkpoint a coherent/legal database state, which isn't all that much.  We may have an existing bug about doing this checkpointing already?

Priority-wise, the memory win for Tarako devices is potentially meaningful, but thus far my investigations have suggested that bug 885110 and platform improvements to actually let us use APIs on the workers would do more for us there by reducing garbage generation/etc.

Comment 9

[BMO Automation]

Firefox OS is not being worked on