972328 - [email/IMAP] Slices get too big; front-end needs to issue slice shrink requests during synchronization or back-end needs to forcibly truncate

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::E-Mail, defect)

Description

[Andrew Sutherland [:asuth] (he/him)]

Currently during large synchronizations, it's quite possible for the back-end to tell the front-end about ALL of the new messages it is seeing which is likely to be dangerous, memory-wise.

If the front-end would tell the back-end to shrink the slice as things were happening, the problem would go away.  If the back-end would just declare the front-end negligent in bounding the number of headers it knows about and unilaterally shrink the slice's subscription, the problem would go away.  (Slices as originally designed had a better idea of what the viewable range versus the buffered range was, but this complicated things for no benefit, so was discarded.  It would mainly be useful since it allows the back-end to be more informed about how it unilaterally shrinks the range.)

The front-end does not tell the back-end to shrink the slice because MessageListCard._onScroll if messageSlice.pendingRequestCount.  The (sane) rationale behind that logic was that we didn't want the front-end to queue up more than one growth request at a time, nor did it make sense to request more growth when new messages were still coming in.  It's infinite scroll with buffering, so might as well wait until we're free to do that.  It would have been smarter to only prevent growth and not prevent shrinks...


== Proposed Solution

1) Always issue shrinks from the front-end.

2) Rework pendingRequestCount so it goes to zero/goes down when database requests are filled rather than when syncs complete.  It still makes sense to avoid repeatedly requesting growth before the back-end has filled our request, but it's bad UX to prevent our infinite scroll from working.  This would be important for the virtual coordinate space change bug 796474 as well as the mutex killing bug 888807.  So during an inbox open after we fetch from the DB we'd set pendingRequestCount to 0 when the headers hit the front-end even though we're only just kicking off the refresh at that point.

It may make sense to try and clean up the event generation "requested" and "moreExpected" booleans while doing this since (requested && !moreExpected) is what drives the oncomplete notification and also sets pendingRequestCount to zero.  I think the flags had grander purposes at some point, but it would probably make more sense to either normalize it and "status" to use a more explicit event bus, or just to be tracking "syncStatus" and "loadStatus"/"loading", and have the latter be what stops us from requesting more growth until satisfied.

3) Make the slice-opener specify a failsafe limit for the maximum number of headers it should know about.  When adding messages to a slice, the back-end will enforce this limit by unilaterally performing a shrinkage.  The viewslice protocol was designed so the back-end could do stuff like this without desynchronization; the front-end is only ever making an advisory request.

== Sync Context / Triggers

These situations can trigger the sync problem:

- Initial or growth synchronizations where the IMAP server returns batch requests in ascending UID order (which is every server I've seen).  The problem is that:
-- our slice knows it only wants small N messages
-- the messages are coming in oldest to newest with high probability since higher UIDs correlates with newer messages (but does not guarantee it!)
-- our slice is characterized as [latched to "now", the first message we heard about because it's the oldest]
-- every header that's getting added is falls into that range!

- Refreshes covering 'now'.  These are actually much worse since for invariant and implementation simplicity reasons we do everything in our power to catch up to 'now'.  If we last synchronized 4 days ago, we explicitly sync from then until now, moving in a past-to-future direction.  Bisection logic will trigger, but the front-end won't derive the benefit.  And of course all of the messages fall in the sync range.

Comment 1

[BMO Automation]

Firefox OS is not being worked on