814251 - [email/IMAP] implement IMAP pipelining of FETCH requests to reduce latency

Status: VERIFIED FIXED

(Firefox OS Graveyard :: Gaia::E-Mail, defect)

Description

[Andrew Sutherland [:asuth] (he/him)]

The IMAP library we are currently using does not pipeline requests.  If we say "A01 UID FETCH 1 ...", "A02 UID FETCH 2 ...", we will not send the request for the second fetch until the first fetch has fully completed with "A01 OK".  This introduces network latency into our synchronization process that is not required.

This happens primarily during body part fetching where we issue separate fetch requests for body pats independently.  When a message contains multiple body parts we currently split them into separate FETCH requests because of another deficiency in the IMAP lib relating to a fetch request that returns multiple literals (that I may have already resolved?), which is silly and we don't need to do.  But otherwise, we issue body fetches from newest to oldest which is desirable, especially in the face of IMAP servers like Dovecot that always sort the UIDs we provide and return results in that order rather than the order we request.  (Note: This was also partially done because a FETCH request of multiple UIDs can still only specify one set of part numbers to fetch, so in theory each message may have different body part numbers, although in practice coalescing is possible.)

To try and improve a gmail unit test_compose.js test run which was taking a rather long time, I quick-hack implemented a naive implementation that can be seen here:
https://github.com/asutherland/gaia-email-libs-and-more/tree/naive-imap-pipelining

It works for gmail, especially given the consistent 100+ms latencies on turning around the I/O requests.  However, it is way too naive for dovecot which can end up returning all the fetch results and only then issuing all the "OK" results.

Comment 1

[James Lal [:lightsofapollo]]

Attachment: Github PR

Some lame tests (which could be removed or improved) and tons of manual testing on my own Zimbra/Gmail accounts... Initial working version likely some tests are still failing but ready for review.

Comment 2

[Johnny Stenback (:jst)]

tef+ as this will help email perf which is a blocking issue at this point.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

landed in gaia-email-libs-and-more/master:
https://github.com/mozilla-b2g/gaia-email-libs-and-more/pull/176
https://github.com/mozilla-b2g/gaia-email-libs-and-more/commit/6fad178747b5eea92f6cf324b5ff1719800204ae

landed in gaia/master:
https://github.com/mozilla-b2g/gaia/pull/9189
https://github.com/mozilla-b2g/gaia/commit/320a0f6a3738cb4dffbe7ba22a8510a2cbd10351

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

Uplifted commit 320a0f6a3738cb4dffbe7ba22a8510a2cbd10351 as:
v1-train: 1238936b68807c2b0a9482d27896822cde1547ce
v1.0.1: 22b9acfdb8723cb1b1c140da5cdaf943f3da907b

Comment 5

[Tracy Walker [:tracy]]

This appears to be ok from a user perspective.  Would need white box testing numbers to provide a more detailed picture of the improvements gained here.