631414 - limit batch uploads based on the size of the packet being uploaded

Status: RESOLVED DUPLICATE of bug 569295

(Firefox :: Sync, defect)

Description

[:Atoll]

Currently we limit batch uploads to 100 records per batch.  Sync permits up to 256KB per record, which could lead to a maximum size of 25MB incoming data per batch.  At an average row size of 1KB, we would see 100KB per batch.

It would be helpful to introduce an actual byte size limit for these batches, so that we can avoid conflicts with production network buffers (*).  Capping the batches at a certain size will also help reduce the impact of interrupted batches, both in bytes transmitted (at a cost per byte, for some) and in user experience (retransmits of a batch cause less delay with smaller batches).

Mobile devices may benefit from a lower batch size limit than desktop devices, which would help keep memory usage down during syncs of a lot of data and reduce the impact of interrupted uploads over cellular.

If there are concerns about large rows leading to single row batches, we could also set a minimum number of rows per batch (say, 10) to protect the service from small-batch inefficiencies.

I tentatively suggest that a batch size of 128KB is sufficient for desktops (4 seconds per batch on a 256Kbps upload DSL circuit), and have no strong opinion on mobile (less is probably good).  We can afford a few extra back and forth cycles through the API in exchange for predictable incoming batch sizes.

(*) see bug 631401, which affects primarily cellular data users uploading batches larger than 64KB.