776777 - Desired batch size and item size for POST requests in sync loadtests?

Status: RESOLVED WONTFIX

(Cloud Services Graveyard :: Server: Sync, defect, P4)

Description

[Ryan Kelly [:rfkelly]]

The sync loadtests include the use of "POST /storage/collection" to stores batches of items.  We need to know an appropriate number of items to sent per batch, and an appropriate size per item, so that this emulates what would be sent by a real FF sync session.

The old grinder-based sync loadtests send batches of 10 records, each with a random payload size approximately between 500 and 2000 bytes.  I feel like this is too few records to be realistic.  But if it's in the original loadtests then it was based on real data at some stage, right?

The new funkload-based sync loadtests send batches of 100 records, each with a random payload size approximately between 500 and 2000 bytes.  This seems more realistic to me, but I don't have hard data to back that up.

So: to generate realistic load, how many items should we send per batch, and how big should each item payload be?

Some details on client behaviour from IRC:
------
<nalexander> rfkelly: android sync also uploads in batches of 100 records.
<nalexander> rfkelly: sorry, 50 records, max 1mb per upload.
<nagios-svc-scl2> [127] mon1.scl2.svc:cepmon_sync_503 is OK: (null)
<rfkelly> nalexander: any idea of total records uploaded per session?
<rfkelly> e.g. does it usually take multiple batches to get things done?
<nalexander> rfkelly: on average? We batch per engine, so generally I expect one upload for form history, bookmarks, passwords and probably several uploads for history.
<rfkelly> nalexander thanks
<nalexander> Since we are syncing every 5 minutes.  But on phones, we may not get too many shots to sync, so we might have a very large sync every few days.
------

Comment 1

[Ryan Kelly [:rfkelly]]

------
<atoll> rnewman: POST batch size still 100?
<rnewman> 50
<rnewman> on Androuid
<atoll> oh, interesting
<rnewman> and 1MB
------

Comment 2

[Richard Newman [:rnewman]]

Dammit, my typo immortalized in Bugzilla.

My research in February suggested that the average history record was 450 bytes.

Batches for 100 items with a curve from, say, 350 up to several KB, with a mean of 450, sounds about right to me.

Comment 3

[James Bonacci [:jbonacci]]

Anything we want to do here for Sync 1.5 load tests?

Comment 4

[Ryan Kelly [:rfkelly]]

Adding Bob who did some modelling based on current sync data usage, and may be able to suggest more realistic payload breakdown for each collection time.

Comment 5

[Mark Mayo [:mmayo]]

Bob, new relic might be a good place to check for per-collection distribution sizes.

Comment 6

[Ryan Kelly [:rfkelly]]

So here's what we actually currently do.  Despite comments in the code and in this bug saying it's batches of 100 small items, it's really batches of 10 big items:

            items_per_batch = 10
            for i in range(items_per_batch):
                id = base64.urlsafe_b64encode(os.urandom(10)).rstrip("=")
                id += str(int((time.time() % 100) * 100000))
                payload = self.auth_token * random.randint(50, 200)
                wbo = {'id': id, 'payload': payload}
                data.append(wbo)
            data = json.dumps(data)
            response = self.session.post(url, data=data, **reqkwds)

The self.auth_token string is 200 bytes long, so the payload here is uniformly distributed between 10000 and 40000 bytes in length.  So yeah, that probably skews a little big!

We should target something closer to Richard's Comment 2 unless Bob has more precise requirements.  Bob, it's also feasible to use a different distribution for different collections if there are patterns that show up in the data.

Comment 7

[Ryan Kelly [:rfkelly]]

Attachment: sync-loadtest-payload-size-skew.diff

Attached patch would bring us in line with what Richard suggests above.  Putting it out there as a starting point.

Comment 8

[Mark Mayo [:mmayo]]

(In reply to Mark Mayo [:mmayo] from comment #5)
> Bob, new relic might be a good place to check for per-collection
> distribution sizes.

Ignore this. I looked and couldn't get any useful info our of New Relic re: query and/or response sizes..

Comment 9

[Tarek Ziadé (:tarek)]

Comment on attachment 8385820 [details] [diff] [review]
sync-loadtest-payload-size-skew.diff

Review of attachment 8385820 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM

Comment 10

[James Bonacci [:jbonacci]]

Contents of original email thread (read top down):

Tarek,

Chris and Mark need some input from you on transaction sizes for the loads test/loads cluster...
ie how do we vary the body/size of the transaction for TS/Sync load tests...

Thanks for getting back to us during your PTO :-)


James
---------------------------------------------------------------------------------------------------

Varying payload specifically for the sync tests - there's some concern that without trying to resemble 
the distribution of record sizes from real-world users we see in production sync, we end up with a loads 
workload on the staging servers that's too far from the real world to tell us much about how a given 
ec2 instance type will stand up in the wild.

I hope that made sense?

-Mark


---------------------------------------------------------------------------------------------------

On 5/03/2014 7:41 AM, Mark Mayo wrote:
> Varying payload specifically for the sync tests - there's some concern
> that without trying to resemble the distribution of record sizes from
> real-world users we see in production sync, we end up with a loads
> workload on the staging servers that's too far from the real world to
> tell us much about how a given ec2 instance type will stand up in the wild.

Is this essentially the same question as Bug 776777?

   https://bugzilla.mozilla.org/show_bug.cgi?id=776777


  Ryan

---------------------------------------------------------------------------------------------------


I believe that is the same ticket that bobm mentioned in the meeting.

james

---------------------------------------------------------------------------------------------------
> Le 04/03/14 23:12, James Bonacci a écrit :
>
>     Tarek,
>
>     Chris and Mark need some input from you on transaction sizes for the
>     loads test/loads cluster...
>     ie how do we vary the body/size of the transaction for TS/Sync load
>     tests...
It's already the case - payloads sizes are randomly varying between a
min and a max size

This behavior was copied over the grinder initial load tests we had a
few years ago

Comment 11

[Ryan Kelly [:rfkelly]]

Comment on attachment 8385820 [details] [diff] [review]
sync-loadtest-payload-size-skew.diff

Committed r=tarek
https://github.com/mozilla-services/server-syncstorage/commit/aa4545d7171deadfae9241a9f0d1f6f012c1762b

Comment 12

[Ryan Kelly [:rfkelly]]

So the above tweak should be a good start, but we can continue to refine based on more detailed modelling of production payload size distribution.

Comment 13

[James Bonacci [:jbonacci]]

Should we change this to Resolved and get some load tests going?

Comment 14

[Ryan Kelly [:rfkelly]]

We can get some load tests going, but I think we should leave this open pending further analysis of prod data patterns.

Comment 15

[James Bonacci [:jbonacci]]

Fair enough.

Comment 16

[Ryan Kelly [:rfkelly]]

Assigning to :bobm for further triage and/or closing depending on what ops wants out of this

Comment 17

[Jason Thomas [:jason]]

Please reopen if this still needed.