1062115 - Please deploy server-syncstorage 1.5.9 to stage

Status: VERIFIED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Ryan Kelly [:rfkelly]]

This version of syncstorage contains two small fixes coming out of the previous deployment:

    Bug 1060153 - check for non-null connection before clearing _result attr
    Bug 1060153 - add more error codes to the retry list

The latter should add a bit of additional protection against the TokuDB strangeness we were seeing in Bug 1059025 and friends.

Let's deploy this to stage, and hopefully through to production, with the planned TokuDB storage engine in place.

Comment 1

[James Bonacci [:jbonacci]]

Yes, let's do this. I closed out bug 1059025 for now.
We can open new bugs, if needed, as we learn more...

Comment 2

[Bob Micheletto [:bobm]]

This has been deployed to stage.

Comment 3

[James Bonacci [:jbonacci]]

OK, I will start on this Thu afternoon during QA's Loads workshop #2...

Comment 4

[James Bonacci [:jbonacci]]

:rfkelly did you mean for two of the same bug listed here?
https://bugzilla.mozilla.org/show_bug.cgi?id=1062115#c0

Comment 5

[Ryan Kelly [:rfkelly]]

Whoops, that should be Bug 1059653.

Comment 6

[James Bonacci [:jbonacci]]

OK. I verified that we are using the same 4 c1.xlarge instances:
ec2-174-129-114-111
ec2-54-224-103-67
ec2-54-83-130-1
ec2-107-22-37-43

Verified the new code version: server-syncstorage 1.5.9-1.el6 x86_64 41926595

Verified processes, files, logs.

Will start load testing around 1:30 - 2PDT

Comment 7

[James Bonacci [:jbonacci]]

FYI - all the quick tests were good.

Comment 8

[James Bonacci [:jbonacci]]

Running 30min load tests now.
Will run a 60min combined after that.

Comment 9

[James Bonacci [:jbonacci]]

Short, single-node load tests looked good:
https://loads.services.mozilla.com/run/60f20fd9-1a1a-4fcd-b1c8-5730075f963a
https://loads.services.mozilla.com/run/25c3eb05-8da5-4cfb-8425-a2c66309438d
https://loads.services.mozilla.com/run/966e3d8a-338e-43d2-933e-f20f7e166619
https://loads.services.mozilla.com/run/47950443-618d-4d43-91d6-7e5bbf4f20d5

60min combined load test also looked good:
https://loads.services.mozilla.com/run/02c31db8-bd30-48a1-a2c2-719b363c1235



For the nginx access logs, nothing unexpected seen.

For the applications logs:
sync.err logs have the usual "Exception KeyError: KeyError(20668080,)..." messages

sync.log shows the usual messages plus a whole mess of these on Sync1 and Sync4 from yesterday and the day before:
{"time": "2014-09-04T02:13:13.432809Z", "v": 1, "message": "Large timestamp skew detected: -252", "hostname": "ip-10-158-40-250", "pid": 15962, "op": "mozsvc.user", "name": "mozsvc.user"}

not sure what happened there...

Sync2 log is long enough I still see the older "{"error": "InternalError('(InternalError) (1032..."
filtering that out shows an older 999 and some 'skew detected' messsages

Finally the '406s' are here and there, but we have seen those before...

So, I call this good.