Please deploy server-syncstorage 1.5.9 to stage

VERIFIED FIXED

Status

Cloud Services
Operations: Deployment Requests
VERIFIED FIXED
3 years ago
3 years ago

People

(Reporter: rfkelly, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [qa+])

(Reporter)

Description

3 years ago
This version of syncstorage contains two small fixes coming out of the previous deployment:

    Bug 1060153 - check for non-null connection before clearing _result attr
    Bug 1060153 - add more error codes to the retry list

The latter should add a bit of additional protection against the TokuDB strangeness we were seeing in Bug 1059025 and friends.

Let's deploy this to stage, and hopefully through to production, with the planned TokuDB storage engine in place.
Yes, let's do this. I closed out bug 1059025 for now.
We can open new bugs, if needed, as we learn more...
Status: NEW → ASSIGNED
Whiteboard: [qa+]
This has been deployed to stage.
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
OK, I will start on this Thu afternoon during QA's Loads workshop #2...
:rfkelly did you mean for two of the same bug listed here?
https://bugzilla.mozilla.org/show_bug.cgi?id=1062115#c0
(Reporter)

Comment 5

3 years ago
Whoops, that should be Bug 1059653.
OK. I verified that we are using the same 4 c1.xlarge instances:
ec2-174-129-114-111
ec2-54-224-103-67
ec2-54-83-130-1
ec2-107-22-37-43

Verified the new code version: server-syncstorage 1.5.9-1.el6 x86_64 41926595

Verified processes, files, logs.

Will start load testing around 1:30 - 2PDT
FYI - all the quick tests were good.
Running 30min load tests now.
Will run a 60min combined after that.
Short, single-node load tests looked good:
https://loads.services.mozilla.com/run/60f20fd9-1a1a-4fcd-b1c8-5730075f963a
https://loads.services.mozilla.com/run/25c3eb05-8da5-4cfb-8425-a2c66309438d
https://loads.services.mozilla.com/run/966e3d8a-338e-43d2-933e-f20f7e166619
https://loads.services.mozilla.com/run/47950443-618d-4d43-91d6-7e5bbf4f20d5

60min combined load test also looked good:
https://loads.services.mozilla.com/run/02c31db8-bd30-48a1-a2c2-719b363c1235



For the nginx access logs, nothing unexpected seen.

For the applications logs:
sync.err logs have the usual "Exception KeyError: KeyError(20668080,)..." messages

sync.log shows the usual messages plus a whole mess of these on Sync1 and Sync4 from yesterday and the day before:
{"time": "2014-09-04T02:13:13.432809Z", "v": 1, "message": "Large timestamp skew detected: -252", "hostname": "ip-10-158-40-250", "pid": 15962, "op": "mozsvc.user", "name": "mozsvc.user"}

not sure what happened there...

Sync2 log is long enough I still see the older "{"error": "InternalError('(InternalError) (1032..."
filtering that out shows an older 999 and some 'skew detected' messsages

Finally the '406s' are here and there, but we have seen those before...

So, I call this good.
Status: RESOLVED → VERIFIED
Blocks: 1063278
You need to log in before you can comment on or make changes to this bug.