1100162 - High rate of 503s on sync-129

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[Ryan Kelly [:rfkelly]]

Several users in #sync and sumo are reporting 503s from sync-129.  Examples:

   https://support.mozilla.org/en-US/questions/1031263
   https://support.mozilla.org/en-US/questions/1031161
   https://pastebin.mozilla.org/7332411

Interestingly the errors are on requests like GET meta/global and GET crypto/keys, rather than the more usual "large POST request times out" error.  I wonder if it's related to a memcache issue, particularly because of meta/global getting involved.

Bob, any insight?

Comment 1

[Ryan Kelly [:rfkelly]]

Although, crypto/keys shouldn't be touching memcache AFAIK, so maybe it's not that...

Comment 2

[Alex]

Attachment: error-1416205445542.txt

I've been hitting this consistently on every single sync attempt on all of my systems for the last 3 days, it's kinda annoying ;)

Comment 3

[Justin Lazaro [:jlaz] (use needinfo)]

Sync-node-129 is now being migrated as we speak.  You should have node-reassignment very soon.  Sorry about the issues Alex!

Comment 4

[Justin Lazaro [:jlaz] (use needinfo)]

Node fully migrated.  According to bobm, looks like MySQL had died on this host.  Please reopen if you continue to experience any issues