1655147 - 503s on node sync-607

Status: RESOLVED WORKSFORME

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[anindya49]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0

Steps to reproduce:

Trying to sync tabs, bookmarks between multiple machines using the menu on the top right.

Actual results:

Synced tabs is empty, bookmarks not syncing, and there are failures logged due to server maintenance in about:sync-log. An excerpt:

1595618353212 Sync.LogManager DEBUG Flushing file log
1595618353229 Sync.LogManager DEBUG Log cleanup threshold time: 1594754353229
1595618353274 Sync.LogManager DEBUG Done deleting files.
1595618354318 FirefoxAccounts INFO Polling device commands.
1595618354318 Sync.Service DEBUG User-Agent: Firefox/78.0.2 (Windows NT 10.0; Win64; x64) FxSync/1.80.0.20200708170202.desktop
1595618354319 Sync.Service INFO Starting sync at 2020-07-24 12:19:14 in browser session w-qwttqoo4fm
1595618354319 Sync.Service DEBUG In sync: should login.
1595618354319 Sync.Service INFO User logged in successfully - verifying login.
1595618354320 FirefoxAccounts TRACE not checking freshness of profile as it remains recent
1595618354321 Sync.BrowserIDManager DEBUG unlockAndVerifyAuthState already has (or can fetch) sync keys
1595618354321 Sync.Status DEBUG Status.login: error.sync.reason.serverMaintenance => success.status_ok
1595618354321 Sync.Status DEBUG Status.service: error.login.failed => error.login.failed
1595618354321 Sync.Service DEBUG Fetching unlocked auth state returned success.status_ok
1595618354322 FirefoxAccounts TRACE not checking freshness of profile as it remains recent
1595618354325 Services.Common.RESTRequest DEBUG GET request to https://api.accounts.firefox.com/v1/account/device/commands?index=0
1595618354366 Sync.Resource DEBUG GET fail 503 https://sync-607-us-west-2.sync.services.mozilla.com/1.5/93252030/info/collections
1595618354366 Sync.Resource WARN GET request to https://sync-607-us-west-2.sync.services.mozilla.com/1.5/93252030/info/collections failed with status 503
1595618354366 Sync.Status DEBUG Status.login: success.status_ok => error.login.reason.server
1595618354366 Sync.Status DEBUG Status.service: error.login.failed => error.login.failed
1595618354366 Sync.ErrorHandler DEBUG Got Retry-After: 3600
1595618354366 Sync.Status DEBUG Status.login: error.login.reason.server => error.sync.reason.serverMaintenance
1595618354367 Sync.Status DEBUG Status.service: error.login.failed => error.login.failed
1595618354367 Sync.SyncScheduler DEBUG Got backoff notification: 3600000ms
1595618354367 Sync.SyncScheduler DEBUG Fuzzed minimum next sync: 1595621954367
1595618354367 Sync.ErrorHandler ERROR Sync encountered a login error
1595618354368 Sync.SyncScheduler DEBUG Clearing sync triggers and the global score.
1595618354369 Sync.SyncScheduler DEBUG Next sync in 3643410.6499600704 ms. (why=schedule)
1595618354370 Sync.Service DEBUG Exception calling WrappedLock: Error: Login failed: error.sync.reason.serverMaintenance(resource://services-sync/service.js:1043:15) JS Stack trace: [e-mail]:1043:15
1595618354371 Sync.Service DEBUG Not syncing: login returned false.
1595618354372 FirefoxAccounts TRACE not checking freshness of profile as it remains recent

Expected results:

Normal syncing behaviour: Tabs show up in synced tabs, bookmarks get updated.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 2

[Lina Butler (ex-Mozilla)]

Hi! That looks like a server hiccup for sure, but looks like that node is back up now. Are you still seeing issues?

Comment 3

[anindya49]

Thanks! I did not change anything on my end. Unfortunately sync is still failing. Excerpt from just now:

1595627519624 Sync.Resource DEBUG GET fail 503 https://sync-607-us-west-2.sync.services.mozilla.com/1.5/93252030/info/collections
1595627519624 Sync.Resource WARN GET request to https://sync-607-us-west-2.sync.services.mozilla.com/1.5/93252030/info/collections failed with status 503
1595627519624 Sync.Status DEBUG Status.login: success.login => error.login.reason.server
1595627519624 Sync.Status DEBUG Status.service: success.status_ok => error.login.failed
1595627519624 Sync.ErrorHandler DEBUG Got Retry-After: 3600
1595627519624 Sync.Status DEBUG Status.login: error.login.reason.server => error.sync.reason.serverMaintenance

All my machines report a failure from sync-607-us-west-2.sync.services.mozilla.com. It started about 18 hrs ago. If I load the site directly in my browser I get the "It works!" message but something is still failing.

Comment 4

[Lina Butler (ex-Mozilla)]

Thanks for double-checking, and sorry you're still seeing issues!

Hey :bobm, could you please check out this node at your convenience? Thanks!

Comment 5

[anindya49]

Thank you very much for the quick response! I'll keep checking and update if anything changes.

Comment 6

[Mjules]

Hello,

I can confirm I've still got the same issue with the same server.
Anything I can do to help debug?

Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0

Comment 8

[ngreentux]

Add another one to the list of people having this same issue. Has been happening for me since around about early to mid friday New Zealand Time (so about 10am'ish)

Comment 9

[deerfieldtech]

Attachment: error-sync-1595854858864.txt

Was about to report the same problem, none of my devices can sync for about 5 days now:

Comment 10

[deerfieldtech]

Was about to report the same problem, none of my devices can sync for about 5 days now.

https://sync-607-us-west-2.sync.services.mozilla.com/ reports: "It Works! SyncStorage is successfully running on this host."

Snip from recent (five minutes ago) weave log:

1595854858358 FirefoxAccounts TRACE not checking freshness of profile as it remains recent
1595854858363 FirefoxAccounts INFO Already fetching device list, return existing promise
1595854858363 Sync.BulkKeyBundle INFO BulkKeyBundle being created for undefined
1595854858856 Sync.Resource DEBUG GET fail 503 https://sync-607-us-west-2.sync.services.mozilla.com/1.5/92318095/info/collections
1595854858856 Sync.Resource WARN GET request to https://sync-607-us-west-2.sync.services.mozilla.com/1.5/92318095/info/collections failed with status 503
1595854858857 Sync.Status DEBUG Status.login: success.login => error.login.reason.server
1595854858857 Sync.Status DEBUG Status.service: success.status_ok => error.login.failed
1595854858857 Sync.ErrorHandler DEBUG Got Retry-After: 3600
1595854858857 Sync.Status DEBUG Status.login: error.login.reason.server => error.sync.reason.serverMaintenance
1595854858857 Sync.Status DEBUG Status.service: error.login.failed => error.login.failed
1595854858857 Sync.SyncScheduler DEBUG Got backoff notification: 3600000ms
1595854858857 Sync.SyncScheduler DEBUG Fuzzed minimum next sync: 1595858458857
1595854858858 Sync.ErrorHandler ERROR Sync encountered a login error
1595854858858 Sync.SyncScheduler DEBUG Clearing sync triggers and the global score.
1595854858859 Sync.SyncScheduler DEBUG Next sync in 3610321.4949290073 ms. (why=schedule)
1595854858862 Sync.Service DEBUG Exception calling WrappedLock: Error: Login failed: error.sync.reason.serverMaintenance(resource://services-sync/service.js:1043:15) JS Stack trace: [e-mail]:1043:15
1595854858862 Sync.Service DEBUG Not syncing: login returned false.
1595854858863 FirefoxAccounts TRACE not checking freshness of profile as it remains recent

Comment 11

[anindya49]

Hi, as of this morning I am able to sync again. Will keep monitoring. Does it work for anyone else?

Comment 12

[ngreentux]

(In reply to anindya49 from comment #11)

Hi, as of this morning I am able to sync again. Will keep monitoring. Does it work for anyone else?

Can confirm sync is now functional again for me as well. Will also continue to keep an eye on it

Comment 13

[Mjules]

The same here, sync works again between my devices.

Comment 14

[Lina Butler (ex-Mozilla)]

Thanks for confirming, everyone!