997964 - No longer able to Sync on Stage with new/current accounts due to TS/Verifier error

Status: VERIFIED FIXED

(Cloud Services :: Server: Token, defect, P1)

Description

[James Bonacci [:jbonacci]]

With the latest deploy for TS/Verifier/FxA Stage, we are no longer able to sync using current Stage or new Stage accounts.

After Sign Up or Sign In, after the account has been verified (according to the UI), we are seeing the following in the sync error logs:

1397762888241	Sync.Service	INFO	Starting sync at 2014-04-17 12:28:08
1397762888241	Sync.Service	DEBUG	In sync: should login.
1397762888242	Sync.Status	DEBUG	Status.service: success.status_ok => success.status_ok
1397762888242	Sync.Status	DEBUG	Status.service: success.status_ok => success.status_ok
1397762888242	Sync.Service	INFO	Logging in user [e-mail]
1397762888243	Sync.BrowserIDManager	DEBUG	_ensureValidToken already has one
1397762888243	Sync.BrowserIDManager	DEBUG	_findCluster returning https://sync-1-us-east-1.stage.mozaws.net/1.5/BLAH/
1397762888243	Sync.Service	DEBUG	Cluster value = https://sync-1-us-east-1.stage.mozaws.net/1.5/BLAH/
1397762888243	Sync.Service	DEBUG	Setting cluster to https://sync-1-us-east-1.stage.mozaws.net/1.5/BLAH/
1397762888244	Sync.Service	DEBUG	Caching URLs under storage user base: https://sync-1-us-east-1.stage.mozaws.net/1.5/BLAH/
1397762888244	Sync.BrowserIDManager	DEBUG	_ensureValidToken already has one
1397762888882	Sync.Service	DEBUG	verifyLogin failed: Error JS Stack trace: [e-mail]:413 < [e-mail]:698 < [e-mail]:975 < [e-mail]:143 < [e-mail]:98 < [e-mail]:72 < [e-mail]:986 < sync/<@service.js:1232 < [e-mail]:72 < [e-mail]:1228
1397762888882	Sync.Status	DEBUG	Status.login: success.login => error.login.reason.network
1397762888882	Sync.Status	DEBUG	Status.service: success.status_ok => error.login.failed
1397762888882	Sync.SyncScheduler	DEBUG	Clearing sync triggers and the global score.
1397762888882	Sync.SyncScheduler	DEBUG	Next sync in 3600000 ms.


Specific config changes:
services.sync.log.appender.file.logOnSuccess;true
identity.fxaccounts.remote.force_auth.uri;https://accounts.stage.mozaws.net/force_auth?service=sync&context=fx_desktop_v1
identity.fxaccounts.remote.signin.uri;https://accounts.stage.mozaws.net/signin?service=sync&context=fx_desktop_v1
identity.fxaccounts.remote.signup.uri;https://accounts.stage.mozaws.net/signup?service=sync&context=fx_desktop_v1
services.sync.tokenServerURI;https://token.stage.mozaws.net/1.0/sync/1.5
identity.fxaccounts.auth.uri;https://api-accounts.stage.mozaws.net/v1
identity.fxaccounts.settings.uri;https://accounts.stage.mozaws.net/settings

Comment 1

[Edwin Wong [:edwong]]

Karl and myself also repro this issue.

Comment 2

[James Bonacci [:jbonacci]]

I tried syncing while watching the TS and the Verifier logs.
I saw no activity at all.
It's as if we are not ever talking to Stage.

Comment 3

[John Morrison [:jrgm]]

I looked at this with bobm, with us both running tcpdump (me on desktop and him on the sync server my client was talking to). There are definitely TCP packets being exchanged on port 443, but nothing shows up in the nginx logs on sync-2-us-east-1.stage.mozaws.net.

Comment 4

[James Bonacci [:jbonacci]]

Yep. I tried a TS load test and a Verifier load test.
All looks good.

I have accounts with sync-1-us-east-1 and with sync-2-us-east-1. Neither gets me anything, so it's not just a specific sync node in Stage (as far as I can tell).

Comment 5

[James Bonacci [:jbonacci]]

Just to confirm where we are at, I can reproduce the error by simply changing the one env var:
services.sync.tokenServerURI to point to Stage

Comment 6

[Chris Karlof [:ckarlof]]

Something is blowing up on the storage server endpoints. The call to https://sync-2-us-east-1.stage.mozaws.net/1.5/2442748/info/collections doesn't return properly (as seen in the console) and I see this message in the console:

> [Exception... "<no message>"  nsresult: "0x805a1ff4 (<unknown>)"  location: "JS frame :: resource://services-sync/resource.js :: Channel_onStopRequest :: line 508"  data: no]

Comment 7

[Ryan Kelly [:rfkelly]]

Visitig https://sync-2-us-east-1.stage.mozaws.net/1.5/2442748/info/collections in a browser gives me an SSL error about "Peer's Certificate has been revoked." so that's probably what's causing these failures.

Comment 8

[Ryan Kelly [:rfkelly]]

SSL issues have been resolved and I can now sync successfully, closing this out.

Comment 9

[James Bonacci [:jbonacci]]

Yes. Thanks. Works for me as well.
I went down a rabbit hole rather than trying some simple tests to show the cert issue.
Sorry about that.