1648581 - Invalid storage node being assigned in stage

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Operations: Sync, task)

Description

[Ryan Kelly [:rfkelly]]

Ref https://github.com/mozilla-mobile/firefox-ios/issues/6554#issuecomment-649694078

The Firefox iOS testsuite is failing while trying to do a sync, with the error:

2020-06-25 18:49:34.903 [Error] [StorageClient.swift:406] getFailureInfo(_:_:) > Response: 0. Got error Error Domain=NSURLErrorDomain Code=-1003 "A server with the specified hostname could not be found." UserInfo={NSUnderlyingError=0x600002ee0bd0 {Error Domain=kCFErrorDomainCFNetwork Code=-1003 "(null)" UserInfo={_kCFStreamErrorCodeKey=8, _kCFStreamErrorDomainKey=12}}, NSErrorFailingURLStringKey=https://sync-4-us-east-1.stage.mozaws.net/1.5/4969246/info/collections, NSErrorFailingURLKey=https://sync-4-us-east-1.stage.mozaws.net/1.5/4969246/info/collections, _kCFStreamErrorDomainKey=12, _kCFStreamErrorCodeKey=8, NSLocalizedDescription=A server with the specified hostname could not be found.}.

The host https://sync-4-us-east-1.stage.mozaws.net does indeed seem to be non-existent. Has it gone down, or been removed?

Comment 1

[Bob Micheletto [:bobm]]

(In reply to Ryan Kelly [:rfkelly] from comment #0)

The host https://sync-4-us-east-1.stage.mozaws.net does indeed seem to be non-existent. Has it gone down, or been removed?

It is down. The py-sync nodes exist currently to support further automation testing. All staging tests should be happening on durable Sync in GCP. We'll be decommissioning these staging nodes in the near future. Would it help if I just removed the record for sync-4 and re-assigned all the users there?

Comment 2

[Ryan Kelly [:rfkelly]]

Would it help if I just removed the record for sync-4 and re-assigned all the users there?

Yes please :-)

We worked around it on the client side by deleting and re-creating this test account, which had a similar effect. I think it would be worth re-assigning any other remaining users on the dead nodes to avoid future confusion. There is a "remove_node" script in the tokenserver package that IIRC is meant to help with this, including un-assigning any users currently assigned to that node.

Comment 3

[Bob Micheletto [:bobm]]

Closing this out. We don't have a py-sync environment in stage anymore.