1301378 - ATMO V2: http://atmo.dev.mozaws.net doesn't give me access to the web nodes after a deployment

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Mauro Doglio [:mdoglio]]

Ysterday we pushed some changes to the atmo repository and, as a consequence, the deployment process has been triggered. It looks like this process created 2 new ec2 instances, so the command

aws ec2 describe-instances --region us-west-2 --filters "Name=tag:App,Values=atmo" "Name=tag:Type,Values=app" --query "Reservations[*].Instances[*].[PrivateIpAddress]" --output text

now outputs

172.31.44.236
172.31.33.208
172.31.36.205
172.31.36.204

The first 2 nodes were apparently created on the previous deployment on august and I can ssh into them, while the last 2 nodes are new and I can't ssh into them. I believe the web requests to atmo are now load balanced between those 4 nodes, because I can see *some* of them are logged on the first 2 machines but not all of them.

Comment 1

[Mauro Doglio [:mdoglio]]

:robotblake do you know what's going on?

Comment 2

[Blake Imsland [:robotblake]]

:mdoglio it looks like it didn't create your user on the new nodes, I'm looking into it.  Currently we don't throw away the old nodes immediately though they are replaced in the ELB so hitting the URL will hit the correct machines.

Comment 3

[Mauro Doglio [:mdoglio]]

Although not ideal, the issue with the zombie nodes is not a big deal for me. The blocker for me is that I don't have access to the new nodes when they are created, so I don't have a way to check the logs. I'll update the bug description to reflect this.

Comment 4

[Blake Imsland [:robotblake]]

:mdoglio, your user should now exist on both of the new nodes and will exist on new nodes going forward.