860100 - please deploy bigtent log_emails_issue178 to production

Status: RESOLVED FIXED

(Cloud Services :: Server: Identity, defect)

Description

[Lloyd Hilaiel [:lloyd]]

this is sha 2a48a7ea243085b242381858d8752a2fc74e4923

Goal is to put this in production for about a day so we can collect logs of emails that are failing to log into bigtent - both what the user typed and what yahoo reported.

This information will allow us to understand the root cause of the issue that's affecting 25% of logins with yahoo emails.  We'll be able to understand the average number of retries per user and whether this is related to yahoo aliases or yahoo plus, and we might discover that all of our explanations are wrong.

Comment 1

[John Morrison [:jrgm]]

Branch is log_emails_issue178?

And I think the ver in browserid.spec needs a bump.

Comment 2

[John Morrison [:jrgm]]

Okay, I had to increment the version for the the spec file to 16. 

So the sha for head of log_emails_issue178 is now 

57762b081cd438de3091f012eec0722da5b2db4a

Comment 3

[John Morrison [:jrgm]]

build command: ./build.sh browserid-bigtent log_emails_issue178 57762b081cd438de3091f012eec0722da5b2db4a

output rpm on r6 build machine: /home/jrgm/workspace/browserid-bigtent/rpmbuild/RPMS/x86_64/browserid-bigtent-0.2013.01.17-16.el6_115091.x86_64.rpm

Comment 4

[John Morrison [:jrgm]]

gene: Could you deploy this to AWS bigtent stage environment? (And later to production).

Comment 5

[Gene Wood [:gene]]

this has been deployed to stack 0409

https://github.com/mozilla/identity-ops/wiki/Interim-AWS-Stack-List#stack-0409--staging-bigtent-webhead-keysign-dbwrite-gene-builds-dbread

Comment 6

[Gene Wood [:gene]]

reopen when you're ready for prod

Comment 7

[John Morrison [:jrgm]]

I can see by logging into one of the bigtent servers in the 0409 stack that it is running the updated version:

> # curl -s http://127.0.0.1:3030/ver.txt
> 57762b0 Merge pull request #184 from jrgm/log_emails_issue178-version-bump
> locale svn r11509

But when I dig from home, or from other external locations, the CNAME returned is for the 0326 stack:

> $ dig +short yahoo.login.anosrep.org
> bt-login-anosrep-org-0326-1243940736.us-west-2.elb.amazonaws.com.
> 50.112.142.225
> 50.112.183.250
> 54.244.10.131

Comment 8

[Gene Wood [:gene]]

Doh, yes, I didn't switch it live, my fault. I've just updated dynect and this'll be fixed in the next 5 mins.

Comment 9

[John Morrison [:jrgm]]

Ok, the stage deploy of this rpm is good and I get the expected log message for no email match. We can use this in production.

Comment 10

[John Morrison [:jrgm]]

I've pointed my dns at the two regions for 0410, and checked that ver.txt is correct 57762b0, and have used both to successfully and (intentionally) get an id mismatch

https://github.com/mozilla/identity-ops/wiki/Interim-AWS-Stack-List#stack-0410--production-bigtent-w-temp-email-address-logging

We can take traffic for a while.

Comment 11

[John Morrison [:jrgm]]

This was deployed in production at '[15:50:53]	<gene>	jrgm : switched at 3:50pm take 3 mins to propogate'