1140283 - https://login.anosrep.org/wsapi/session_context not responding/completing GET request

Status: VERIFIED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

The GET request for https://login.anosrep.org/wsapi/session_context never seems to finish (or, at least times out, in the end, but never completes).

STR:

1. Load http://beta.123done.org/
2. Wait for the "Sign in with your email" button

Actual:

It never appears -- the AJAX spinner remains

Expected:

Compare with http://123done.org/, which loads the button just fine

In talking with :jrgm on IRC:

10:31:00 PM 
<
jrgm> 
stephend: yeah, so that would be https://login.anosrep.org/wsapi/session_context not responding

This is breaking out BIDPOM tests on staging, which are now publicly available:

https://webqa-ci.mozilla.com/view/BIDPOM/

Comment 1

[John Morrison [:jrgm]]

Yeah, so, :relud, that would most likely mean a problem with the webhead being able to connect to the readonly databases. 

I could look into this, but not really sure I'd be looking at the right boxes. :relud, I should catch up with you tomorrow about this.

Comment 2

[John Morrison [:jrgm]]

I should add, the connection problem wouldn't be credentials related, but more a network layer problem. Maybe an SG change?

Comment 3

[Daniel Thorn [:relud]]

yeah, i'm trying to update deployments, and last nights didn't go well. this should be fixed some time today. thanks for the bug report!

Comment 4

[Daniel Thorn [:relud]]

got the new deployments working, and this appears to be fixed now.

Comment 5

[John Morrison [:jrgm]]

Yeah, so nothing got started on the webheads it appears. On the box that I *think* is the stage webhead, I tried to do `sudo systemctl start persona-browserid.service`, but that crashed with "can't open database: Error: Access denied for user 'persona-ro'@'%' to database 'persona'".

I dunno, maybe puppet didn't run completely?

Comment 6

[Daniel Thorn [:relud]]

yeah, puppet had some issues. I think they are all resolved now.

Comment 7

[John Morrison [:jrgm]]

Sorry, but still not running ok. The webheads are network-blocked from sending requests to the dbwriter (the sockets get stuck in SYN_SENT). Some sg is not enabled.

Comment 8

[Daniel Thorn [:relud]]

okay, pretty sure it's working this time. i can create account, verify account, login to account, logout of account, delete account, login with yahoo, and login with gmail.

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Daniel Thornton [:relud] from comment #8)
> okay, pretty sure it's working this time. i can create account, verify
> account, login to account, logout of account, delete account, login with
> yahoo, and login with gmail.

Thx, Daniel - it is indeed: https://webqa-ci.mozilla.com/view/BIDPOM/job/bidpom.stage.saucelabs/61/

Verified FIXED.