785498 - errors when accessing http://graphs.allizom.org/server/collect.cgi from staging

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Kim Moir [:kmoir] ET]

talos-mtnlion-r5-004: 
		Stopped Fri, 24 Aug 2012 12:08:47
NOISE: Outputting talos results => {'results_urls': ['http://graphs.allizom.org/server/collect.cgi'], 'datazilla_urls': ['http://10.8.73.29/views/api/load_test']}
Generating results file: tsvgr: 
		Started Fri, 24 Aug 2012 12:08:47
Generating results file: tsvgr: 
		Stopped Fri, 24 Aug 2012 12:08:47
Generating results file: tsvgr_opacity: 
		Started Fri, 24 Aug 2012 12:08:47
Generating results file: tsvgr_opacity: 
		Stopped Fri, 24 Aug 2012 12:08:47
NOISE: Posting result 0 of 2 to http://graphs.allizom.org/server/collect.cgi, attempt 0
NOISE: Posting result 0 of 2 to http://graphs.allizom.org/server/collect.cgi, attempt 1
NOISE: Posting result 0 of 2 to http://graphs.allizom.org/server/collect.cgi, attempt 2
NOISE: Posting result 0 of 2 to http://graphs.allizom.org/server/collect.cgi, attempt 3
NOISE: Posting result 0 of 2 to http://graphs.allizom.org/server/collect.cgi, attempt 4
Generating results file: tsvgr: 
		Started Fri, 24 Aug 2012 12:11:22
Generating results file: tsvgr: 
		Stopped Fri, 24 Aug 2012 12:11:22
Generating results file: tsvgr_opacity: 
		Started Fri, 24 Aug 2012 12:11:22
Generating results file: tsvgr_opacity: 
		Stopped Fri, 24 Aug 2012 12:11:22


Is this the server that we should be connecting to for staging builds or should we be using another one?

Comment 1

[Robert Helmer [:rhelmer]]

That looks like the right URL, can you post the error(s) you're getting back from the server?

Thanks!

Comment 2

[Kim Moir [:kmoir] ET]

FAIL: Graph server unreachable (5 attempts)
RETURN:send failed, graph server says:
RETURN:(1045, "Access denied for user 'graphs_stage'@'10.22.70.211' (using password: YES)")

Comment 3

[Robert Helmer [:rhelmer]]

(In reply to Kim Moir [:kmoir] from comment #2)
> FAIL: Graph server unreachable (5 attempts)
> RETURN:send failed, graph server says:
> RETURN:(1045, "Access denied for user 'graphs_stage'@'10.22.70.211' (using
> password: YES)")

Hmm let me check that out, sounds like graphs_stage no longer has access to write (writes go to master, reads come from replication slaves so it's feasible that reads would be working while writes busted)

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Looks like a ZLB error.  I'm checking now.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

mysql> update mysql.user set host='10.22.70.2__' where user='graphs_stage';
Query OK, 1 row affected (0.02 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> flush privileges;
Query OK, 0 rows affected (0.06 sec)

----

That should straighten things out.  The source IP changed when zeus failed over from one load balancer to another.  There's a bug to fix this for good.

Reopen if that didn't fix?

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

would we need to get the os type in the database tables?

Comment 7

[Kim Moir [:kmoir] ET]

Thanks Dustin, am rerunning the tests now.

Comment 8

[Kim Moir [:kmoir] ET]

I'm still getting errors writing to the graphs server when I run my tests in staging.

TinderboxPrint: cycle time: 00:25:14<br>

TinderboxPrint:FAIL: Graph server unreachable (5 attempts)

TinderboxPrint:send failed, graph server says:

TinderboxPrint:(1044, "Access denied for user 'graphs_stage'@'10.22.70.2__' to database 'graphs_stage2_prod'")

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, let me dig a little deeper into the permissions here.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

OK, I was doing the ACLs wrong.  GRANT for the win.  I checked access from graphs1.stage and successfully logged in with the creds used by the webapp.

Comment 11

[Kim Moir [:kmoir] ET]

Thanks Dustin - it worked. Green builds!!!