errors when accessing http://graphs.allizom.org/server/collect.cgi from staging

RESOLVED FIXED

Status

Data & BI Services Team
DB: MySQL
RESOLVED FIXED
5 years ago
3 years ago

People

(Reporter: kmoir, Assigned: dustin)

Tracking

Details

(Reporter)

Description

5 years ago
talos-mtnlion-r5-004: 
		Stopped Fri, 24 Aug 2012 12:08:47
NOISE: Outputting talos results => {'results_urls': ['http://graphs.allizom.org/server/collect.cgi'], 'datazilla_urls': ['http://10.8.73.29/views/api/load_test']}
Generating results file: tsvgr: 
		Started Fri, 24 Aug 2012 12:08:47
Generating results file: tsvgr: 
		Stopped Fri, 24 Aug 2012 12:08:47
Generating results file: tsvgr_opacity: 
		Started Fri, 24 Aug 2012 12:08:47
Generating results file: tsvgr_opacity: 
		Stopped Fri, 24 Aug 2012 12:08:47
NOISE: Posting result 0 of 2 to http://graphs.allizom.org/server/collect.cgi, attempt 0
NOISE: Posting result 0 of 2 to http://graphs.allizom.org/server/collect.cgi, attempt 1
NOISE: Posting result 0 of 2 to http://graphs.allizom.org/server/collect.cgi, attempt 2
NOISE: Posting result 0 of 2 to http://graphs.allizom.org/server/collect.cgi, attempt 3
NOISE: Posting result 0 of 2 to http://graphs.allizom.org/server/collect.cgi, attempt 4
Generating results file: tsvgr: 
		Started Fri, 24 Aug 2012 12:11:22
Generating results file: tsvgr: 
		Stopped Fri, 24 Aug 2012 12:11:22
Generating results file: tsvgr_opacity: 
		Started Fri, 24 Aug 2012 12:11:22
Generating results file: tsvgr_opacity: 
		Stopped Fri, 24 Aug 2012 12:11:22


Is this the server that we should be connecting to for staging builds or should we be using another one?
That looks like the right URL, can you post the error(s) you're getting back from the server?

Thanks!
(Reporter)

Comment 2

5 years ago
FAIL: Graph server unreachable (5 attempts)
RETURN:send failed, graph server says:
RETURN:(1045, "Access denied for user 'graphs_stage'@'10.22.70.211' (using password: YES)")
(In reply to Kim Moir [:kmoir] from comment #2)
> FAIL: Graph server unreachable (5 attempts)
> RETURN:send failed, graph server says:
> RETURN:(1045, "Access denied for user 'graphs_stage'@'10.22.70.211' (using
> password: YES)")

Hmm let me check that out, sounds like graphs_stage no longer has access to write (writes go to master, reads come from replication slaves so it's feasible that reads would be working while writes busted)
Assignee: server-ops-database → rhelmer
Component: Server Operations: Database → Graph Server
Product: mozilla.org → Webtools
QA Contact: cshields
(Reporter)

Updated

5 years ago
Blocks: 731278
(Assignee)

Comment 4

5 years ago
Looks like a ZLB error.  I'm checking now.
(Assignee)

Comment 5

5 years ago
mysql> update mysql.user set host='10.22.70.2__' where user='graphs_stage';
Query OK, 1 row affected (0.02 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> flush privileges;
Query OK, 0 rows affected (0.06 sec)

----

That should straighten things out.  The source IP changed when zeus failed over from one load balancer to another.  There's a bug to fix this for good.

Reopen if that didn't fix?
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Assignee)

Updated

5 years ago
Component: Graph Server → Server Operations: Database
Product: Webtools → mozilla.org
QA Contact: cshields
would we need to get the os type in the database tables?
(Reporter)

Comment 7

5 years ago
Thanks Dustin, am rerunning the tests now.
(Reporter)

Comment 8

5 years ago
I'm still getting errors writing to the graphs server when I run my tests in staging.

TinderboxPrint: cycle time: 00:25:14<br>

TinderboxPrint:FAIL: Graph server unreachable (5 attempts)

TinderboxPrint:send failed, graph server says:

TinderboxPrint:(1044, "Access denied for user 'graphs_stage'@'10.22.70.2__' to database 'graphs_stage2_prod'")
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 9

5 years ago
Ah, let me dig a little deeper into the permissions here.
Assignee: rhelmer → dustin
Status: REOPENED → NEW
(Assignee)

Comment 10

5 years ago
OK, I was doing the ACLs wrong.  GRANT for the win.  I checked access from graphs1.stage and successfully logged in with the creds used by the webapp.
Status: NEW → RESOLVED
Last Resolved: 5 years ago5 years ago
Resolution: --- → FIXED
(Reporter)

Comment 11

5 years ago
Thanks Dustin - it worked. Green builds!!!
Product: mozilla.org → Data & BI Services Team
You need to log in before you can comment on or make changes to this bug.