620572 - Graph server failing to connect to its database

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Phil Ringnalda (:philor)]

From http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1292890769.1292891497.14098.gz

FAIL: Failed to send data 5 times... quitting
RETURN:send failed, graph server says:
RETURN:(2003, "Can't connect to MySQL server on '10.2.70.130' (110)")

So far it has only taken out two Talos runs, so it could have just been a couple-minute hiccup, but if it keeps on, I'll have to close the tree, thus the severity.

Comment 1

[matthew zeier [:mrz]]

10.2.70.130 is a stage database.  Need to understand why production is trying to access a stage database.

Comment 2

[alice nodelman [:alice] [:anode]]

10.2.70.130 is pamo, this is the first step in getting amo performance test results inserted into a given amo db.  I'm working with justdave currently on getting this working.

Comment 3

[Phil Ringnalda (:philor)]

Another four, and we're closed.

Comment 4

[alice nodelman [:alice] [:anode]]

Done as work on bug 620015.

Comment 5

[Ravi Pina [:ravi]]

Can someone explain the relationship of the tree to graphs and why this outage requires the tree to close?

Comment 6

[Shyam Mani [:fox2mike]]

(In reply to comment #2)
> 10.2.70.130 is pamo, this is the first step in getting amo performance test
> results inserted into a given amo db.  I'm working with justdave currently on
> getting this working.

10.2.70.130 is the stage DB. Production should *never* be connecting to the stage DB. Why isn't this being tested with graphs-stage?

Comment 7

[alice nodelman [:alice] [:anode]]

It has been tested on graphs-stage.  We were attempting a stepped move to production.  Roll out production graph server code reporting to pamo, check that new graph server code works along with various releng updates, switch from pamo to production amo.

Comment 8

[Phil Ringnalda (:philor)]

(In reply to comment #5)
> Can someone explain the relationship of the tree to graphs and why this outage
> requires the tree to close?

Talos tests post their results to graph server, so if it refuses to accept them, the test run turns red, rightfully, because we never really look at the individual numbers in the build log, only averages and graphs over time.

Comment 9

[Dave Miller [:justdave]]

I was told removing the "amodb" config value from the config file would make it skip the portion of code that's erroring, so that's been removed until we straighten this out.

Comment 10

[alice nodelman [:alice] [:anode]]

This just completed green:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1292893090.1292893304.22364.gz

Comment 11

[matthew zeier [:mrz]]

(In reply to comment #7)
> It has been tested on graphs-stage.  We were attempting a stepped move to
> production.  Roll out production graph server code reporting to pamo, check
> that new graph server code works along with various releng updates, switch from
> pamo to production amo.

This dependency was missed from the deployment docs.

Comment 12

[alice nodelman [:alice] [:anode]]

I'm seeing more green runs, I believe that you are clear to re-open.

Comment 13

[Ravi Pina [:ravi]]

(In reply to comment #8)

Wouldn't it be better (if it is possible) to queue the results if it cannot connect to the DB?  At face value it seems like the right thing to do to prevent something like this since otherwise all other functions are otherwise working, right?

Comment 14

[Phil Ringnalda (:philor)]

Reopened.

Comment 15

[Shyam Mani [:fox2mike]]

Closing this out since the immediate issue is fixed.

(In reply to comment #13)
> (In reply to comment #8)
> 
> Wouldn't it be better (if it is possible) to queue the results if it cannot
> connect to the DB?  At face value it seems like the right thing to do to
> prevent something like this since otherwise all other functions are otherwise
> working, right?

I opened Bug 620596 for this.