743728 - posting to graph server failing with "MySQL server has gone away"

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Mike Taylor [:bear]]

we are seeing this in production:

FAIL: Graph server unreachable (5 attempts)
RETURN:send failed, graph server says:
RETURN:(2006, 'MySQL server has gone away')
RETURN:  File "/var/www/html/graphs/server/pyfomatic/collect.py", line 271, in handleRequest
RETURN:    metadata = MetaDataFromTalos(databaseCursor, databaseModule, inputStream)
RETURN:  File "/var/www/html/graphs/server/pyfomatic/collect.py", line 63, in __init__
RETURN:    self.doDatabaseThings(databaseCursor)
RETURN:  File "/var/www/html/graphs/server/pyfomatic/collect.py", line 91, in doDatabaseThings
RETURN:    databaseCursor.connection.rollback()
RETURN:
RETURN:
Traceback (most recent call last):
  File "run_tests.py", line 681, in ?
    main()
  File "run_tests.py", line 678, in main
    test_file(arg, options, parser.parsed)
  File "run_tests.py", line 644, in test_file
    raise e
utils.talosError: 'Graph server unreachable (5 attempts)\nsend failed, graph server says:\n(2006, \'MySQL server has gone away\')\n  File "/var/www/html/graphs/server/pyfomatic/collect.py", line 271, in handleRequest\n    metadata = MetaDataFromTalos(databaseCursor, databaseModule, inputStream)\n  File "/var/www/html/graphs/server/pyfomatic/collect.py", line 63, in __init__\n    self.doDatabaseThings(databaseCursor)\n  File "/var/www/html/graphs/server/pyfomatic/collect.py", line 91, in doDatabaseThings\n    databaseCursor.connection.rollback()\n\n'
program finished with exit code 1
elapsedTime=1607.760000
TinderboxPrint:<a href = "http://hg.mozilla.org/integration/mozilla-inbound/rev/74670610b919">rev:74670610b919</a>

TinderboxPrint: cycle time: 00:24:09<br>
TinderboxPrint: tp5row
TinderboxPrint: tp5row_modlistbytes: 56.0MB
TinderboxPrint: tp5row_%cpu: 63.14
TinderboxPrint: tp5row_main_rss: 111.8MB
TinderboxPrint: tp5row_pbytes: 112.8MB
TinderboxPrint: tp5row_shutdown: 1259.0
TinderboxPrint: tp5row_responsiveness: 548.0


TinderboxPrint:FAIL: Graph server unreachable (5 attempts)
TinderboxPrint:send failed, graph server says:
TinderboxPrint:(2006, 'MySQL server has gone away')
TinderboxPrint:  File "/var/www/html/graphs/server/pyfomatic/collect.py", line 271, in handleRequest
TinderboxPrint:    metadata = MetaDataFromTalos(databaseCursor, databaseModule, inputStream)
TinderboxPrint:  File "/var/www/html/graphs/server/pyfomatic/collect.py", line 63, in __init__
TinderboxPrint:    self.doDatabaseThings(databaseCursor)
TinderboxPrint:  File "/var/www/html/graphs/server/pyfomatic/collect.py", line 91, in doDatabaseThings
TinderboxPrint:    databaseCursor.connection.rollback()

Comment 1

[Sheeri Cabral [:sheeri]]

mysql> select user,host,password from mysql.user where user='graphs';
+--------+-------------+-------------------------------------------+
| user   | host        | password                                  |
+--------+-------------+-------------------------------------------+
| graphs | 10.2.74.62  | *FFA33DBCA327E9EFAA1EB63C85E3556949196DFE |
| graphs | 10.2.74.110 | *5D0DA90E087C5AD5C96EB0826DCE63CDB9B91158 |
| graphs | 10.2.10.103 | *FFA33DBCA327E9EFAA1EB63C85E3556949196DFE |
+--------+-------------+-------------------------------------------+
3 rows in set (0.00 sec)

The 10.2.10.103 host is the new ACL, and that's set with the old password. So I fixed the password for the proper IP and dropped the old user.

Comment 2

[Mike Taylor [:bear]]

trees opened at 1120 PDT