971421 - ETL having problems starting every 1/2 hour

Status: RESOLVED WONTFIX

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[Kyle Lahnakoski [:ekyle]]

Every half hour, since the upgrade to 8gig, I am getting an error (below).  The logs should have more to say about what's happening.  Please send me some of those logs.


ERROR: Can not start
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 521, in start
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 527, in <module>

caused by
	ERROR: Problem with main ETL loop
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 412, in main
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 519, in start
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 527, in <module>

caused by
	ERROR: expecting commit() or rollback() before close
	at File "/data/www/Bugzilla-ETL/bzETL/util/db.py", line 127, in close
	at File "/data/www/Bugzilla-ETL/bzETL/util/db.py", line 109, in __exit__
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 400, in main
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 519, in start
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 527, in <module>

Comment 1

[Kendall Libby [:fubar] (he/him)]

Aha, it's losing connection to MySQL:

14:35:59 - WARNING: can not commit()
	at File "/data/www/Bugzilla-ETL/bzETL/util/db.py", line 107, in __exit__
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 400, in main
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 519, in start
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 527, in <module>

caused by
	WARNING: (2013, 'Lost connection to MySQL server during query')


I think the private one's gotten far enough behind that it's not able to finish or catch up (re-running by hand I see):
14:37:54 - Updating 11501 bugs:

I don't see any of those errors on public.

Comment 2

[Kyle Lahnakoski [:ekyle]]

Does it mention which query it's failing on?  I will make it take less time.

Comment 3

[Kendall Libby [:fubar] (he/him)]

It doesn't, unfortunately. Here's the block immediately before, maybe it'll help:

14:50:29 - Waiting on thread etl

14:50:31 - [Bug 972215]: Matched added flag {
    "modified_by": "[e-mail]",
    "modified_ts": 1392283760000,
    "request_status": "+",
    "request_type": "review",
    "value": "review+"
} to multiple removed flags [
    {
        "duration_days": 0.0,
        "modified_by": "[e-mail]",
        "modified_ts": 1392282785000,
        "previous_modified_ts": 1392276637000,
        "previous_status": "?",
        "previous_value": "review?([e-mail])",
        "request_status": "d",
        "request_type": "review",
        "requestee": "[e-mail]"
    },
    {
        "duration_days": 0.0,
        "modified_by": "[e-mail]",
        "modified_ts": 1392283760000,
        "previous_modified_by": "[e-mail]",
        "previous_modified_ts": 1392282833000,
        "previous_status": "?",
        "previous_value": "review?([e-mail])",
        "request_status": "+",
        "request_type": "review",
        "requestee": "[e-mail]",
        "value": "review+"
    }
].  Using the best.
14:50:31 - [Bug 972215]: Matching on modified_ts fixed it

14:50:33 - WARNING: can not commit()
        at File "/data/www/Bugzilla-ETL/bzETL/util/db.py", line 107, in __exit__
        at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 400, in main
        at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 519, in start

Comment 4

[Kyle Lahnakoski [:ekyle]]

How about instances of "Problem executing SQL:", those should be showing up.

Comment 5

[Kendall Libby [:fubar] (he/him)]

Sorry, none.

Comment 6

[Kyle Lahnakoski [:ekyle]]

This problem disappeared.  It may have been configuration on the server.