Closed Bug 971421 Opened 8 years ago Closed 8 years ago

ETL having problems starting every 1/2 hour

Categories

(bugzilla.mozilla.org :: Infrastructure, defect)

x86_64
Windows 7
defect
Not set
normal

Tracking

()

RESOLVED WONTFIX

People

(Reporter: ekyle, Assigned: fubar)

References

Details

Every half hour, since the upgrade to 8gig, I am getting an error (below).  The logs should have more to say about what's happening.  Please send me some of those logs.


ERROR: Can not start
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 521, in start
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 527, in <module>

caused by
	ERROR: Problem with main ETL loop
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 412, in main
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 519, in start
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 527, in <module>

caused by
	ERROR: expecting commit() or rollback() before close
	at File "/data/www/Bugzilla-ETL/bzETL/util/db.py", line 127, in close
	at File "/data/www/Bugzilla-ETL/bzETL/util/db.py", line 109, in __exit__
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 400, in main
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 519, in start
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 527, in <module>
Aha, it's losing connection to MySQL:

14:35:59 - WARNING: can not commit()
	at File "/data/www/Bugzilla-ETL/bzETL/util/db.py", line 107, in __exit__
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 400, in main
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 519, in start
	at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 527, in <module>

caused by
	WARNING: (2013, 'Lost connection to MySQL server during query')


I think the private one's gotten far enough behind that it's not able to finish or catch up (re-running by hand I see):
14:37:54 - Updating 11501 bugs:

I don't see any of those errors on public.
Assignee: server-ops-webops → klibby
Does it mention which query it's failing on?  I will make it take less time.
Flags: needinfo?(klibby)
It doesn't, unfortunately. Here's the block immediately before, maybe it'll help:

14:50:29 - Waiting on thread etl

14:50:31 - [Bug 972215]: Matched added flag {
    "modified_by": "ejchen@mozilla.com",
    "modified_ts": 1392283760000,
    "request_status": "+",
    "request_type": "review",
    "value": "review+"
} to multiple removed flags [
    {
        "duration_days": 0.0,
        "modified_by": "evanxd@mozilla.com",
        "modified_ts": 1392282785000,
        "previous_modified_ts": 1392276637000,
        "previous_status": "?",
        "previous_value": "review?(yurenju.mozilla@gmail.com)",
        "request_status": "d",
        "request_type": "review",
        "requestee": "yurenju.mozilla@gmail.com"
    },
    {
        "duration_days": 0.0,
        "modified_by": "ejchen@mozilla.com",
        "modified_ts": 1392283760000,
        "previous_modified_by": "evanxd@mozilla.com",
        "previous_modified_ts": 1392282833000,
        "previous_status": "?",
        "previous_value": "review?(ejchen@mozilla.com)",
        "request_status": "+",
        "request_type": "review",
        "requestee": "ejchen@mozilla.com",
        "value": "review+"
    }
].  Using the best.
14:50:31 - [Bug 972215]: Matching on modified_ts fixed it

14:50:33 - WARNING: can not commit()
        at File "/data/www/Bugzilla-ETL/bzETL/util/db.py", line 107, in __exit__
        at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 400, in main
        at File "/data/www/Bugzilla-ETL/bzETL/bz_etl.py", line 519, in start
Flags: needinfo?(klibby)
How about instances of "Problem executing SQL:", those should be showing up.
Flags: needinfo?(klibby)
Sorry, none.
Flags: needinfo?(klibby)
This problem disappeared.  It may have been configuration on the server.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
Component: WebOps: Bugzilla → Infrastructure
Product: Infrastructure & Operations → bugzilla.mozilla.org
You need to log in before you can comment on or make changes to this bug.