1134140 - Many "AttributeError: 'NoneType' object has no attribute 'open'" exceptions on stage

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Ed Morley [:emorley]]

See:
https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors?tw[end]=1424257973&tw[start]=1424236373

Seems like infra/DB issues, with DB disconnects & subsequent exceptions from trying to close DB connections that no longer existed.

Probably nothing to worry about wrt the deploy, but I want to check just in case.

Comment 1

[Ed Morley [:emorley]]

Lots of exceptions similar to:

Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/newrelic-2.44.0.36/newrelic/hooks/application_celery.py", line 66, in wrapper return wrapped(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__ return self.run(*args, **kwargs)
File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/etl/tasks/buildapi_tasks.py", line 55, in fetch_push_logs routing_key='pushlog'
File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/model/derived/refdata.py", line 122, in __exit__ self.disconnect()
File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/model/derived/refdata.py", line 116, in disconnect self.dhub.disconnect()
File "/usr/lib/python2.7/site-packages/datasource/bases/SQLHub.py", line 245, in disconnect if self.connection[host_type]['con_obj'].open:
AttributeError: 'NoneType' object has no attribute 'open'

This corresponds to this datasource line:

https://github.com/mozilla/treeherder-service/blob/52a653911e450fb5c5e9359438fd90a87c85b82a/vendor/datasource/bases/SQLHub.py#L245

            if self.connection[host_type]['con_obj'].open:
                self.connection[host_type]['con_obj'].commit()
                self.connection[host_type]['con_obj'].close()

It seems like datasource should be:
(a) Explicitly checking to see if self.connection[host_type]['con_obj'] is not None first, and raising an exception that's actually clear eg "Connections lost".
(b) Handling the exception in such a way that we don't stop trying to close the remaining connections for other host types.

Comment 2

[Ed Morley [:emorley]]

We also got a fair number of:

https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/3126432836

Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/newrelic-2.44.0.36/newrelic/hooks/application_celery.py", line 66, in wrapper return wrapped(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__ return self.run(*args, **kwargs)
File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/model/tasks.py", line 29, in process_objects jm.process_objects(limit)
File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/model/derived/jobs.py", line 2357, in process_objects self.load_job_data(rows)
File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/model/derived/jobs.py", line 1374, in load_job_data push_timestamps
File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/model/derived/jobs.py", line 1749, in _set_data_ids self.JOB_PH_MACHINE_NAME] = id_lookups['machines'][machine_name]['id']
KeyError: 'tst-linux64-spot-1588'

Comment 3

[Ed Morley [:emorley]]

This is causing cycle-data to not run, and thus a major contributor to bug 1134621 (though perhaps not the only factor).

Comment 4

[Ed Morley [:emorley]]

This didn't occur with Django 1.5, but does with 1.7. This is fixed in latest datasource, which we have in the repo thanks to bug 1125585, and the PR in bug 1134740 will make us start using that latest version from vendor/ rather than the globally installed one on stage/prod. 

This seemed like a quicker solution than trying to update the globally installed datasource - especially given that bug 1070470 wants to remove the globally installed packages that are listed in pure.txt (and should not have been installed to start with).

I'll also file a bug to figure out why the operationerror wasn't surfacing in new relic and only the connection close one.

Comment 5

[Ed Morley [:emorley]]

Sorry for the spam, messed up the mid-air correction.

Comment 6

[Ed Morley [:emorley]]

So it would help if I'd read the full error message in bug 1134621 comment 4 - whilst we do need the updated datasource package to avoid OperationErrors in the future (and I've seen them on non cycle-data transactions on stage already iirc), the root cause of the worst of them is bug 1134804, so I don't think this blocks the deploy.

Comment 7

[Ed Morley [:emorley]]

We've had no more of these in New Relic & cycle-data has successfully run (bug 1134621 comment 6).