Closed Bug 1125856 Opened 9 years ago Closed 9 years ago

Use the master db to perform queries on data ingestion

Categories

(Tree Management :: Treeherder: Data Ingestion, defect, P1)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mdoglio, Assigned: mdoglio)

References

Details

Attachments

(1 file)

Now that bug 1125464 has landed, some new errors appear in newrelic. 
See https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/2973275709?original_error_id=2973401858 for an example.
We should always use the master db when ingesting data to prevent errors due to delays in replication.
Assignee: nobody → mdoglio
There's also:
https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/2973462084/similar_errors?original_error_id=2973462084

treeherder.etl.mixins:CollectionNotLoadedException

Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/newrelic-2.40.0.34-py2.6-linux-x86_64.egg/newrelic/hooks/application_celery.py", line 66, in wrapper return wrapped(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__ return self.run(*args, **kwargs)
File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/etl/tasks/buildapi_tasks.py", line 32, in fetch_buildapi_running RunningJobsProcess().run()
File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/etl/buildapi.py", line 576, in run filter_to_project=filter_to_project)
File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/etl/mixins.py", line 167, in load raise CollectionNotLoadedException(errors)
CollectionNotLoadedException: [mozilla-beta] Error posting data to jobs: {"detail": "u'2e4c23ca12cc67d7fdfcb826606da696f5d6eda6'"} [mozilla-inbound] Error posting data to jobs: {"detail": "u'c9d1a42d6a557343c8edfb3cd2b8a5f05232a8cb'"}
Blocks: 1125464
Priority: -- → P1
Attachment #8554659 - Flags: review?(emorley)
:edmorley the error you posted seems to be the etl process counter part of the error I linked from newrelic.
Comment on attachment 8554659 [details] [review]
Github PR #351 on treeherder-service

These changes look good; though I haven't yet looked to see if we need any more than this. But let's land this on stage and see how the errors look :-)
Attachment #8554659 - Flags: review?(emorley) → review+
The etl nodes on both staging and production are missing new flows to the readonly db.
Depends on: 1126226
No longer depends on: 1126226
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: