1211414 - [Meta] Interact directly with the DB instead of making HTTP requests to our own API

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P3)

Description

[Ed Morley [:emorley]]

Whilst it sounded all nice from a "we can have our ETL nodes running on a completely separate system" point of view, having the ETL tasks hit the external API rather than interacting with the DB directly adds additional load and means heavy-load issues with the API cause failure in our own ingestion, and not just external submitters.

IMO we should make as many of our internal tasks interact with the DB directly and not have the additional round-trip of hitting our webheads to go via the API.

We'll obviously need to make sure we still have enough tests for eg the python-client, since we'll not be using it internally, but IMO it will be worth it.

Note also that the idea of "let's have another webhead just for internal requests" won't work on Heroku, where you cannot differentiate between webheads like that (and worker nodes cannot open web-facing ports).

Comment 1

[Ed Morley [:emorley]]

Tweaking summary so this is easier to find (just took ages).

Comment 2

[Ed Morley [:emorley]]

Bug 1273231 made the log parser interact directly with the DB when updating the parse status of logs, which was a big win from a "requests Treeherder makes to itself per minute" point of view.

Another massive contributor is the log parser querying the bugscache for bug suggestions, which is currently the #3 most frequent type of API request (at 120-200 requests per minute):
https://rpm.newrelic.com/accounts/677903/applications/4180461/transactions#id=5b225765625472616e73616374696f6e2f46756e6374696f6e2f747265656865726465722e7765626170702e6170692e726566646174613a427567736361636865566965775365742e6c697374222c22225d&sort_by=throughput

Comment 3

[William Lachance (:wlach)]

I believe this is the only reason we have the (IMO rather silly) bugscache API, so we can probably go ahead and kill it after this is done. I would just convert the searching logic there into a simple function called inside `treeherder/model/error_summary.py`.

Comment 4

[Ed Morley [:emorley]]

bc uses it I believe, though there's likely in underlying better way to solve the use case

Comment 5

[Ed Morley [:emorley]]

Though looks unused now, perhaps after Cameron added the ability to submit your own text log summary json blob, but still have bug suggestions generated. 

https://github.com/mozilla/autophone/search?utf8=%E2%9C%93&q=bugscache_uri

Comment 7

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #5)
> Though looks unused now, perhaps after Cameron added the ability to submit
> your own text log summary json blob, but still have bug suggestions
> generated. 
> 
> https://github.com/mozilla/autophone/search?utf8=%E2%9C%93&q=bugscache_uri

Filed bug 1281031 & opened a PR to remove the unused variable.

Checking the prod weblogs shows no other consumers apart from Treeherder itself, so once bug 1280910 lands we can remove the bugscache API.

Comment 8

[Ed Morley [:emorley]]

We're no longer hitting our own API \o/ :-)