1153324 - Support Treeherder posting to Elasticsearch from Heroku

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P2)

Description

[Ed Morley [:emorley]]

Broken out of bug 1145606:

(In reply to Ed Morley [:emorley] (formerly :edmorley) from comment #1)
> Just occurred to me: Before moving to Heroku (or plain AWS for that matter),
> we'd also need to either (a) rewrite OrangeFactor to use Treeherder's API
> directly, or (b) find a way for the submit-star-comment task to be able to
> POST to ElasticSearch (eg an additional VPN tunnel or shim stood up
> somewhere).

(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> I've been told that IT can stand up a netflow as needed as long as the AWS
> host has a static IP.
> 
> Another option would be to stand up a proxy on brasstacks for the requests.
> 
> We do want to rewrite OF, but I suspect that will be a fair-sized project.

(In reply to Ed Morley [:emorley] (formerly :edmorley) from comment #4)
> I think the brasstacks proxy idea will be the easiest - since Heroku itself
> uses dynamic IPs for dynos [1] and whilst you can use the third party Heroku
> addon "Proximo" [2] to work around this, it seems overkill given how easy it
> would be to do on brasstacks.
> 
> [1] https://devcenter.heroku.com/articles/dynos#ip-addresses
> [2] https://devcenter.heroku.com/articles/proximo

(In reply to Kendall Libby [:fubar] from comment #5)
> If something in AWS needs to write to a local ES cluster, I would suggest a
> (local) proxy that does auth. Flow works as long as the IP doesn't change,
> and may be slow to have updated. ES doesn't do auth natively, but there are
> plugins/proxies that do. (and if it's generic enough, I will steal it for
> DXR. mwahaha.)

(In reply to Ed Morley [:emorley] (formerly :edmorley) from comment #6)
> I think we could do something as simple as adding another endpoint to the
> existing orangefactor API, except make this post to ES, so long as a
> suitable credential is passed at the same time. The OF API side can generate
> the ES request, so that all we expose to the API is the ability to add
> single doc with the specified properties (eg bug ID, start time, revision
> etc), rather than a direct way to abuse ES.

Comment 1

[Ed Morley [:emorley]]

If OrangeFactor v2 (which will use Treeherder's API directly and not ElasticSearch) is written before we move to Heroku, then this bug won't be required. (Though OrangeFactor v2 depends on most of the deps of bug 1179263, so a way out).

Comment 2

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:switch-to-orangefactor-api > mozilla:master

Comment 3

[Ed Morley [:emorley]]

Comment on attachment 8701896 [details] [review]
[treeherder] mozilla:switch-to-orangefactor-api > mozilla:master

The Treeherder part to follow bug 1235097 :-)

Comment 4

[William Lachance (:wlach)]

Comment on attachment 8701896 [details] [review]
[treeherder] mozilla:switch-to-orangefactor-api > mozilla:master

This part looks fine, but it probably doesn't make sense to r+ it while we're considering the other side...

Comment 5

[Ed Morley [:emorley]]

(This is waiting on bug 1235097, which is blocked on reviews.)

Comment 6

[Ed Morley [:emorley]]

Comment on attachment 8701896 [details] [review]
[treeherder] mozilla:switch-to-orangefactor-api > mozilla:master

The Treeherder counterpart for the new OrangeFactor API being added in bug 1235097, now updated to use Hawk auth instead.

Before this lands, bug 1235097 will need to land, and the hawk key will need to be set in the Treeherder environment (bug 1238584).

Comment 7

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/296cfd3bfcf1593eea9065112c57ade1029d2c8a
Bug 1153324 - Submit to OrangeFactor's API rather than ES directly

Since Elasticsearch is protected by VPN (ES doesn't do auth) and setting
up access is more hassle than it's worth on Heroku. Instead a new API
endpoint has been added to OrangeFactor that forwards our classification
report on for us.

A hardcoded API key is used for authentication, which will need to be
set in the environment of prod before this lands.

https://github.com/mozilla/treeherder/commit/47965837bf65a6f6deed7a288a3824d6735f4113
Bug 1153324 - Remove unused MIRROR_CLASSIFICATIONS setting

Since it's been replaced by the presence (or not) of the OrangeFactor
API password instead.

Comment 8

[Ed Morley [:emorley]]

This worked fine on Heroku (I temporarily added the credentials there, classified a job and then verified a new entry appears via the OrangeFactor UI):
https://rpm.newrelic.com/accounts/677903/applications/7385291/transactions?tw%5Bend%5D=1455275595&tw%5Bstart%5D=1455273795#id=5b224f746865725472616e73616374696f6e2f43656c6572792f7375626d69742d656c61737469637365617263682d646f63222c22225d