1145606 - (treeherder-heroku-prototype) Set up a prototype Treeherder heroku instance

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P2)

Description

[Ed Morley [:emorley]]

Given our discussion at the last meeting, I'm filing this as a placeholder for us looking into the viability of Heroku for Treeherder.

Comment 1

[Ed Morley [:emorley]]

Just occurred to me: Before moving to Heroku (or plain AWS for that matter), we'd also need to either (a) rewrite OrangeFactor to use Treeherder's API directly, or (b) find a way for the submit-star-comment task to be able to POST to ElasticSearch (eg an additional VPN tunnel or shim stood up somewhere).

On that note, I'm presuming publishing to pulse doesn't require any specific flows?

Comment 2

[Mauro Doglio [:mdoglio]]

yeah that's correct, using pulse doesn't require any special flow

Comment 3

[Jonathan Griffin (:jgriffin)]

I've been told that IT can stand up a netflow as needed as long as the AWS host has a static IP.

Another option would be to stand up a proxy on brasstacks for the requests.

We do want to rewrite OF, but I suspect that will be a fair-sized project.

Comment 4

[Ed Morley [:emorley]]

I think the brasstacks proxy idea will be the easiest - since Heroku itself uses dynamic IPs for dynos [1] and whilst you can use the third party Heroku addon "Proximo" [2] to work around this, it seems overkill given how easy it would be to do on brasstacks.

[1] https://devcenter.heroku.com/articles/dynos#ip-addresses
[2] https://devcenter.heroku.com/articles/proximo

Comment 5

[Kendall Libby [:fubar] (he/him)]

If something in AWS needs to write to a local ES cluster, I would suggest a (local) proxy that does auth. Flow works as long as the IP doesn't change, and may be slow to have updated. ES doesn't do auth natively, but there are plugins/proxies that do. (and if it's generic enough, I will steal it for DXR. mwahaha.)

Comment 6

[Ed Morley [:emorley]]

I think we could do something as simple as adding another endpoint to the existing orangefactor API, except make this post to ES, so long as a suitable credential is passed at the same time. The OF API side can generate the ES request, so that all we expose to the API is the ability to add single doc with the specified properties (eg bug ID, start time, revision etc), rather than a direct way to abuse ES.

Comment 7

[Ed Morley [:emorley]]

Filed bug 1153324 for this.

Comment 8

[Ed Morley [:emorley]]

->P2 since standing up a prototype is in the Q2 goals list.

Comment 9

[Mauro Doglio [:mdoglio]]

Attachment: Heroku deployment setup

Comment 10

[Ed Morley [:emorley]]

I've run out of time today, I'll take a look tomorrow :-)

Comment 11

[William Lachance (:wlach)]

Comment on attachment 8604074 [details] [review]
Heroku deployment setup

I don't think I have anything more to add over the other reviews. I know little about Heroku so can't say for sure whether this way is the best way of doing things -- the use of environment variables makes me a bit uneasy, but maybe that's just the way it is.

Anyway, very exciting! Thanks for taking it on.

Comment 12

[Mauro Doglio [:mdoglio]]

:wlach the heroku architecture is influenced by [the 12factor app methodology](http://12factor.net), which includes storing the app configuration in the environment.

Comment 13

[Ed Morley [:emorley]]

I've linked heroku-treeherder to the treeherder github repo (but not enabled auto-deploy on push to master, since this PR hasn't landed) - but diffs and things now link up to github :-)

Comment 14

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/597282fe5825d12f736d808c570e40b7a69e5c20
Bug 1145606 - Setup treeherder to deploy on heroku

I added a Procfile listing all the different python services treeherder needs.
Heroku provides deployment-specific settings via environment variables, so I had to modify the settings file to listen to them where that wasn't the case. I created an enviroment variable IS_HEROKU which allows to have a heroku-only configuration where needed.
The db service is provided by Amazon RDS, which requires a ssl connection. To enable ssl in the MySQLdb python client I had to modify Datasource (and bump up the version used).
The cache service is provided by the memcachier heroku addon. Heroku recommends to use pylibmc, so I set it up according to the docs here https://devcenter.heroku.com/articles/memcachier#python.
The amqp service is provided by the CloudAMQP addon.
I added a post_compile script that runs every time we deploy. We should run every build step we require in there, like static asset minification, collection, etc.
To share the oauth credentials among the various services I used an environment variable. I also added an option to export_project_credentials so that the credentials can be printed to stdout. This should come handy when we will need to update the environment-stored credentials with the ones in the db.

Comment 15

[Mauro Doglio [:mdoglio]]

:edmorley kindly enabled automatic deployments from master now that the heroku branch has been merged.

Comment 16

[Mauro Doglio [:mdoglio]]

I set up treeherder-heroku to use the mozilla account on papertrail.
See https://papertrailapp.com/systems/treeherder/events

Comment 17

[Ed Morley [:emorley]]

Some remaining tasks from the discussion at the meeting, for moving to Heroku for real:

* Migrating database
 ** Checking the current prototype DB is fine to use - or do we need to create a new RDS instance on someone else's AWS account?
* Publishing to pulse?
* Figuring out stage vs prod
* Switching over the domains + SSL certs etc
* Coming up with a transition plan

Comment 18

[Ed Morley [:emorley]]

We now have a Treeherder prototype running on https://treeherder-heroku.herokuapp.com/ so this bug is complete.

I'll file a new one for tracking the remaining tasks for moving prod/stage to Heroku for real.