1326094 - Enable review apps

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder, defect)

Description

[Armen [:armenzg]]

It would be great if we could have Heroku review apps enabled.
It would allow each PR to have their on version of Treeherder running.
This can be enabled on the Heroku pipeline.

I assume it could help with finding issues that are only encountered with production data (e.g. job notes that are *not* associated to a job).

Comment 1

[Ed Morley [:emorley]]

Currently the Treeherder Heroku apps are not self-contained, since they rely on an external AWS RDS MySQL instance for each environment. As such, each new review app will be missing it's DB.

We could experiment with using the third-party MySQL Heroku addon for review apps only (it's not that suitable for prod use due to price amongst other issues) via app.json - however that still won't help with the root reason for this request (prod parity), since the DB will start empty.

It sounds like the reason for this being filed is not discovering bug 1326090 until SETA was tested on stage. The following would help avoid that:
1) Freeing up the prototype Heroku instance and being happy to deploy people's partial PRs there more often (either once James is no longer using it, or saying that longer term feature work should use their own instance rather than taking long term control over it)
2) Bug 1303763 - Resolve differences between Vagrant/prod DB schemas
3) Bug 1296634 - Export Treeherder job snapshots to S3 (or similar bugs to help bootstrap a local dev instance from a subset of prod data)

Comment 2

[Armen [:armenzg]]

Thanks Ed. You're right on your assesment.

Would these bugs do for few good contributor bugs?

Comment 3

[Ed Morley [:emorley]]

(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #2)
> Would these bugs do for few good contributor bugs?

It depends on which bug you mean?
* The current bug (experimenting with adding an app.json and using a heroku addon for mysql instead of RDS, eg https://elements.heroku.com/addons/jawsdb) -> a contributor could try this
* changing process around the prototype instance (freeing it up) -> not contributor friendly
* Resolve differences between Vagrant/prod DB schemas -> not contributor friendly, needs DB access
* Export Treeherder job snapshots to S3 (or similar) -> possibly something a contributor could experiment with, but we'd need to decide what the best direction was first

Comment 4

[Ed Morley [:emorley]]

I think for now this is not worth the complexity, since:
* Treeherder relies on external RDS, so we'd have to substitute in a Heroku MySQL addon (not that hard, but still requires changes to the DB env setup etc)
* Treeherder doesn't yet ingest Pulse data out of the box (something we want to fix, but isn't yet), since Pulse doesn't support anonymous accounts. So a new review app would only ingest buildbot jobs, which are ever-decreasing.
* We already have the ability to point a local UI at a prod/stage API for testing against real data
* We already have the ability to build a custom UI and push to eg github pages (that uses stage/prod APR, to demo a new UI feature (instructions on read the docs)
* Due to the number of dynos/addons used by Treeherder, to reduce costs we'd probably want to set the review apps to manual creation - that is, they don't open for every PR, only the ones we select. ie a human will still have to get involved, so it won't save much time over just pushing a branch to the prototype instance, or the contributor using github pages etc.