1176484 - (treeherder-heroku) Move treeherder stage/prod to Heroku

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P2)

Description

[Ed Morley [:emorley]]

In bug 1145606, a prototype Heroku instance was set up:
https://treeherder-heroku.herokuapp.com/

This meta bug is for tracking:
* the remaining tasks so that we have feature parity with stage/production, and are fully prepared for the move
* deciding the plan/timing of transition
* performing the migration of data from production & flicking the switch

Comment 1

[Ed Morley [:emorley]]

I asked James to delete the even older prototype he set up under his user account, to avoid confusion over the apps (it was still appearing in my account too), which he's now done (in case anyone is wondering where that app has gone from their dashboard).

Comment 2

[Ed Morley [:emorley]]

I discovered a bug with the `heroku releases` CLI feature, which lists my email address for all GitHub auto-deployed commits. Filed https://github.com/heroku/heroku/issues/1722 but was asked to open a support ticket, which I've done::
https://help.heroku.com/tickets/278051

Comment 3

[Ed Morley [:emorley]]

Attachment: export-heroku-app-info.sh

Script to export all metadata about a Heroku app.

Use like:
./export-heroku-app-info treeherder-stage

A file of form "herokuinfo-treeherder-stage.txt" will be created, allowing for easy diffing of configuration against other Heroku apps.

NB: Make sure to login to the Heroku client first, and that you have run `heroku update` recently.

Comment 4

[Ed Morley [:emorley]]

Attachment: export-envs.sh

Script to dump all SCL3 and Heroku environment variables, for easy diffing between environments. Files of form `env-{scl3,heroku}-{stage,prod}.sh` will be created.

Comment 5

[Ed Morley [:emorley]]

For some reason there have been increasingly more rabbitmq queue size warnings on heroku-stage over the last 3 weeks.

I've struggled to find out why (perhaps just increasing number of jobs), so have just bumped the number of log parser workers.

I've also increased the default worker count, since the cycle_data + fetch_bugs tasks are now taking long enough that they cause a backlog of detect_intermittent tasks (they are run on the same dyno, we should probably break them out).

Comment 6

[James Graham [:jgraham]]

We could also just turn detect_intermittents off. No-one seems to love it.

Comment 7

[Ed Morley [:emorley]]

(In reply to James Graham [:jgraham] from comment #6)
> We could also just turn detect_intermittents off. No-one seems to love it.

Ty, i'll file a bug for that.

--

Unrelatedly, deploys were failing on stage due to bug 1264074 comment 16. I've fixed up the environment variable for now, but have also filed:
https://help.heroku.com/tickets/394841

Comment 8

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #5)
> I've struggled to find out why (perhaps just increasing number of jobs), so
> have just bumped the number of log parser workers.

Another warning today - bumped up to 9 log parser workers.

I suspect bug 1286935 playing a roll too.