1294425 - Productionify notification daemon

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Rail Aliiev [:rail]]

Filing this as a tracker for the following work:

1) move or copy the repo from https://github.com/cgsheeh/pulse-notify to https://github.com/mozilla-releng/

2) Setup https://dashboard.heroku.com/apps/release-notifications (mostly env variables), similar to https://dashboard.heroku.com/apps/release-notifications-dev/settings, but
  * Separate s3 bucket
  * separate pulse queue
  * separate irc user
  * send emails to [e-mail]

3) configure a separate branch (production) which should be automatically deployed by travis.

Probably something else that I missed...

We use heroku apps under the Mozilla account, which may require some initial setup (adding a user to the group, permissions, etc). I can help with this.

Comment 1

[Johan Lorenzo [:jlorenzo]]

I started to configure the Heroku project. Step 1 is done (project forked). A branch called "production" is ready there. Once the configuration on the Heroku side is done, I'll update .travis.yml to deploy automatically there.


On the way, I came across a few questions:
* May I have the rights to create a new S3 bucket? Or if it already exists, should I target "mozilla-release-logs"?
* Should we turn on influxdb? Or should we first get the bot started and then turn measures on?
* Should I register the IRC bot's email address to [e-mail] [1]? Or another address would be more suited?
* I saw the "pulse-notify" user is already registered[2]. May I get its password?
* I didn't see the list of existing queues at [2]. Would a queue named "release-notifications" work? If so, I'll create the queue.


[1] https://wiki.mozilla.org/IRC#Register_your_nickname
[2] https://pulseguardian.mozilla.org/all_pulse_users

Comment 2

[Rail Aliiev [:rail]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #1)
> I started to configure the Heroku project. Step 1 is done (project forked).
> A branch called "production" is ready there. Once the configuration on the
> Heroku side is done, I'll update .travis.yml to deploy automatically there.

\o/

> 
> On the way, I came across a few questions:
> * May I have the rights to create a new S3 bucket? Or if it already exists,
> should I target "mozilla-release-logs"?

Let's do it in IRC.

> * Should we turn on influxdb? Or should we first get the bot started and
> then turn measures on?

It would be great to enable it as well. ;)

> * Should I register the IRC bot's email address to
> [e-mail] [1]? Or another address would be
> more suited?

The former sounds good to me.

> * I saw the "pulse-notify" user is already registered[2]. May I get its
> password?

I'll double check if the password in the releng private repo. 

> * I didn't see the list of existing queues at [2]. Would a queue named
> "release-notifications" work? If so, I'll create the queue.

Yes, that queue name should work. BTW, it'll be created automatically when you start the daemon, there is no need to explicitly create it.

Comment 3

[Johan Lorenzo [:jlorenzo]]

Attachment: PR

Here's the PR that handles deployment. I noticed there are 2 ways to perform it: either on the Travis or on the Heroku side. I don't know what's the best practice here. At a first glance, I don't see any big differences. Heroku waits until CI build is green[1] by watching GH commits statuses[2]. That also makes one secret less to share. However, deployment is now less explicit as it used to be. This discrepancy can mislead newcomers.

Thus, Travis doesn't currently know anything about deployment, but I don't mind changing it. 


(In reply to Rail Aliiev [:rail] from comment #2)
> (In reply to Johan Lorenzo [:jlorenzo] from comment #1)
> > * May I have the rights to create a new S3 bucket? Or if it already exists,
> > should I target "mozilla-release-logs"?

For the record, a dedicated user has been created. It has all rights to access to "mozilla-release-logs".


> > * Should we turn on influxdb?
> 
> It would be great to enable it as well. ;)

Okay. I didn't find documentation regarding pre-configured InfluxDB for Heroku. Do you confirm it has to be installed like described here[3]


> > * I saw the "pulse-notify" user is already registered[2]. May I get its
> > password?
> 
> I'll double check if the password in the releng private repo. 

New passwords have also been added in the private repo.


[1] https://devcenter.heroku.com/articles/github-integration#automatic-deploys
[2] https://developer.github.com/v3/repos/statuses/
[3] https://docs.influxdata.com/influxdb/v0.13/introduction/installation/#hosting-on-aws

Comment 4

[Rail Aliiev [:rail]]

Comment on attachment 8782424 [details] [review]
PR

(In reply to Johan Lorenzo [:jlorenzo] from comment #3)
> Created attachment 8782424 [details] [review]
> PR
> 
> Here's the PR that handles deployment. I noticed there are 2 ways to perform
> it: either on the Travis or on the Heroku side. I don't know what's the best
> practice here. At a first glance, I don't see any big differences. Heroku
> waits until CI build is green[1] by watching GH commits statuses[2]. That
> also makes one secret less to share. However, deployment is now less
> explicit as it used to be. This discrepancy can mislead newcomers.
> 
> Thus, Travis doesn't currently know anything about deployment, but I don't
> mind changing it. 

I think your approach is good - less secrets, less headache.

> Okay. I didn't find documentation regarding pre-configured InfluxDB for
> Heroku. Do you confirm it has to be installed like described here[3]

We use 3-rd party hosting for this. I don't know the details, but we can figure out this on IRC.

Comment 5

[Johan Lorenzo [:jlorenzo]]

Attachment: PR part 2: Add static prod configuration

Comment 6

[Johan Lorenzo [:jlorenzo]]

Attachment: PR part 3: Send email to correct mailing list

Comment 7

[Johan Lorenzo [:jlorenzo]]

The daemon has run for 3 days without manual intervention. It sends emails to [e-mail] as if the sender was [e-mail]. S3 buckets are filled with TC logs. IRC notifications are also occurring in #release-notifications.

There were some small things left to configure:
* give the rights to send email to the AWS account
* add the static configuration (see PR part 2)
* Use [e-mail], instead of [e-mail] as recipient.

There are a few nits to fix, like changing the content of the email (bug 1296699), but they don't seem to be production-wise issues. I think we can close this bug, at the moment. If there's a problem still undetected, feel free to reopen it.