1467026 - move travis-ci irc bot from #releng and relengbot from #buildduty to a newly created #releng-bots

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Comment 1

[Chris Cooper [:coop] (he/him)]

I looked at the deploy process and there are a couple of things wrong.

First, the pipeline that was setup by Selena originally was no longer connected. Not sure whether her something in her config changed (LDAP disabled during parental leave?). This was preventing any new deployments from happening, so I reconnected the pipeline using my own creds, but someone in releng should really own this.

Second, I updated the HUBOT_IRC_ROOMS config var to #releng-bots.

The app builds OK, but is crashing on startup with the following error:

10 Jun 2018 12:14:44.250106 <45>1 2018-06-10T16:14:43.000000+00:00 app api - - Build succeeded
10 Jun 2018 12:14:46.918144 <45>1 2018-06-10T16:14:46.680791+00:00 heroku web.1 - - Starting process with command `bin/hubot -a irc`
10 Jun 2018 12:14:53.470124 <45>1 2018-06-10T16:14:53.140894+00:00 heroku web.1 - - Process exited with status 1
10 Jun 2018 12:14:53.47094 <190>1 2018-06-10T16:14:53.059416+00:00 app web.1 - -
10 Jun 2018 12:14:53.529134 <45>1 2018-06-10T16:14:53.161301+00:00 heroku web.1 - - State changed from starting to crashed
10 Jun 2018 12:14:53.541182 <190>1 2018-06-10T16:14:53.059609+00:00 app web.1 - - /app/node_modules/hubot-irc/node_modules/irc/node_modules/irc-colors/lib/irc-colors.js:3
10 Jun 2018 12:14:53.541110 <190>1 2018-06-10T16:14:53.059997+00:00 app web.1 - - const colors = {
10 Jun 2018 12:14:53.54199 <190>1 2018-06-10T16:14:53.060077+00:00 app web.1 - - ^^^^^
10 Jun 2018 12:14:53.541209 <190>1 2018-06-10T16:14:53.063501+00:00 app web.1 - - [Sun Jun 10 2018 16:14:53 GMT+0000 (UTC)] ERROR Cannot load adapter irc - SyntaxError: Use of const in strict mode.

I tried updating the heroku stack from 14 to 16, but the app still crashes the same way.

To me, it seems like we might need to update a bunch of dependencies here to get things working, i.e. I was hoping to simply update the config var and be done, but it's not that simple.

Jordan: can you find someone in releng of CIDuty to own this?

Comment 2

[Chris Cooper [:coop] (he/him)]

I got it working by specifying the node version to LTS (8.11.2): https://github.com/mozilla/relengbot/commit/4c43b31fe24056cea1ccb6b1eb882990c0a1abab

So the bot is working again, but if CIDuty is going to use it, someone should assume real ownership, someone who isn't me ;)

Comment 3

[Zsolt Fay [:zfay]]

As far as I'm aware we don't have access to heroku. Would it be a good thing to get a ciduty account for it or after moving all the necessary bots there wouldn't be any use for it?

Comment 4

[Chris Cooper [:coop] (he/him)]

(In reply to Zsolt Fay [:zsoltfay] from comment #3)
> As far as I'm aware we don't have access to heroku. Would it be a good thing
> to get a ciduty account for it or after moving all the necessary bots there
> wouldn't be any use for it?

I think the better question is whether the limited services provided by relengbot are worth preserving. If you're planning on extending it, then sure, you should have the access required to affect deployments.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

We're getting a few emails a day with subject "ALERT relengbot failure, needs restart: Heroku/relengbot [relengbot]", body:

relengbot failure, needs restart: Heroku/relengbot

2018-06-22 00:01:18.808
237 <190>1 2018-06-22T00:01:18.430844+00:00 app web.1 - - [Fri Jun 22 2018 00:01:18 GMT+0000 (UTC)] ERROR ERROR: err_cannotsendtochan: relengbot #buildduty Cannot send to channel (no external messages)

Go to the log: https://api.heroku.com/myapps/[e-mail]/addons/logentries

You are receiving this email because your Logentries alarm "relengbot failure, needs restart"
has been triggered.
----

Perhaps the mode changed on the channel as part of the deprecation, and the bot needs to stop trying to use it ?

Comment 6

[Jordan Lund (:jlund)]

looking into this. need heroku access for that app.

Comment 7

[Jordan Lund (:jlund)]

moved all travis-ci notifying bots to dedicated #releng-bots

* hg build/tools
* hg build/buildbot-configs
* hg build/buildbotcustom
* hg build/mozharness
* git mozilla-releng/build-puppet
* git mozilla/tc-coalesce

Comment 8

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #6)
> looking into this. need heroku access for that app.

coop got me access. Looking at config vars and recent deployments, HUBOT_IRC_ROOMS is setup correctly with #releng-bots. Not sure where it's getting #buildduty from. Since unplugging and plugging things back in usually works, I restarted all the dynos with the production app.

Since we are hitting this error nearly daily, we should know if that helps.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Seem to still be getting mail like comment #5.

Comment 10

[Jordan Lund (:jlund)]

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #9)
> Seem to still be getting mail like comment #5.

I stopped the web app, signed into irc manually as relengbot. Manually registered with nickserve w/ existing pw, added env HUBOT_IRC_DEBUG:true to heroku app, and restarted the web app.

I haz no idea where relengbot app is getting "#buildduty" from but hoping either the manual on/off stuff worked or debug will reveal more info.

Comment 11

[Jordan Lund (:jlund)]

okay debug showed some more context. these were coming from sns

 2018-07-11 23:43:47.119 154 <190>1 2018-07-11T23:43:46.763849+00:00 app web.1 - - 11 Jul 23:43:46 - SEND: PRIVMSG #buildduty :[sns alert] test

I guess the channel is inferred from the sns topic? I deleted: the usw2 and use1 buildduty topics like: https://console.aws.amazon.com/sns/v2/home?region=us-east-1#/topics/arn:aws:sns:us-east-1:314336048151:buildduty

and created new ones in each. e.g. https://us-west-2.console.aws.amazon.com/sns/v2/home?region=us-west-2#/topics/arn:aws:sns:us-west-2:314336048151:releng-bots

I tested a post and it seems like it's working:

# from #releng-bots
relengbot> [sns alert] test
16:46:05 
<•jlund> Jordan Lund wee

I think this should resolve the issue. We will see.