1293544 - alertmanager.allizom.org is returning 500 errors and needs to get back to a stable changeset

Status: VERIFIED FIXED

(Testing :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

eg http://alertmanager.allizom.org/data/setadetails/?date=2016-08-09&buildbot=1&branch=fx-team&inactive=1

We first noticed at Aug 8 1400 PDT, seems to be pretty consistent. Caused a tree closure because buildbot didn't cope very well.

Comment 1

[Dan Minor [:dminor]]

I no longer have access to that machine (and never worked on alertmanager :) )

jmaher is the primary contact, but he is on PTO. Adding some other people who might be able to help.

Comment 2

[Kim Moir [:kmoir] ET]

last time jgriffin was able to login and fix it, I don't have credentials

Comment 3

[Kim Moir [:kmoir] ET]

Also, jgriffin would it be possible to share the credentials with releng so we can look at it when ateam members are unavailable?

Comment 4

[Armen [:armenzg]]

It seems that a security change was deployed to alertmanager in the last day or so:
https://github.com/mozilla/ouija/commit/8ab444e1c80e603758759c1c71417b0a53ede131

Unfortunately, the change came in with other development changes that are meant for the Heroku/TaskCluster work not to be deployed to alertmanager.

glob is currently working on getting us back into a good state.

So far, he installed the sqlalchemy python package which was missing for the latest changes.
Instead of trying to fix the latest development changes we're going to try to rollback as best as possible.

glob is now backing up the database just in case rolling back fails.

We're hoping that reverting to 8ab444e1c80e603758759c1c71417b0a53ede131 might get us back into a good state.
We might need to go further in the commit history and cherry pick that change.

Comment 5

[:glob ✱]

(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #4)
> It seems that a security change was deployed to alertmanager in the last day
> So far, he installed the sqlalchemy python package which was missing for the
> latest changes.

for the record..

ouija doesn't use a virtualenv, which made this all the more interesting.

there were two versions of MySQL-python installed, one by pip and one by apt-get.  i had to purge the apt-installed package to resolve versioning mismatches between MySQLdb and _mysql.

Comment 6

[Kim Moir [:kmoir] ET]

re: comment 4

Armen: is there a way we can pin the revision that the existing seta server uses so that it is not brought down by code deploys meant for the new server?

Comment 7

[:glob ✱]

(In reply to Kim Moir [:kmoir] from comment #6)
> Armen: is there a way we can pin the revision that the existing seta server
> uses so that it is not brought down by code deploys meant for the new server?

i found no evidence of automated code deployment, so there shouldn't be a problem.

i set up a nightly backup of the database (to /mnt/mysql/backups), and checked out revision 8ab444e1c80e603758759c1c71417b0a53ede131.

Comment 8

[Armen [:armenzg]]

I will create a new branch on mozilla/ouija to be used for alertmanager.
We will stay on that branch.

Comment 9

[Armen [:armenzg]]

Thanks glob! I will take it from here since I'm getting access.

It seems that 8ab444e1c80e603758759c1c71417b0a53ede131 gives us an issue with the dates returned in the json data to be a day into the future.

I'm going to revert to eed18721a3c993e251c2f0708a557073adc62193 and cherry pick the sql injection change.

Comment 10

[Armen [:armenzg]]

This is what I've done so far.
* Created a new branch with only the change we want [1]
* Updated the host and restarted apache [2]

Unfortunately, I still see the bad date in here:
http://alertmanager.allizom.org/data/setadetails/?date=2016-08-09&buildbot=1&branch=fx-team&inactive=1

> {"jobtypes": {"2016-08-10":

[1]
https://github.com/mozilla/ouija/commits/production-alertmanager

armenzg@armenzg-thinkpad:~/repos/ouija$ git checkout -b production-alertmanager
Switched to a new branch 'production-alertmanager'
armenzg@armenzg-thinkpad:~/repos/ouija$ git reset --hard eed18721a3c993e251c2f0708a557073adc62193
HEAD is now at eed1872 merged updated and cleaned up updatedb.py
armenzg@armenzg-thinkpad:~/repos/ouija$ git push origin production-alertmanager
Total 0 (delta 0), reused 0 (delta 0)
To [e-mail]:mozilla/ouija.git
 * [new branch]      production-alertmanager -> production-alertmanager
armenzg@armenzg-thinkpad:~/repos/ouija$ git cherry-pick 8ab444e1c80e603758759c1c71417b0a53ede131
[production-alertmanager 948a182] ensure we avoid common SQL injection attacks
 Author: Joel Maher <[e-mail]>
 Date: Mon Aug 8 17:29:54 2016 +0000
 1 file changed, 31 insertions(+), 11 deletions(-)
armenzg@armenzg-thinkpad:~/repos/ouija$ git push
Counting objects: 4, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 823 bytes | 0 bytes/s, done.
Total 4 (delta 3), reused 0 (delta 0)
To [e-mail]:mozilla/ouija.git
   eed1872..948a182  production-alertmanager -> production-alertmanager

[2]
ubuntu@ouija:~/ouija$ git checkout -b production-alertmanager origin/production-alertmanager
M	tools/seta.cfg
Branch production-alertmanager set up to track remote branch production-alertmanager from origin.
Switched to a new branch 'production-alertmanager'
ubuntu@ouija:~/ouija$ sudo apache2ctl restart
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using ouija.allizom.org. Set the 'ServerName' directive globally to suppress this message

Comment 11

[Armen [:armenzg]]

I also cherrypicked the following change [1]
So far kmoir says that checkconfig works.

[1]
armenzg@armenzg-thinkpad:~/repos/ouija$ git cherry-pick 4e8f99e8191d372e1abaa2f3863022a6803e607c
[production-alertmanager 8699ce0] fix date off by 1 for setadetails/ query
 Author: Joel Maher <[e-mail]>
 Date: Mon Aug 8 19:30:02 2016 +0000
 1 file changed, 1 insertion(+), 1 deletion(-)
armenzg@armenzg-thinkpad:~/repos/ouija$ git push
Counting objects: 4, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 410 bytes | 0 bytes/s, done.
Total 4 (delta 3), reused 0 (delta 0)
To [e-mail]:mozilla/ouija.git
   948a182..8699ce0  production-alertmanager -> production-alertmanager

Comment 12

[Armen [:armenzg]]

To make it extra clear, we can now merge anything to master since alertmanager does not use `master` anymore.

Comment 13

[Armen [:armenzg]]

I believe this is fixed.
I'm waiting for confirmation of reconfigurations to have completed.

Comment 14

[Justin Wood (:Callek)]

All but the last master has reconfig'd (its in progress)