1309882 - old nightly releases not getting removed from prod db

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (backend), defect)

Description

[[e-mail] (:bhearsum)]

I just noticed that while nightly history appears to be getting removed fine, nightly releases themselves are not. Eg: All of the nightlies from July and August are still in the db.

Given that history is still being removed, it seems most likely that this is an issue with the retention script

Comment 1

[[e-mail] (:bhearsum)]

So this is interesting. I ran the "releases" query by hand and got:
ERROR 1267 (HY000): Illegal mix of collations (utf8_general_ci,COERCIBLE) and (latin1_swedish_ci,NUMERIC) for operation '<'

I discovered that my connection collation was different than the database collation:
mysql> show variables like '%collat%';                                                                                          
+----------------------+-------------------+
| Variable_name        | Value             |
+----------------------+-------------------+
| collation_connection | utf8_general_ci   |
| collation_database   | latin1_swedish_ci |
| collation_server     | latin1_swedish_ci |
+----------------------+-------------------+

After setting collation_connection to match the others, things worked fine. I was shelling into a container and then running the script/mysql to check this, so I suspect the same thing is happening in production.

Ben - would you mind checking these three variables in production, preferably from the mysql command line run from one of the docker containers? I can override collation_connection from the mysql command line, but I want to be sure of what collation_database is before I do.

Comment 2

[[e-mail] (:bhearsum)]

Hm, actually. After fiddling with the data a bit more - I'm not able to reproduce this. Eg:
root@752480d5bccc:/app# python scripts/manage-db.py -d $DBURI cleanup-dryrun 5
/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py:638: SADeprecationWarning: The 'listeners' argument to Pool (and create_engine()) is deprecated.  Use event.listen().
  Pool.__init__(self, creator, **kw)
/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py:145: SADeprecationWarning: Pool.add_listener is deprecated.  Use event.listen()
  self.add_listener(l)
Releases rows to be deleted:
Firefox-mozilla-aurora-nightly-20151223004004
releases_history (dated) rows to be deleted:
  - Firefox-mozilla-central-nightly-latest: 50 history rows
Total Deleted: 0


And after running without -dryrun, Firefox-mozilla-aurora-nightly-20151223004004 got removed.

Ben - can you also check what MAX_AGE is being set to? It's measured in days, and I think we wanted it set to 14.

Comment 3

[Benson Wong [:mostlygeek]]

MAX_AGE is set to 180 days.
Please confirm the change to 14 days. :)

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Benson Wong [:mostlygeek] from comment #3)
> MAX_AGE is set to 180 days.
> Please confirm the change to 14 days. :)

Yup, 14 is correct. It gets passed to https://github.com/mozilla/balrog/blob/master/scripts/run.sh#L47, then https://github.com/mozilla/balrog/blob/master/scripts/run-batch-deletes.sh#L29, then https://github.com/mozilla/balrog/blob/master/scripts/manage-db.py#L193, and finally subbed into https://github.com/mozilla/balrog/blob/master/scripts/manage-db.py#L31.

Comment 5

[[e-mail] (:bhearsum)]

This looks fixed now. The oldest nightlies still in the db that aren't pointed at are from 20161028004017 now. Thanks Ben!