1035631 - [prod] update-crash-adu-by-build-signature failing

Status: RESOLVED FIXED

(Socorro :: Database, task)

Description

[Robert Helmer [:rhelmer]]

Just happened to notice this in #socorro-alerts:

23:42 < nagios-phx1> Mon 23:42:47 PDT [1393] 
                     sp-admin01.phx1.mozilla.com:Socorro Admin - crontab is 
                     CRITICAL: CRITICAL - update-crash-adu-by-build-signature 
                     (CrashAduByBuildSignatureCronApp) 
                     (http://m.mozilla.org/Socorro+Admin+-+crontab)

Comment 1

[Robert Helmer [:rhelmer]]

Looks like bad data is getting in somewhere:

DataError: source string too short for "YYYY" formatting field
DETAIL:  Field requires 4 characters, but only 1 remain.
HINT:  If your source string is not fixed-width, try using the "FM" modifier.

Comment 2

[Robert Helmer [:rhelmer]]

Here it is, bad build ID:

breakpad=# select uuid, build from reports_clean where date_processed > '2014-07-07' and date_processed < '2014-07-08' and length(build::text) < 5;
                 uuid                 | build 
--------------------------------------+-------
 5715f6f4-8783-47be-9599-0584b2140707 |     2
(1 row)

Comment 3

[Robert Helmer [:rhelmer]]

Removed this so reporting can continue (crontabber should do so automatically) - however we should fix this ASAP, these should be thrown out and not accepted to reports_clean:

breakpad=# delete from reports_clean where uuid = '5715f6f4-8783-47be-9599-0584b2140707' and date_processed > '2014-07-07' and date_processed < '2014-07-08';
DELETE 1

Comment 4

[Lonnen :lonnen]

This is happening again. I'm on it.

Comment 5

[Lonnen :lonnen]

breakpad=# select uuid, build from reports_clean where date_processed > '2014-07-07' and length(build::text) < 5;                 uuid                 | build
--------------------------------------+-------
 38759a54-0348-4302-8661-e6fef2140709 |     2
(1 row)

breakpad=# delete from reports_clean where uuid = '38759a54-0348-4302-8661-e6fef2140709' and date_processed > '2014-07-07' and length(build::text) < 5;
DELETE 1

Comment 6

[Robert Helmer [:rhelmer]]

And again:

breakpad=# select uuid, build from reports_clean where date_processed > '2014-07-07' and length(build::text) < 5; 
                 uuid                 | build 
--------------------------------------+-------
 68768022-59dc-4032-b6ee-8fdd02140710 |     2
(1 row)

breakpad=# delete from reports_clean where uuid = '68768022-59dc-4032-b6ee-8fdd02140710' and date_processed > '2014-07-07' and length(build::text) < 5;
DELETE 1

Comment 7

[Robert Helmer [:rhelmer]]

And again:

breakpad=# select date_processed, uuid, build from reports_clean where date_processed > '2014-07-15' and length(build::text) < 5; 
        date_processed         |                 uuid                 | build 
-------------------------------+--------------------------------------+-------
 2014-07-15 10:19:39.277057+00 | 6c02a441-3a06-4e8d-a305-dd03d2140715 |     2
(1 row)

breakpad=# delete from reports_clean where uuid = '6c02a441-3a06-4e8d-a305-dd03d2140715' and date_processed > '2014-07-15' and length(build::text) < 5;
DELETE 1

Comment 8

[Robert Helmer [:rhelmer]]

Note that this should be fixed in bug 1035689 which is why we don't see it on stage too, that has landed in git but has yet to ship.

Comment 9

[Robert Helmer [:rhelmer]]

Happened again today:

breakpad=# select date_processed, uuid, build from reports_clean where date_processed > '2014-08-11' and length(build::text) < 5; 
        date_processed         |                 uuid                 | build 
-------------------------------+--------------------------------------+-------
 2014-08-11 10:27:44.86963+00  | 422d6785-fd4b-44e1-8e51-44b9b2140811 |     2
 2014-08-11 10:27:41.072022+00 | d607f170-8343-4f85-acbf-18f3a2140811 |     2
(2 rows)

breakpad=# delete from reports_clean where uuid='422d6785-fd4b-44e1-8e51-44b9b2140811' or uuid='d607f170-8343-4f85-acbf-18f3a2140811' and date_processed > '2014-08-11' and length(build::text) <5;
DELETE 2

Comment 10

[Robert Helmer [:rhelmer]]

breakpad=# select date_processed, uuid, build from reports_clean where date_processed > '2014-08-15' and length(build::text) < 5;
        date_processed         |                 uuid                 | build 
-------------------------------+--------------------------------------+-------
 2014-08-15 05:28:51.577495+00 | b5719d3e-8e32-4dbb-9358-cfc8c2140815 |     2
(1 row)

breakpad=# delete from reports_clean where date_processed > '2014-08-15' and length(build::text) < 5 and uuid='b5719d3e-8e32-4dbb-9358-cfc8c2140815';
DELETE 1

Comment 11

[Robert Helmer [:rhelmer]]

Oh... I think this was already fixed in https://github.com/mozilla/socorro/commit/a5ed28b69916d7600871f930b610efe56acb6af4 we just haven't run the migration

Comment 12

[Robert Helmer [:rhelmer]]

This should now be fixed.