1384643 - Make Coverage ETL Health Monitor

Status: RESOLVED FIXED

(Testing :: Code Coverage, enhancement)

Description

[Kyle Lahnakoski [:ekyle]]

Monitor the data going into ActiveData so that we can get feedback about what is missing, and maybe add alerts when coverage is incomplete.

Work started here:

https://github.com/klahnakoski/coco-diff/blob/master/status.py

Oh dear! This is what it looks like (July 26, 2017). The delay with ingestion is expected, but only 1/2 of the tasks with coverage are getting in. This looks bad, but the losses are probably caused by a single bug.

> 2017-07-26 16:44:26 - 07484bfdb96b - 0% DONE - 2017-07-25 12:27:00 Tasks 3/150 (2%)  Files 0/7899 (0%)
> 2017-07-26 16:44:32 - 80394cbcae0f - 0% DONE - 2017-07-25 12:18:00 Tasks 5/150 (3%)  Files 0/8797 (0%)
> 2017-07-26 16:45:10 - a2058025fd77 - 0% DONE - 2017-07-25 08:33:00 Tasks 9/153 (6%)  Files 0/3107 (0%)
> 2017-07-26 16:45:57 - dcfb58fcb6dd - 0% DONE - 2017-07-25 01:06:00 Tasks 27/183 (15%)  Files 0/7491 (0%)
> 2017-07-26 16:46:14 - f6528783c525 - 5% DONE - 2017-07-25 00:57:00 Tasks 37/180 (21%)  Files 1236/5276 (23%)
> 2017-07-26 16:46:27 - 131e19a573e9 - 0% DONE - 2017-07-25 00:26:00 Tasks 13/157 (8%)  Files 0/6017 (0%)
> 2017-07-26 16:46:38 - 899590a34d56 - 3% DONE - 2017-07-24 23:08:00 Tasks 33/159 (21%)  Files 1000/8128 (12%)
> 2017-07-26 16:46:47 - bf4634a50775 - 0% DONE - 2017-07-24 21:50:00 Tasks 26/170 (15%)  Files 0/8180 (0%)
> 2017-07-26 16:47:00 - 462d7561089c - 10% DONE - 2017-07-24 16:28:00 Tasks 35/179 (20%)  Files 3987/7779 (51%)
> 2017-07-26 16:47:17 - 60a5308fa987 - 23% DONE - 2017-07-24 11:01:00 Tasks 52/164 (32%)  Files 5731/7792 (74%)
> 2017-07-26 16:47:37 - 758016d02607 - 25% DONE - 2017-07-24 10:58:00 Tasks 44/166 (27%)  Files 6243/6732 (93%)
> 2017-07-26 16:47:58 - 5928d905c0bc - 48% DONE - 2017-07-23 16:19:00 Tasks 47/194 (24%)  Files 12176/6149 (198%)
> 2017-07-26 16:48:10 - f1b0580cb69b - 55% DONE - 2017-07-23 10:21:00 Tasks 38/82 (46%)  Files 11236/9432 (119%)
> 2017-07-26 16:48:30 - 23d9d005d92a - 60% DONE - 2017-07-23 10:00:00 Tasks 62/194 (32%)  Files 12029/6451 (186%)
> 2017-07-26 16:48:43 - fd3634c4b16d - 32% DONE - 2017-07-23 09:27:00 Tasks 43/194 (22%)  Files 11045/7669 (144%)
> 2017-07-26 16:48:59 - c023ce417c98 - 35% DONE - 2017-07-23 09:25:00 Tasks 49/193 (25%)  Files 12003/8675 (138%)
> 2017-07-26 16:49:19 - c22502562670 - 38% DONE - 2017-07-22 16:24:00 Tasks 80/191 (42%)  Files 7000/7657 (91%)
> 2017-07-26 16:49:37 - 7ce557b85b61 - 32% DONE - 2017-07-22 11:12:00 Tasks 74/189 (39%)  Files 6128/7537 (81%)
> 2017-07-26 16:49:51 - a599289ac64b - 15% DONE - 2017-07-22 09:38:00 Tasks 49/193 (25%)  Files 3898/6675 (58%)
> 2017-07-26 16:50:03 - 057d626fc5e1 - 23% DONE - 2017-07-22 01:12:00 Tasks 67/181 (37%)  Files 5522/9053 (61%)
> 2017-07-26 16:50:14 - a8bb5530a092 - 21% DONE - 2017-07-22 00:51:00 Tasks 54/173 (31%)  Files 5491/8024 (68%)

Comment 1

[Kyle Lahnakoski [:ekyle]]

this now inserts missing artifacts into the work queue.