1502958 - Run the Buildhub cron job every 6 hours instead of 24

Status: VERIFIED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[Peter Bengtsson [:peterbe]]

Long story short; Buildhub is missing new builds and it's probably due to problems with the Lambda function. [0]

It seems the cron job "plugs the holes" when it runs. The cron job runs every 24h and once a week we set something so that it "starts from scratch". 

Can we just increase the cadence from 24h to 6h?


[0]Tonnes of cryptic internal kinto errors recently https://sentry.prod.mozaws.net/operations/buildhub-prod/

Comment 1

[Peter Bengtsson [:peterbe]]

Here's the latest metrics on how long that cron job takes: https://irccloud.mozilla.com/pastebin/YsmRU4Td/

If you look at `S3_INVENTORY_TO_KINTO_RUN` for `SCOPE: ENV:PROD` and ignore the first on on Mondays it says it takes about 4 hours.

Comment 2

[:wezhou]

Done.

Since it's a little past 18:00 UTC as of this comment, next run should be in 6 hours at 24:00 UTC.

Comment 3

[Peter Bengtsson [:peterbe]]

Seems to have worked...


SCOPE: ENV:PROD
+-------------------------+-------------------------+---------------------+
|          WHEN           |        TIME AGO         |      TIME TOOK      |
+=========================+=========================+=====================+
| Tue 2018-10-30T06:00:00 | 6 hours 18 minutes ago  | 1 hour 33 seconds   |
+-------------------------+-------------------------+---------------------+
| Tue 2018-10-30T00:00:00 | 12 hours 18 minutes ago | 55 minutes 1 second |
+-------------------------+-------------------------+---------------------+
| Mon 2018-10-29T20:00:00 | 16 hours 18 minutes ago | 3 hours 49 minutes  |
+-------------------------+-------------------------+---------------------+
| Mon 2018-10-29T16:00:00 | 20 hours 18 minutes ago | 23 hours 12 minutes |
+-------------------------+-------------------------+---------------------+
| Sat 2018-10-27T20:00:00 | 2 days 16 hours ago     | 4 hours 4 minutes   |
+-------------------------+-------------------------+---------------------+
| Fri 2018-10-26T22:00:00 | 3 days 14 hours ago     | 5 hours 19 minutes  |
+-------------------------+-------------------------+---------------------+
| Wed 2018-10-24T20:00:00 | 5 days 16 hours ago     | 3 hours 57 minutes  |
+-------------------------+-------------------------+---------------------+
| Tue 2018-10-23T20:00:00 | 6 days 16 hours ago     | 3 hours 47 minutes  |
+-------------------------+-------------------------+---------------------+
| Mon 2018-10-22T20:00:00 | 1 week 16 hours ago     | 3 hours 43 minutes  |
+-------------------------+-------------------------+---------------------+
| Mon 2018-10-22T16:00:00 | 1 week 20 hours ago     | 23 hours 12 minutes |
+-------------------------+-------------------------+---------------------+
| MEDIAN                  | --                      | 3 hours 53 minutes  |
+-------------------------+-------------------------+---------------------+