765451 - Update builds-YYYY-MM-DD.js.gz more than once a day so there are no gaps between the daily archives and builds-4hr

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Phil Ringnalda (:philor)]

According to my vague memory, back in the dark ages we were updating the daily file that sits next to builds-4hr.js.gz pretty often, maybe as often as we updated 4hr. That was on an old and overloaded cruncher, which was constantly becoming crunchered by all the weight it was carrying.

According to my entirely too clear memory, when asked whether it would be okay to switch to just creating the daily file once a day at midnight, I said "sure, that would only be a problem for someone starting up a new instance of tbpl, and how often does that happen?"

The answer is, that happens every single time a tbpl developer wants to work on a patch: unsuspend the VM where you have tbpl installed, it fetches the 4hr, because tbpl developers work at times when pushes don't happen, that gives you little or no data, give import-buildbot-data.py the command to import one day's worth, and that gives you absolutely nothing, give it the command to import two day's worth, and you can then work by scrolling back to yesterday's pushes.

In the shiny scl3 world of boundless resources, can we afford to update the daily file every 4 hours? That would make import-buildbot-data.py -d 1 import the last 4 hours from 4hr.js.gz, and everything since midnight from that day's file, which is plenty to work with except for poor edmorley, who will be doing that just after midnight and still needing -d 2.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

It looks like it takes 4 to 6 minutes to generate the file for 24 hours worth of data, so generating every four hours seems like it would be fine. What do you think catlee & bear ?

Comment 2

[Mike Taylor [:bear]]

I don't see why we can't run it hourly and see what the impact is.

Some of the beefier briar-patch jobs are nice'd so they should play nicely with this new kid on the block.

Comment 3

[Chris AtLee [:catlee]]

@hourly WFM

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Is there any advantage in running it hourly ? The TBPL dev still needs to grab builds-4hr.js.gz anyway, so it just causes data overlap and more load the db.

Comment 5

[Mike Taylor [:bear]]

I would love to see philor put the pressure on tbpl so that they can get more realtime ;)

but no, hourly just because I feel that is more appropriate for the data - but it is just as arbitrary as every 4 hours

Comment 6

[Phil Ringnalda (:philor)]

tbpl's import-buildbot-data.py doesn't have any option to not fetch builds-4hr, so if you start up at 12:15, you're going to get 08:15 - 12:15 in builds-4hr, and 00:00 - 12:00 in the daily. 4 hours isn't really all that arbitrary, it's "we really hope generating the builds-4hr file isn't ever busted longer than this (though it has been a few times), but this is still small enough to generate quickly and fetch quickly and process quickly" as a reason to do builds-4hr, and then since you're fetching it anyway, having the daily file update every 4 hours keeps you from missing anything while minimizing the overlap.

Except, maybe it doesn't, and maybe 3 hours would be better than 4. If you start up at 12:04 and get 08:04 - 12:04 in builds-4hr, and the daily got a quick update at 08:00 but is getting a slow one at 12:00, maybe you get 00:00 - 08:02 from the daily, and have a confusing 2 minute gap that makes you think you broke something if a run you're looking at falls in that gap?

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [puppet-manifests] generate every 3 hours

Comment 8

[Chris AtLee [:catlee]]

Comment on attachment 636577 [details] [diff] [review]
[puppet-manifests] generate every 3 hours

Review of attachment 636577 [details] [diff] [review]:
-----------------------------------------------------------------

don't think this works because report-daily.sh generates data for "yesterday" and we want data for today.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [puppet-manifests] generate every 3 hours

I think this should work, but not sure how I test that prior to landing. During the day we generate the builds so far, using date -d '' to return the previous midnight; then at midnight generate again with date -d yesterday to include the last 3 hours of the day.

Comment 10

[Chris AtLee [:catlee]]

Comment on attachment 636956 [details] [diff] [review]
[puppet-manifests] generate every 3 hours

Review of attachment 636956 [details] [diff] [review]:
-----------------------------------------------------------------

I think I used -d 'now' in the previous incarnation of this script, but -d '' works too.

Comment 11

[Ed Morley [:emorley]]

Is this f+ patch ready for a full review? :-)

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

We have, for as long as I can remember, had a report-today.sh running every fifteen minutes, updating the current day's buildjson, with this content:

#!/bin/sh
buildapi_lib="<%= @buildapi_lib %>"
buildapi_lock="<%= @buildapi_lock %>"
buildapi_log="<%= @buildapi_log %>"
buildapi_www="<%= @buildapi_www %>"
reporter_cfg="<%= @reporter_cfg %>"
builddata_dir="<%= @builddata_dir %>"
lockfile=${buildapi_lock}/builds-today
builds_today=${builddata_dir}/buildjson/builds-$(date +\%Y-\%m-\%d).js.gz
builds_yesterday=${builddata_dir}/buildjson/builds-$(date -d yesterday +\%Y-\%m-\%d).js.gz

(
    /usr/bin/flock -w 1 200 || exit

    python2.7 "${buildapi_www}/run-reporter" \
        --config ${reporter_cfg} -z \
        -o ${builds_today} \
        --startdate $(date +\%Y-\%m-\%d) \
        >> ${buildapi_log}/reporter-today.log 2>&1

    # also generate the l10n report, based on that report and yesterday's
    python2.7 ${buildapi_www}/buildapi/buildapi/scripts/l10n_report.py \
        -o ${builddata_dir}/buildjson/l10n_reports/$(date +\%Y-\%m-\%d).txt \
        ${builds_yesterday} ${builds_today} \
        >> ${buildapi_log}/reporter-today.log 2>&1

    # and fix up some links
    ln -sf ${builddata_dir}/buildjson/l10n_reports/$(date +\%Y-\%m-\%d).txt ${builddata_dir}/buildjson/l10n_reports/latest.txt
    rm -f ${lockfile}
) 200>${lockfile}

This has been in place at least since we migrated buildapi to the releng web cluster, about 18 months ago.

Comment 13

[Ed Morley [:emorley]]

That's great :-)