If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Update builds-YYYY-MM-DD.js.gz more than once a day so there are no gaps between the daily archives and builds-4hr

RESOLVED FIXED

Status

Release Engineering
General
P2
normal
RESOLVED FIXED
5 years ago
5 months ago

People

(Reporter: philor, Assigned: nthomas)

Tracking

({sheriffing-untriaged})

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

5 years ago
According to my vague memory, back in the dark ages we were updating the daily file that sits next to builds-4hr.js.gz pretty often, maybe as often as we updated 4hr. That was on an old and overloaded cruncher, which was constantly becoming crunchered by all the weight it was carrying.

According to my entirely too clear memory, when asked whether it would be okay to switch to just creating the daily file once a day at midnight, I said "sure, that would only be a problem for someone starting up a new instance of tbpl, and how often does that happen?"

The answer is, that happens every single time a tbpl developer wants to work on a patch: unsuspend the VM where you have tbpl installed, it fetches the 4hr, because tbpl developers work at times when pushes don't happen, that gives you little or no data, give import-buildbot-data.py the command to import one day's worth, and that gives you absolutely nothing, give it the command to import two day's worth, and you can then work by scrolling back to yesterday's pushes.

In the shiny scl3 world of boundless resources, can we afford to update the daily file every 4 hours? That would make import-buildbot-data.py -d 1 import the last 4 hours from 4hr.js.gz, and everything since midnight from that day's file, which is plenty to work with except for poor edmorley, who will be doing that just after midnight and still needing -d 2.
(Assignee)

Comment 1

5 years ago
It looks like it takes 4 to 6 minutes to generate the file for 24 hours worth of data, so generating every four hours seems like it would be fine. What do you think catlee & bear ?

Comment 2

5 years ago
I don't see why we can't run it hourly and see what the impact is.

Some of the beefier briar-patch jobs are nice'd so they should play nicely with this new kid on the block.
@hourly WFM
(Assignee)

Comment 4

5 years ago
Is there any advantage in running it hourly ? The TBPL dev still needs to grab builds-4hr.js.gz anyway, so it just causes data overlap and more load the db.

Comment 5

5 years ago
I would love to see philor put the pressure on tbpl so that they can get more realtime ;)

but no, hourly just because I feel that is more appropriate for the data - but it is just as arbitrary as every 4 hours
(Reporter)

Comment 6

5 years ago
tbpl's import-buildbot-data.py doesn't have any option to not fetch builds-4hr, so if you start up at 12:15, you're going to get 08:15 - 12:15 in builds-4hr, and 00:00 - 12:00 in the daily. 4 hours isn't really all that arbitrary, it's "we really hope generating the builds-4hr file isn't ever busted longer than this (though it has been a few times), but this is still small enough to generate quickly and fetch quickly and process quickly" as a reason to do builds-4hr, and then since you're fetching it anyway, having the daily file update every 4 hours keeps you from missing anything while minimizing the overlap.

Except, maybe it doesn't, and maybe 3 hours would be better than 4. If you start up at 12:04 and get 08:04 - 12:04 in builds-4hr, and the daily got a quick update at 08:00 but is getting a slow one at 12:00, maybe you get 00:00 - 08:02 from the daily, and have a confusing 2 minute gap that makes you think you broke something if a run you're looking at falls in that gap?
(Assignee)

Updated

5 years ago
Assignee: nobody → nrthomas
Priority: -- → P2
(Assignee)

Comment 7

5 years ago
Created attachment 636577 [details] [diff] [review]
[puppet-manifests] generate every 3 hours
Attachment #636577 - Flags: review?(catlee)
Comment on attachment 636577 [details] [diff] [review]
[puppet-manifests] generate every 3 hours

Review of attachment 636577 [details] [diff] [review]:
-----------------------------------------------------------------

don't think this works because report-daily.sh generates data for "yesterday" and we want data for today.
Attachment #636577 - Flags: review?(catlee) → review-
(Assignee)

Comment 9

5 years ago
Created attachment 636956 [details] [diff] [review]
[puppet-manifests] generate every 3 hours

I think this should work, but not sure how I test that prior to landing. During the day we generate the builds so far, using date -d '' to return the previous midnight; then at midnight generate again with date -d yesterday to include the last 3 hours of the day.
Attachment #636956 - Flags: feedback?(catlee)
(Assignee)

Updated

5 years ago
Attachment #636577 - Attachment is obsolete: true
Comment on attachment 636956 [details] [diff] [review]
[puppet-manifests] generate every 3 hours

Review of attachment 636956 [details] [diff] [review]:
-----------------------------------------------------------------

I think I used -d 'now' in the previous incarnation of this script, but -d '' works too.
Attachment #636956 - Flags: feedback?(catlee) → feedback+
QA Contact: lsblakk → hwine

Updated

5 years ago
Whiteboard: [sheriff-want]
Is this f+ patch ready for a full review? :-)

Updated

5 years ago
Keywords: sheriffing-untriaged
Whiteboard: [sheriff-want]
Product: mozilla.org → Release Engineering

Updated

3 years ago
Blocks: 1088354

Updated

3 years ago
Summary: Can we afford to update builds-YYYY-MM-DD.js.gz more than once a day now? → Update builds-YYYY-MM-DD.js.gz more than once a day so there are no gaps between the daily archives and builds-4hr
We have, for as long as I can remember, had a report-today.sh running every fifteen minutes, updating the current day's buildjson, with this content:

#!/bin/sh
buildapi_lib="<%= @buildapi_lib %>"
buildapi_lock="<%= @buildapi_lock %>"
buildapi_log="<%= @buildapi_log %>"
buildapi_www="<%= @buildapi_www %>"
reporter_cfg="<%= @reporter_cfg %>"
builddata_dir="<%= @builddata_dir %>"
lockfile=${buildapi_lock}/builds-today
builds_today=${builddata_dir}/buildjson/builds-$(date +\%Y-\%m-\%d).js.gz
builds_yesterday=${builddata_dir}/buildjson/builds-$(date -d yesterday +\%Y-\%m-\%d).js.gz

(
    /usr/bin/flock -w 1 200 || exit

    python2.7 "${buildapi_www}/run-reporter" \
        --config ${reporter_cfg} -z \
        -o ${builds_today} \
        --startdate $(date +\%Y-\%m-\%d) \
        >> ${buildapi_log}/reporter-today.log 2>&1

    # also generate the l10n report, based on that report and yesterday's
    python2.7 ${buildapi_www}/buildapi/buildapi/scripts/l10n_report.py \
        -o ${builddata_dir}/buildjson/l10n_reports/$(date +\%Y-\%m-\%d).txt \
        ${builds_yesterday} ${builds_today} \
        >> ${buildapi_log}/reporter-today.log 2>&1

    # and fix up some links
    ln -sf ${builddata_dir}/buildjson/l10n_reports/$(date +\%Y-\%m-\%d).txt ${builddata_dir}/buildjson/l10n_reports/latest.txt
    rm -f ${lockfile}
) 200>${lockfile}

This has been in place at least since we migrated buildapi to the releng web cluster, about 18 months ago.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
That's great :-)
Component: Tools → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.