Closed Bug 601740 Opened 11 years ago Closed 10 years ago

buildbot JSON should be generated at higher frequency and with less latency

Categories

(Release Engineering :: General, defect, P4)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: dbaron, Assigned: catlee)

References

Details

(Whiteboard: [json])

A problem I've run into building a tinderboxpushlog replacement off of the data in http://build.mozilla.org/builds/ is that the JSON files there are updated only every 5 minutes, and the updates tend to be pushing data that's already 10 minutes or so old.

If we want a production-quality TBPL replacement to be based on the data there, they need to be more up-to-date.
Whiteboard: [json]
assigning to Catlee for prioritization.
Assignee: nobody → catlee
Priority: -- → P4
Is this just a matter of tweaking one number, or would it create load problems? If so, would shortening the time span help? For example, we could replace builds-4hr.js.gz with builds-10min.js.gz; generating that might be faster and could done at a higher frequency.
It would also waste less time on the TBPL side. TBPL's import script imports builds-4hr.js.gz every 5 minutes, so it only really uses 2% of that data most of the time.
(In reply to comment #2)
> Is this just a matter of tweaking one number, or would it create load
> problems? If so, would shortening the time span help? For example, we could
> replace builds-4hr.js.gz with builds-10min.js.gz; generating that might be
> faster and could done at a higher frequency.
> It would also waste less time on the TBPL side. TBPL's import script imports
> builds-4hr.js.gz every 5 minutes, so it only really uses 2% of that data
> most of the time.

It's mostly a matter of load, although that seems to be much better now that we fixed our redis cache. (http://people.mozilla.org/~catlee/sattap/79452b19.png)

I'll increase the frequency to once a minute, and we'll see how the machine behaves.
FYI, there is an every-2-minute cron job that pulls the files to build.m.o, but we can get changed or move to a push model if necessary.
I think I changed that to run every minute.
So you did. Carry on!
60 seconds ought to be enough for anybody.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.