make tinderboxpushlog not need tinderbox anymore

RESOLVED DUPLICATE of bug 625979

Status

RESOLVED DUPLICATE of bug 625979
8 years ago
3 years ago

People

(Reporter: dbaron, Unassigned)

Tracking

Dependency tree / graph

Details

Something we've talked about for a bit is making tinderboxpushlog not need tinderbox anymore.  I couldn't find a bug on it, so I'm filing one.

So, since the tinderbox server was nearly unusable for much of the day today, I hacked tinderboxpushlog to get what it could out of the data at http://build.mozilla.org/builds/ .  I have something working, though it's not all that useful for anything other than topline tree status.

The code I wrote today has two parts:

First, I wrote a python script to filter out only the parts of the json that we need, and also merge some of them together.  This lives at http://hg.mozilla.org/users/dbaron_mozilla.com/buildbot-json-tools/

Second, I hacked tinderboxpushlog (in a rather ugly way) to use the data from that instead of tinderbox; my current hack is in my tbpl patch queue here:
http://hg.mozilla.org/users/dbaron_mozilla.com/tinderboxpushlog-patches/raw-file/108078f96062/buildbot-json
(note that that's a link to the version as I write this, not to the latest version).


Big things that I know are missing:
 * starring (see bug 553549)
   + this would also need builds-pending.json and builds-running.json to have a unique id for the build in them; currently I'm making those up in my python code
 * build scrape data (I have no way to access the TinderboxPrint: output)
 * access to build logs
 * I haven't yet made back/forward in time (or merging of the per-day json files when needed) work yet, but that's next on my todo list, and something entirely solvable in tbpl (which the others aren't)

I'd also note there's a bit of a delay involved in generating the data; there are cron jobs to generate the data in http://build.mozilla.org/builds/ and then a cron job every 5 minutes to rsync it (leading to potentially up to 10 minutes delay, or maybe even a drop worse if the data generation takes a long time), and then I have a cron job running my python script every five minutes and uploading the results to dbaron.org (I can't use people.mozilla.com due to lack of .htaccess support).

A working instance of this is at http://dbaron.org/mozilla/tinderboxpushlog/ , at least for now, but please don't use too much of my bandwidth.
And I should note that the output of the python script is served with Apache and the following .htaccess:

AddType application/json .gz
AddEncoding x-gzip .gz
Header set Access-Control-Allow-Origin "*"
... and also:

ExpiresActive on
ExpiresDefault "access plus 1 minute"


Another issue that we need to solve is what to use as the unique identifier for a build; we'll definitely need that for starring.


I'm also somewhat disturbed by the filenames being based on Pacific-time dates rather than UTC-dates.  That can get dangerous around time changes.
(In reply to comment #2)
> Another issue that we need to solve is what to use as the unique identifier for
> a build; we'll definitely need that for starring.
> 

Do you foresee problems with using revision ID of last cset?  That appeals to me because it's instantly, publicly and permanently available, and besides try repo, locally searchable.
well, we definitely would need to combine that with other information; we do a lot of builds for each changeset and we need to distinguish them.  The question is what the best (and hopefully still reasonably compact) way to distingush them is.
oic; sorry, I totally missed the issue here (confused myself over "build").  The problem you're looking to solve is the lack of unique job identifier across all job states (pending/running/finished) in buildbot, right?  Maybe we should just fix that.
cjones: yes

Also, I moved my changes out of my TBPL patch queue into a proper hg repo here:
http://hg.mozilla.org/users/dbaron_mozilla.com/buildbotpushlog/
Would it make sense to merge tinderbox's json[2].js into what you're getting from buildbot, for the time being?  (Is that possible?)  It would be nice to get the most up-to-date build info along with access to tb logs, when they show up.  I fear that "* build scrape data" in comment 0 may take some time.
No longer depends on: 601313
Depends on: 611448
Depends on: 625887
Depends on: 656902
Depends on: 656919

Updated

7 years ago
Depends on: 630538
Depends on: 669137
If you are a tracking bug tracking tinderboxdeath, and you aren't tracking treestatusdeath, you're a dead tracking bug.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 625979
(Assignee)

Updated

4 years ago
Product: Webtools → Tree Management
(Assignee)

Updated

3 years ago
Product: Tree Management → Tree Management Graveyard
You need to log in before you can comment on or make changes to this bug.