Closed Bug 487036 Opened 15 years ago Closed 7 years ago

write tool to read buildbot db for BuildID+changesets of nightlies, and then construct URL to feed to hg pushlog

Categories

(Socorro :: General, task, P3)

Product:
Socorro
Component:
General
Platform:
x86
All
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: beltzner, Assigned: peterbe)

References

Details

(Keywords: student-project, Whiteboard: [automation])

Attachments

(1 file, 1 obsolete file)

latest code from last piece of feedback
9.11 KB, patch
Details | Diff | Splinter Review 
Knowing when a nightly build was "cut" from trunk or branch is useful for people looking to test changes, form regression windows, draw graphs, etc. Right now that knowledge can only be obtained from looking at a specific nightly log, Tinderbox, in the build itself, or pretty much everywhere except hg

Instead, it would be useful to create a tag as part of the nightly build process, indicating where exactly a nightly build was cut off.

I think I'd proposed this before, and people had mentioned "OMG TAG NOIZZE" or somesuch, but I figured that I'd get the bug on record so we could hash it out in terms of what the implementation benefits and costs would be.
Yeah, I'd like to understand the reasons against more clearly, because it seems like it would be nice for a couple reasons, regression hunting notable among them.

This could have an unintended side benefit, in that we could tag tip every night at (say) 1am with "BRANCH_NIGHTLY_APRIL_4_2009" and then all platforms could build the same code (my understanding is that right now nightlies each just pull tip, even if that changes between one platform's pull and the next). But that's only useful if it's convenient, I think tagging for nightlies is useful regardless of whether this fly-along proposal is.
(Updating the summary to match Johnath's comment.)

I think this is a fine idea, but I'd say we should do this for all Firefox nightlies built out of Mercurial (to be inclusive of tracemonkey and whatever future feature branches we have).

I'm not particularly concerned about tag noise given the benefits. It would make it more difficult to find release tags on http://hg.mozilla.org/mozilla-central/tags, but it's not like we don't know them in an intuitive manner.
Summary: tag nightly builds on mozilla-central and mozilla-1.9.1 → create nightly tags for mozilla-central, mozilla-1.9.1, and build the nightlies from them
(In reply to comment #2)
> It would
> make it more difficult to find release tags on
> http://hg.mozilla.org/mozilla-central/tags, but it's not like we don't know
> them in an intuitive manner.

"it's not like we don't NAME them in an intuitive manner"
re: comment 2

I've no objection to just making this a step in the generalized "build a nightly" process for any branch/tree/repo that delivers nightlies.
Copying a couple other people who think about repositories and building.
I only have technical concerns about this:

1) if the tag is done on the `default` branch, there's a commit and push every morning which potentially has commit/push races with developers. It will also cause depend builds to kick off at almost the same time as the nightlies, which doesn't seem ideal but perhaps is not a huge deal.

2) if the tag is done on some other named branch, we have to deal with all of the complexities of having named-branch heads in mozilla-central, which we've been avoiding.
(In reply to comment #6)
> I only have technical concerns about this:
> 
> 1) if the tag is done on the `default` branch, there's a commit and push every
> morning which potentially has commit/push races with developers. It will also
> cause depend builds to kick off at almost the same time as the nightlies, which
> doesn't seem ideal but perhaps is not a huge deal.

We can almost certainly ignore these changes for the purposes of kicking off depend builds.
(In reply to comment #6)
> I only have technical concerns about this:
> 
> 1) if the tag is done on the `default` branch, there's a commit and push every
> morning which potentially has commit/push races with developers. It will also
> cause depend builds to kick off at almost the same time as the nightlies, which
> doesn't seem ideal but perhaps is not a huge deal.
> 

I think we can *mostly* avoid the race by doing hg clone/pull && hg tag && hg pull && hg push. That way long initial clones or pulls have a smaller window in which they can conflict...

The dep builds being kicked off is a major PITA though...

> 2) if the tag is done on some other named branch, we have to deal with all of
> the complexities of having named-branch heads in mozilla-central, which we've
> been avoiding.

If we do this, it should be done on default for these reasons and simple cleanliness as well.
Sure, then the race will only manifest as "push failed". Does that mean we won't have nightlies that day (or will have to manually start them again)?
I thought this was filed previously, but maybe it was just discussed. The idea of accumulating a zillion tags for nightly builds is sort of gross to me, but I do in fact like the concept. Plus, as a bonus, we could get nightlies on all platforms built from the same exact changeset. See also bug 413572.
Why do we need tags?  We have changesets!

If we publish them better (such as on a nightly.mozilla.org or even a post to dev-tree-management every morning when that day's nightly cset is selected) then we can just use a changeset ID for all the platforms, avoid any conflict races, avoid tag pollution in the tree, and sip boat drinks.
Yeah, just publishing the nightly cset ID would certainly be easier and less churn than actually tagging it in VCS.
And all our tools understand changesets already, which is pretty "woo!".
If we wanted, instead of rolling another system we could add quasi-tags to pushlog.
(In reply to comment #11)
> Why do we need tags?  We have changesets!
> 
> If we publish them better (such as on a nightly.mozilla.org or even a post to
> dev-tree-management every morning when that day's nightly cset is selected)
> then we can just use a changeset ID for all the platforms, avoid any conflict
> races, avoid tag pollution in the tree, and sip boat drinks.

The reason I liked tags was that it would allow sleuthing to be done entirely within hg, without cross-correlating against newsgroup posts or websites or file drops.

As things stand, if I want to know whether a fix has made it in to a nightly or not, I go to tinderbox and use that to find the nightly log which will then tell me which changeset it was built with, then hop back to hg to see whether that changeset is before or after the one I care about.  Likewise, if I know (through binary search, for instance) that a regression appeared in the 03-10 nightly, I use tinderbox to get at the logs for those two nightlies and then go back to hg to construct a changeset-to-changeset range.  Since hg understands tags as revision IDs, with tags I could just do:

hg diff -r MARCH_10_NIGHTLY -r MARCH_11_NIGHTLY

(or the hg log equivalent)

Having said all that though - a solution which puts the changesets somewhere easier to see solves that problem almost as well, and if it involves less infra churn, I am all for boat drinks.  Hell, a file-drop style solution might help with bug 474610, too.
Moving this into Future for now, as there's no clear consensus on what we should be doing.

I'm slightly uneasy about having these tags in the repository.  There are other ways to make sure that all our nightlies are based on the same revision, if that's something we want.

That being said, one of the goals of the buildbot database is to make this kind of information easily accessible.  You won't be able to work directly in hg, but maybe it would be good enough?
Component: Release Engineering → Release Engineering: Future
I don't care if it's tags or not, really. I care that it's something predictable that someone can search for in hg, or other tools can read from in order to automatically build regression ranges or identify what changes went into a specific nightly.

For example, we could create a text file in /mozilla called "foop" which just contained the timestamp and changeset of when buildbot pulled for a nightly build. The build automation for nightlies would update that file with the timestamp right after they pull; both that push and the contents of the file would be useful.
(In reply to comment #17)
> For example, we could create a text file in /mozilla called "foop" which just
> contained the timestamp and changeset of when buildbot pulled for a nightly
> build. The build automation for nightlies would update that file with the
> timestamp right after they pull; both that push and the contents of the file
> would be useful.

Yep, this would be great by me. I can hg log it if I want historical data, and we don't need tag soup.
Yeah, so a file like

/mozilla/currentnightly.txt

that had the following-ish contents:

Latest w32 nightly was built at: <timestamp>
using: <changeset>

Latest osx nightly was built at: <timestamp>
using: <changeset>

Latest linux nightly was built at: <timestamp>
using: <changeset>
To clarify this for myself, the idea here is to update a file inside version control that contains this list of revisions.

Something like:
<timestamp> <changeset>

would probably suffice.  After this is done, the nightly builds could be triggered to build from <changeset>.
Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.
Component: Release Engineering: Future → Release Engineering
Priority: -- → P3
Thinking about how to re-word the summary.

Possibly "tool to diff changesets between any two builds".
I think the best way to describe what this bug wants is:
1) Ensure that nightlies on all platforms are built from the same changeset on a particular day
2) Provide a way to locate the current nightly (and its changeset), as well as historical data.

I've got a prototype webapp up for #2 here:
http://ted.mielczarek.org/mozilla/nightly/ (Source at http://hg.mozilla.org/users/tmielczarek_mozilla.com/nightly-db/ )
Whiteboard: [automation]
Keywords: student-project
Found in triage. Investigating.
Assignee: nobody → joduinn
Bug#570814 is tracking, for a given branch, having all nightlies-on-all-OS being built from the same changeset.
Depends on: 570814
I'm untangling these 4 overlapping bugs:

https://bugzilla.mozilla.org/show_bug.cgi?id=449607
https://bugzilla.mozilla.org/show_bug.cgi?id=487036
https://bugzilla.mozilla.org/show_bug.cgi?id=538540
https://bugzilla.mozilla.org/show_bug.cgi?id=584178


After some study, I believe these are about two interwoven but orthogonal issues:

1) improve regression hunting
=============================
There are a few ways to do this:
a) beltzner's tagging proposal. 
b) to look on ftp, at the txt file containing changeset which RelEng already post in a txt file alongside the build. 
c) to create a website reporting from buildbot db and present that in a public UI.

2) organize builds on ftp.m.o so that each build is in a unique and consistent location
=======================================================================================
This means build systems can programmaticly calculate locations of builds and their log files, which is needed to remove a dependency on TinderboxServer. This needs to handle rebuilds of the same changeset (like for respins of nightlies). Using changesets in the dirname is unique, but does not sort chronologically, and does not handle respins. However, using the full BuildID does handle both these usage cases.



I propose the following:

* morph https://bugzilla.mozilla.org/show_bug.cgi?id=487036 into "write tool to read buildbot db for BuildID+changesets of nightlies, and then construct URL to feed to hg pushlog". For example, buildbot status db contains a list of changesets, and buildids. RelEng would write a dashboard which queries the db and generate a URL to send to hgpushlog. For example, this URL shows all changes between Tues nightly (early morning of 14dec) and Wed nightly (early morning of 15dec): http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=66036625795f&tochange=f11f7ed625ba

* FIX https://bugzilla.mozilla.org/show_bug.cgi?id=449607 "change dated dirs on ftp.m.o to use new longer BuildID", using the full BuildID as outlined above.

* WONTFIX https://bugzilla.mozilla.org/show_bug.cgi?id=538540 "stop putting hour number in nightly directories" because of bug#449607

* WONTFIX https://bugzilla.mozilla.org/show_bug.cgi?id=584178 "list hourly tinderbox builds by changeset on ftp.mozilla.org" because of bug#449607


If I've missed anything in all this detangling, please let me know.
See Also: → 449607, 538540, 584178
(In reply to comment #26)
> c) to create a website reporting from buildbot db and present that in a public
> UI.

> * morph https://bugzilla.mozilla.org/show_bug.cgi?id=487036 into "write tool to
> read buildbot db for BuildID+changesets of nightlies, and then construct URL to
> feed to hg pushlog". For example, buildbot status db contains a list of
> changesets, and buildids. RelEng would write a dashboard which queries the db
> and generate a URL to send to hgpushlog. For example, this URL shows all
> changes between Tues nightly (early morning of 14dec) and Wed nightly (early
> morning of 15dec):
> http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=66036625795f&tochange=f11f7ed625ba

This sounds like the most useful thing for regression hunters, assuming this tool can also link to the nightly build on FTP. Generally, people (and tools) trying to download nightly builds want to be able to reliably find a nightly from a specific date, and right now that's kind of hard. I wrote a little webapp to do this a while ago (whose cron job hasn't been running, so the data is out of date): http://ted.mielczarek.org/mozilla/nightly/ It simply scraped the text files on FTP (or the buildbot JSON) for its source data.

Ideally we'd be able to query something like "http://nightly.mozilla.org/mozilla-central/2011/01/01/linux-x86" and get back a JSON blob listing all the x86 Linux nightlies that were produced on that date, along with their Build IDs and the changesets they were built from.
(In reply to comment #26)
> I'm untangling these 4 overlapping bugs:
...
> I propose the following:
> 
> * morph https://bugzilla.mozilla.org/show_bug.cgi?id=487036 into "write tool to
> read buildbot db for BuildID+changesets of nightlies, and then construct URL to
> feed to hg pushlog". For example, buildbot status db contains a list of
> changesets, and buildids. RelEng would write a dashboard which queries the db
> and generate a URL to send to hgpushlog. For example, this URL shows all
> changes between Tues nightly (early morning of 14dec) and Wed nightly (early
> morning of 15dec):
> http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=66036625795f&tochange=f11f7ed625ba
> 
> * FIX https://bugzilla.mozilla.org/show_bug.cgi?id=449607 "change dated dirs on
> ftp.m.o to use new longer BuildID", using the full BuildID as outlined above.
> 
> * WONTFIX https://bugzilla.mozilla.org/show_bug.cgi?id=538540 "stop putting
> hour number in nightly directories" because of bug#449607
> 
> * WONTFIX https://bugzilla.mozilla.org/show_bug.cgi?id=584178 "list hourly
> tinderbox builds by changeset on ftp.mozilla.org" because of bug#449607
> 
> 
> If I've missed anything in all this detangling, please let me know.

per meeting with beltzner yesterday, he likes proposal and says "yes, make it so".
(In reply to comment #26)
...
> I propose the following:
> 
> * morph https://bugzilla.mozilla.org/show_bug.cgi?id=487036 into "write tool to
> read buildbot db for BuildID+changesets of nightlies, and then construct URL to
> feed to hg pushlog". For example, buildbot status db contains a list of
> changesets, and buildids. RelEng would write a dashboard which queries the db
> and generate a URL to send to hgpushlog. For example, this URL shows all
> changes between Tues nightly (early morning of 14dec) and Wed nightly (early
> morning of 15dec):
> http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=66036625795f&tochange=f11f7ed625ba
Summary: create nightly tags for mozilla-central, mozilla-1.9.1, and build the nightlies from them → write tool to read buildbot db for BuildID+changesets of nightlies, and then construct URL to feed to hg pushlog
from irc with laura: 

1) Fixing this will also allow some cleanup of socorro website scraping, so she wants to take this.

2) Pushing over to Laura to figure out scope, schedule and she'll update this bug when she has more info.
Assignee: joduinn → nobody
Component: Release Engineering → Socorro
Product: mozilla.org → Webtools
QA Contact: release → socorro
Assignee: nobody → laura
Blocks: 636136
(In reply to comment #31)
> 1) Fixing this will also allow some cleanup of socorro website scraping, so
> she wants to take this.
> 
> 2) Pushing over to Laura to figure out scope, schedule and she'll update
> this bug when she has more info.

Pulling zandr and arich into this, they can sync up with Laura about implementing.
Final ping for feedback on the spec, otherwise I will implement it as written starting tomorrow.
This sounds about exactly what I'd want. Just one nit, for the "Get buildids by date" API call, instead of just returning a list of build ids, can it return build id/changeset pairs? That'd avoid having to hit the API once per build id if you want that info (and presumably wouldn't be very hard to provide).

Maybe any API call that returns build info (so everything listed there except for "list all products" and "list product versions") should just return build id/changeset/download URL triplets in the result to make things simpler?
Depends on: 640242
peterbe is going to work on this, and have a patch up by 6/10.

I talked to catlee, and rather than build this out from scratch, we're going to integrate it with 
https://wiki.mozilla.org/ReleaseEngineering/BuildAPI
Assignee: laura → peterbe
The doc string in controllers/constructicon.py should describe how it works. I've got a couple of scripts as well to manually test these API methods which I can upload if you like.
Attachment #541732 - Flags: review?(catlee)
(In reply to comment #33)
> Wrote API doc, feedback welcome:
> https://wiki.mozilla.org/Webtools/BuildAPI

For automatic regression finding, we need a way to go from changeset ID to build ID (or build directory) for a given branch+type+OS.  For example,

  input
    firefox/tinderbox-builds/mozilla-central-linux-debug + rev=10312d31fe7e

  output
    1308844600 (or https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1308844600/)
Blocks: 669244
Comment on attachment 541732 [details] [diff] [review]
First alpha version of constructicon

Review of attachment 541732 [details] [diff] [review]:
-----------------------------------------------------------------

Overall looking really good. I can never remember if we need to be able to handle just nightlies for this, or to be able to distinguish them somehow from the depend builds.

::: buildapi/controllers/constructicon.py
@@ +25,5 @@
> +
> +        [{'branch': 'places-linux-talos',
> +          'project': '',
> +          'revision': '797ce3f26f043d30c12a97487f2ee2cb699d76ad',
> +          'url': 'http://hg.mozilla.org/places-linux-talos/rev/797c...ad'}]

I think we need to filter out results like this. 'places-linux-talos' isn't a real branch, and should be 'projects/places' instead.

@@ +47,5 @@
> +
> +    ----------------------------------------------------------------------------
> +
> +    /constructicon/latest/branch/:branch
> +    /constructicon/latest/project/:project

What are possible values of :project here?

@@ +91,5 @@
> +        if not obj:
> +            return "*nothing*"
> +        out = StringIO()
> +        pprint(obj, out)
> +        return out.getvalue()

Why use pprint(obj, out) rather than out = pformat(obj)?

@@ +159,5 @@
> +    def _parse_date(self, date):
> +        """return a datetime.date instance"""
> +        assert isinstance(date, basestring)
> +        dt = datetime.datetime.strptime(date, '%Y-%m-%d')
> +        return datetime.date(dt.year, dt.month, dt.day)

Does this need to be an instance method?

::: buildapi/model/constructicon.py
@@ +17,5 @@
> +                     bsp.c.property_value.contains(value)
> +                     )))
> +    #print_query(q)
> +    return set(x[0] for x in q.execute())
> +

this is going to be painfully slow without indexes on these columns. I'll file a bug to see if we can get those added soon.

@@ +107,5 @@
> +    if project:
> +        url += 'projects/%s' % project
> +    else:
> +        url += '%s' % branch
> +    return url + '/rev/%s' % revision

I think this will fail for branches like 'mozilla-aurora' or 'mozilla-beta', which aren't in the top-level of hg.m.o, nor in projects (they're in /releases/).

@@ +109,5 @@
> +    else:
> +        url += '%s' % branch
> +    return url + '/rev/%s' % revision
> +
> +def getBuild(branch=None, product=None):

rename to getLatestBuild?

this function will return all buildids for the branch, not just nightlies. is that what you want?
Attachment #541732 - Flags: review?(catlee)
I got looped in by Laura and committed to taking a stab at providing the storage and query interface for this data.  Here is the draft doc I put up describing it:
https://wiki.mozilla.org/ReleaseData
Comment on attachment 541732 [details] [diff] [review]
First alpha version of constructicon

Review of attachment 541732 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for the feedback. I'm going to upload a new patch with some changes from your feedback and also corrections of PEP8 style and pyflakes check.

::: buildapi/controllers/constructicon.py
@@ +25,5 @@
> +
> +        [{'branch': 'places-linux-talos',
> +          'project': '',
> +          'revision': '797ce3f26f043d30c12a97487f2ee2cb699d76ad',
> +          'url': 'http://hg.mozilla.org/places-linux-talos/rev/797c...ad'}]

Well, it all depends on what your use for input. Is it likely that the user of this API will input buildIDs that return changesets from such branches?

What we could do is write down a list of branches in the constructicon config file of branches that should never be returned. Return to my first question just above and take that into account and when you are certain I'll go ahead and add such a config setting.

@@ +47,5 @@
> +
> +    ----------------------------------------------------------------------------
> +
> +    /constructicon/latest/branch/:branch
> +    /constructicon/latest/project/:project

The only value in there is "". There are no other values in the database. I should have spotted this earlier. Sorry.

@@ +91,5 @@
> +        if not obj:
> +            return "*nothing*"
> +        out = StringIO()
> +        pprint(obj, out)
> +        return out.getvalue()

Easy to explain: I simply didn't know about pprint.pformat()
That's ok though, I've only done python for about 11 years :)

@@ +159,5 @@
> +    def _parse_date(self, date):
> +        """return a datetime.date instance"""
> +        assert isinstance(date, basestring)
> +        dt = datetime.datetime.strptime(date, '%Y-%m-%d')
> +        return datetime.date(dt.year, dt.month, dt.day)

Not sure. What if someone inputs '2011-02-32'?

Besides, I found a more efficient way to turn the datetime.dateime into datetime.date.

::: buildapi/model/constructicon.py
@@ +17,5 @@
> +                     bsp.c.property_value.contains(value)
> +                     )))
> +    #print_query(q)
> +    return set(x[0] for x in q.execute())
> +

With indexes I found them to be quite OK on my MacBookPro. Averaging around 0.7 seconds per buildid. 

The solution beyond indexes would be a "view" (does MySQL have views?) or some sort of stored procedure that moves all properties out of that table where the name == 'buildid' such that we just need to do an equal operator on the buildid value itself. That would turn our average of 0.7 seconds into 0.07 seconds or better. 

But let's start with indexing it and monitor the pain.

@@ +107,5 @@
> +    if project:
> +        url += 'projects/%s' % project
> +    else:
> +        url += '%s' % branch
> +    return url + '/rev/%s' % revision

Ok. Remember, I'm in the deep end on this. So should it instead be:

def makeRevisionLink(revision, branch, project):
    url = 'http://hg.mozilla.org/'
    if project:
        url += 'projects/%s' % project
    else:
        if branch in ('mozilla-aurora', 'mozilla-beta'):
            url += 'releases/%s' % branch
        else:
            url += '%s' % branch
    return url + '/rev/%s' % revision

I'm all yours. The above code was just to complete the first iteration.

@@ +109,5 @@
> +    else:
> +        url += '%s' % branch
> +    return url + '/rev/%s' % revision
> +
> +def getBuild(branch=None, product=None):

That is a much better name. You're right. 

Will liaise with Laura regarding the business logic of nightlies. Perhaps best treated with being optional E.g. appending "/nightlies" to the REST URL.
Component: Socorro → General
Product: Webtools → Socorro
Peter, can you tie up the loose ends here?
So, I've implemented all the code changes as per the last piece of feedback. Now I have code committed but I don't know where to push it (a git fan-boy in hg land). 

It's arguable if the code is any good from a use point-of-view. As in, we don't know if we're actually going to use it at all. So, if I can at least push the code somewhere I can close this bug.
If whoever decides to deploy this code; it's in the patch at least.
Attachment #541732 - Attachment is obsolete: true
Closing since the code is in.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Is this publicly accessible somewhere ? It doesn't appear to be running on the RelEng buildapi instance.
Did this get deployed?
I don't think it ever got landed or deployed
catlee: do you think we could get it landed and deployed?
nm, peter tells me he thinks the code doesn't work right.  Reopening
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
I think the problem was that the only way to find buildIDs was to parse JSON found in a blob of a column two or three nasty JOINs away. 

I don't what else to do with the code at this point. 

:catlee ?
Flags: needinfo?(catlee)
ccing a few people: we're going to discuss how to do this tomorrow at the work week. I expect the bug to morph a bit but the goal is roughly the same.
I think the task cluster index will handle most of what we want here.
Flags: needinfo?(catlee)
tc index or buildhub should, yes
Status: REOPENED → RESOLVED
Closed: 12 years ago → 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: