Closed Bug 474610 Opened 15 years ago Closed 14 years ago

Hourly/Nightly builds should have some way to see which changeset was used

Categories

(Firefox Build System :: General, defect, P3)

defect

Tracking

(status1.9.2 .2-fixed, status1.9.1 .9-fixed)

VERIFIED FIXED
mozilla1.9.3a2
Tracking Status
status1.9.2 --- .2-fixed
status1.9.1 --- .9-fixed

People

(Reporter: johnath, Assigned: ted)

References

Details

(Keywords: verified1.9.1, verified1.9.2)

Attachments

(2 files, 1 obsolete file)

We drop the builds from various tinderbox machines into directories like:

http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1232535077/

which include zips/installers/whatever depending on platform.

These are valuable for finding regression ranges, and we have people like Jim Jeffrey who, when we find a bug, will do a binary search through them to track down the problem, which is an immensely valuable thing for them to do.

Unfortunately, right now they can't see which changeset a given build is using until they install it.  It would ease their life and get us faster regression range results if there was a way to tell by inspection whether a given hourly matters.

One thing I proposed to Ben was that we just drop a 0-byte file in the directories with the changeset ID as a name.  Of course, we could just change the naming structure of the builds or the directories, but I suspect there are parts of our infra that make assumptions about those names, which is why I suggested the extra file drop - it seems less likely that we make an assumption that would break.

How can we make their lives easier, given that our time investment here gets greatly multiplied by their efforts.
This shouldn't be too tough to do. Here's what would have to done:
* MercurialBuildFactory will need a step added that creates this file
* MozillaStageUpload needs to know how to upload it.

I'm tempted to block this on https://bugzilla.mozilla.org/show_bug.cgi?id=474297, because MozillaStageUpload doesn't have a way to say 'upload this random file too', but I may not get to that bug for a week or two.

Here's the relevant code, in any case:
http://mxr.mozilla.org/mozilla/source/tools/buildbotcustom/process/factory.py#627
http://mxr.mozilla.org/mozilla/source/tools/buildbotcustom/steps/transfer.py#7
I'd be curious on the actual question, with a shadow of my build-database hat on, as this bug request is somewhat tainted by the proposed solution.

In particular, do you want to see the source stamp for a build, or the build for a source stamp?
(In reply to comment #0)
> We drop the builds from various tinderbox machines into directories like:
> 
> http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1232535077/
> 
> One thing I proposed to Ben was that we just drop a 0-byte file in the
> directories with the changeset ID as a name.  
Yep, that would work - nice suggestion. 

We will need to make sure this file doesn't accidentally get bundled up and shipped to users, though!


> Of course, we could just change
> the naming structure of the builds or the directories, but I suspect there are
> parts of our infra that make assumptions about those names, 
Yep, exactly. We did investigate using changesets for dirname, but we needed something that was chronological, and also something that worked equally with the cvs-based systems that we're still running.

> How can we make their lives easier, given that our time investment here gets
> greatly multiplied by their efforts.
Triaging to Future until someone has time for this, and to check the installers. If there's other suggestions that would help make things more efficient, please file separate bugs for them too, ok?
Component: Release Engineering → Release Engineering: Future
Came up in conversation today with Mike, cc'ng him in
Why not ship the file to users?  It's 0 byte!

This would help our regression-range army of awesome work more effectively, which makes me smile.  Don't you want to see me smile? :-(
We do have the changesets in application.ini, and in the case of fennec, the changeset of gecko on platform.ini, too. We're using those for about:buildconfig, and increasingly for l10n builds. Note, about:buildconfig only returns the app revision, IIRC, not the toolkit one.

This bug is about having the changeset of the build without having to actually download it.
I don't know how its being done, but someone is creating an hourly archive available here, that uses zip builds, but some of us testers prefer to install via the installer... This shows the changeset ID and build-times 

http://hourly-archive.localgho.st/hourly-archive2/
As given in my duped bug we should also do that for nightly builds. It will help us a lot for spotting the changeset id of a build when doing a regression test. Sometimes you will get a day range from the reporter and I do not want to always install/unzip the builds. Updating summary.
Summary: Hourly builds should have some way to see which changeset was used → Hourly/Nightly builds should have some way to see which changeset was used
Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.
Component: Release Engineering: Future → Release Engineering
Priority: -- → P3
Assignee: nobody → ted.mielczarek
Status: NEW → ASSIGNED
Component: Release Engineering → Build Config
Product: mozilla.org → Core
QA Contact: release → build-config
Version: other → Trunk
This patch makes us generate a $(package).txt file (like firefox-3.7a2pre.en-US.linux-x86_64.txt) alongside the build package when "make package" is invoked, and uploads it along with the build with "make upload". The text file contains just "$BUILDID $CHANGESET".

From reading post-upload.py, it looks like just uploading this file should make it wind up in all the right directories on FTP.
Attachment #428739 - Flags: review?(bhearsum)
Comment on attachment 428739 [details] [diff] [review]
generate a text file alongside application packages that includes the build ID and source changeset, and upload it with the build package
[Checkin: Comment 13 & 31]

Looks good to me. We should probably unify the release automation with this at some point -- we generate txt files for each build through the Buildbot factory currently. As it stands now, we'll have to throw *these* ones away. I filed bug 548549 on this follow-up.
Attachment #428739 - Flags: review?(bhearsum) → review+
Pushed to m-c:
http://hg.mozilla.org/mozilla-central/rev/42c7d789364e

These should start showing up in hourlies after my push, and nightlies starting tomorrow. (Assuming my reading of post-upload.py is correct.)
Status: ASSIGNED → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Flags: in-testsuite-
Target Milestone: --- → mozilla1.9.3a2
Bv1-CC, keeping m-1.9.2 support c-c has.
Attachment #428963 - Attachment is obsolete: true
Attachment #428972 - Flags: review?(bugspam.Callek)
Attachment #428963 - Flags: review?(bugspam.Callek)
Depends on: 548788
Comment on attachment 428972 [details] [diff] [review]
(Bv2-CC) Copy (the useful part of) it to comm-central (1.9.2+)
[Checkin: See comment 24+28 & 32]

I suspect we may want a slightly different solution for c-c here (package a "list" of changesets perhaps)
Attachment #428972 - Flags: review?(bugspam.Callek) → review?(kairo)
(In reply to comment #17)

Probably, but at least it's a start in the meantime.
Then we could file a follow-up bug.
Attachment #428972 - Flags: review?(kairo) → review+
Ria, Jim - do these do the trick for you when finding regression windows?

(Thanks, Ted!)
(In reply to comment #19)
> Ria, Jim - do these do the trick for you when finding regression windows?
> 
> (Thanks, Ted!)

Yes, its going to work out very well.  So far not had to go 'hunting', but it will be really nice down the road, at least from my standpoint.
Thanks again to Ted.
(In reply to comment #19)
> Ria, Jim - do these do the trick for you when finding regression windows?
> 
> (Thanks, Ted!)

Yes, good idea, thanks for the notification.
This is pretty helpful and looks fantastic. Just checked the mozilla-central folder for nightly builds.

Can we get this for 1.9.2 and 1.9.1 builds too?
Status: RESOLVED → VERIFIED
Comment on attachment 428739 [details] [diff] [review]
generate a text file alongside application packages that includes the build ID and source changeset, and upload it with the build package
[Checkin: Comment 13 & 31]

Can't hurt to ask.

Drivers: this is a small build-config only change that helps nightly testers by providing a text file with Build ID and changeset alongside each build.
Attachment #428739 - Flags: approval1.9.2.2?
Attachment #428739 - Flags: approval1.9.1.9?
Comment on attachment 428972 [details] [diff] [review]
(Bv2-CC) Copy (the useful part of) it to comm-central (1.9.2+)
[Checkin: See comment 24+28 & 32]


http://hg.mozilla.org/comm-central/rev/83078dc64965
Attachment #428972 - Attachment description: (Bv2-CC) Copy (the useful part of) it to comm-central (1.9.2+) → (Bv2-CC) Copy (the useful part of) it to comm-central (1.9.2+) [Checkin: Comment 24]
Attachment #428739 - Attachment description: generate a text file alongside application packages that includes the build ID and source changeset, and upload it with the build package → generate a text file alongside application packages that includes the build ID and source changeset, and upload it with the build package [Checkin: Comment 13]
Depends on: 478221
(In reply to comment #25)
> (Cv1-CC) Restore removed line on "1.9.3" ftb

Ftr, not all c-c builds were affected: odd!?
(In reply to comment #17)
> I suspect we may want a slightly different solution for c-c here (package a
> "list" of changesets perhaps)

I eventually filed bug 549958.
Attachment #428972 - Attachment description: (Bv2-CC) Copy (the useful part of) it to comm-central (1.9.2+) [Checkin: Comment 24] → (Bv2-CC) Copy (the useful part of) it to comm-central (1.9.2+) [Checkin: See comment 24+28]
Comment on attachment 428972 [details] [diff] [review]
(Bv2-CC) Copy (the useful part of) it to comm-central (1.9.2+)
[Checkin: See comment 24+28 & 32]


(In reply to comment #25)
> http://hg.mozilla.org/comm-central/rev/6804e330ba06
> (Cv1-CC) Restore removed line on "1.9.3" ftb

http://hg.mozilla.org/comm-central/rev/450e89a3fdd4
Revert patch Cv1-CC after bug 496236 patch Qv1 landing, Improve a comment.
Comment on attachment 428739 [details] [diff] [review]
generate a text file alongside application packages that includes the build ID and source changeset, and upload it with the build package
[Checkin: Comment 13 & 31]

a1919,1922=beltzner
Attachment #428739 - Flags: approval1.9.2.2?
Attachment #428739 - Flags: approval1.9.2.2+
Attachment #428739 - Flags: approval1.9.1.9?
Attachment #428739 - Flags: approval1.9.1.9+
Is it possible to clone this bug also for Camino and Camino nightly builds?
Camino nightlies are available here: http://caminobrowser.org/download/releases/nightly/, http://ftp.mozilla.org//pub/mozilla.org/camino/nightly/ .
These resources are also lacking a simple text file or hint, which indicates remotely the latest BuildID and changeset of the builds.

Instead of filing a new bug for Camino, wouldn't it be smarter to clone or extend this bug also for the product Camino (and platform: MacOSX) instead of limiting it to Core?
AFAIK Camino is building off of really old branches, so any fix we do here isn't going to affect those builds. You can file a new bug on Camino if you want them to pick up a similar fix. If they moved to building off of a newer branch, it's quite likely they'd get this for free. (Which is why we fix things in the Core platform!)

Pushed to 1.9.2:
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/e99f405b50c1

Pushed to 1.9.1:
http://hg.mozilla.org/releases/mozilla-1.9.1/rev/790e999a3798
Attachment #428739 - Attachment description: generate a text file alongside application packages that includes the build ID and source changeset, and upload it with the build package [Checkin: Comment 13] → generate a text file alongside application packages that includes the build ID and source changeset, and upload it with the build package [Checkin: Comment 13 & 31]
Comment on attachment 428972 [details] [diff] [review]
(Bv2-CC) Copy (the useful part of) it to comm-central (1.9.2+)
[Checkin: See comment 24+28 & 32]


http://hg.mozilla.org/comm-central/rev/e2198795f0f7
(Cv1-CC) Update c-c after m-1.9.2 landing
Attachment #428972 - Attachment description: (Bv2-CC) Copy (the useful part of) it to comm-central (1.9.2+) [Checkin: See comment 24+28] → (Bv2-CC) Copy (the useful part of) it to comm-central (1.9.2+) [Checkin: See comment 24+28 & 32]
(In reply to comment #15)
> Just in case, would this fix obsolete the following '?=' too?

I guess it doesn't when building just these directories.
Verified for 1.9.2 and 1.9.1 by looking at the directories.
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.