All users were logged out of Bugzilla on October 13th, 2018

create a makefile target for packaging a source tarball

RESOLVED FIXED in mozilla1.9.1b2

Status

RESOLVED FIXED
10 years ago
8 months ago

People

(Reporter: kairo, Assigned: kairo)

Tracking

Trunk
mozilla1.9.1b2
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

10 years ago
For release automation, it would be nice to have a makefile target that packages a source tarball of what we have in $topsrcdir with a standardized name in the style of other packages.

This should be implemented to work correctly on both comm-central and mozilla-central, I guess packager.mk with a call from build.mk like the packages would probably be best there. The name of the target should be something like "make package-source".

Ted Mielczarek and Ben Hearsum agreed with that basic strategy on IRC, I'll implement it soon.

Ted also mentioned it should probably be smart and strip .hg so not ship hundreds of megs of history with every source tarball.

Comment 1

10 years ago
I'm not sure we should actually tar up topsrcdir... we still have some bugs where the srcdir can be contaminated during a build. Rather, I think the target should do an `hg archive` in a scratch directory and tar that.
(Assignee)

Comment 2

10 years ago
A hg archive would badly fail for comm-central, I guess :(

Comment 3

10 years ago
Why? you know what repositories are involved, and you can archive each one into the same scratch directory.
(Assignee)

Comment 4

10 years ago
Hrm, it's a mashup of hg repos and cvs modules, and it would mean duplicating the logic we already have in client.py to pull those together correctly, which doesn't sound ideal.
Additionally, did we do something else than just packaging up $topsrcdir for cvs-based releases? SeaMonkey 1.1.x does that and I never heard complaints...

Comment 5

10 years ago
Can we just have client.py do the tarball?
(Assignee)

Comment 6

10 years ago
client.py doesn't exist for Firefox any more, does it?
Additionally, client.py doesn't know about package name and path as that should be bundled in package-name.mk for everything, so we probably would need a makefile invoking it with those as parameters or so.

I can image that it might make sense, if we want to make it really clean, to have us pull the source into a new directory, under the objdir somewhere, with pulling everything needed via client.py and then package up the result of that. We'd have guaranteed pristine source there. Of course, that also means that we don't have the generated configure file(s) there.

Alternatively, we could actually use the normal srcdir and just run the source tarball generation before actually doing the builds.
(Assignee)

Comment 7

10 years ago
I think it's probably best if we create the makefile target to pack up a MOZ_PKG_SRCDIR or such, which can be set to any directory as an env var, so we can easily go and set it to a clean checkout we're doing. I'll default it to the $topsrcdir if the var hasn't been set so it always succeeds.
(Assignee)

Comment 8

10 years ago
Created attachment 341950 [details] [diff] [review]
mozilla patch, v1

This patch is actually pretty simple, the build.mk addition is just for making it available from the toplevel, using the fact that packager.mk is included by browser/installer/Makefile.in - the same way, we make it flawlessly work for comm-central builds, using _their_ $topsrcdir by default (unless we can point MOZ_PKG_SRCDIR to a different directory where we pulled a clean copy of the source).
This also takes care of excluding .hg* and other vcs directories (--exclude-vcs takes care of CVS/ directories but not about .hg* on every currently used tar version).
The nice thing with this patch is also that people can easily do a tarball of their current development tree, even if it might contain patches, extensions pulled from elsewhere or such stuff.

For the release harness, we can make our buildbot stuff pull another clean tree somewhere and hand that one in as MOZ_PKG_SRCDIR if we fear that the tree we built from might be polluted somehow.
Attachment #341950 - Flags: review?(ted.mielczarek)
Comment on attachment 341950 [details] [diff] [review]
mozilla patch, v1

+CREATE_SOURCE_TAR = $(TAR) -c --owner=0 --group=0 --numeric-owner \
+  --mode="go-w" --exclude=".hg*" --exclude-vcs -f

Is this all standard tar, or GNU tar? Just worried that this will break on Mac or other BSD systems.

+	(cd $(MOZ_PKG_SRCDIR) && $(CREATE_SOURCE_TAR) - .) | bzip2 -vf > $(DIST)/$(PKG_SRCPACK_PATH)$(PKG_SRCPACK_BASENAME).tar.bz2

Is there a particular reason you need to pipe tar into bz2 here, or could you make this a single invocation of tar?
Attachment #341950 - Flags: review?(ted.mielczarek) → review+
(Assignee)

Comment 10

10 years ago
(In reply to comment #9)
> (From update of attachment 341950 [details] [diff] [review])
> +CREATE_SOURCE_TAR = $(TAR) -c --owner=0 --group=0 --numeric-owner \
> +  --mode="go-w" --exclude=".hg*" --exclude-vcs -f
> 
> Is this all standard tar, or GNU tar? Just worried that this will break on Mac
> or other BSD systems.

Hmm, interesting objection ;-)
I believe we'll probably only do tarballs for releases on one box, which probably will be Linux. Still, it's good to be able to create source tarballs in other places, probably even including developer machines.
The tar I see on our Mac Tiger machines is actually GNU tar as well, but it's missing the --exclude-vcs option. I tested on the commandline that the following command does what we want (excluding hg and cvs special files/dirs):

+CREATE_SOURCE_TAR = $(TAR) -c --owner=0 --group=0 --numeric-owner \
+  --mode="go-w" --exclude=".hg*" --exclude="CVS" --exclude=".cvs*" -f

> +    (cd $(MOZ_PKG_SRCDIR) && $(CREATE_SOURCE_TAR) - .) | bzip2 -vf >
> $(DIST)/$(PKG_SRCPACK_PATH)$(PKG_SRCPACK_BASENAME).tar.bz2
> 
> Is there a particular reason you need to pipe tar into bz2 here, or could you
> make this a single invocation of tar?

I just copied what we are doing for normal packages, but I've wondered about the same thing myself when looking at what we're doing there - I just figured it's best to do it the same way in all places where we're packaging up stuff in tar.bz2 format.
(Assignee)

Updated

10 years ago
Attachment #341950 - Flags: approval1.9.1b2?
(Assignee)

Updated

10 years ago
Target Milestone: --- → mozilla1.9.1b2
Comment on attachment 341950 [details] [diff] [review]
mozilla patch, v1

a=beltzner (though this is NPOTB and doesn't need explicit approval)
Attachment #341950 - Flags: approval1.9.1b2? → approval1.9.1b2+
(Assignee)

Comment 12

10 years ago
Pushed as http://hg.mozilla.org/mozilla-central/rev/c0e944ed5043 - this will still need a small comm-central patch but I'll do that in a followup.
Status: NEW → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → FIXED
(Assignee)

Updated

10 years ago
Blocks: 464446
(Assignee)

Updated

9 years ago
Blocks: 510770

Updated

8 months ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.