456373 - create a makefile target for packaging a source tarball

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Robert Kaiser]

For release automation, it would be nice to have a makefile target that packages a source tarball of what we have in $topsrcdir with a standardized name in the style of other packages.

This should be implemented to work correctly on both comm-central and mozilla-central, I guess packager.mk with a call from build.mk like the packages would probably be best there. The name of the target should be something like "make package-source".

Ted Mielczarek and Ben Hearsum agreed with that basic strategy on IRC, I'll implement it soon.

Ted also mentioned it should probably be smart and strip .hg so not ship hundreds of megs of history with every source tarball.

Comment 1

[Benjamin Smedberg]

I'm not sure we should actually tar up topsrcdir... we still have some bugs where the srcdir can be contaminated during a build. Rather, I think the target should do an `hg archive` in a scratch directory and tar that.

Comment 2

[Robert Kaiser]

A hg archive would badly fail for comm-central, I guess :(

Comment 3

[Benjamin Smedberg]

Why? you know what repositories are involved, and you can archive each one into the same scratch directory.

Comment 4

[Robert Kaiser]

Hrm, it's a mashup of hg repos and cvs modules, and it would mean duplicating the logic we already have in client.py to pull those together correctly, which doesn't sound ideal.
Additionally, did we do something else than just packaging up $topsrcdir for cvs-based releases? SeaMonkey 1.1.x does that and I never heard complaints...

Comment 5

[Benjamin Smedberg]

Can we just have client.py do the tarball?

Comment 6

[Robert Kaiser]

client.py doesn't exist for Firefox any more, does it?
Additionally, client.py doesn't know about package name and path as that should be bundled in package-name.mk for everything, so we probably would need a makefile invoking it with those as parameters or so.

I can image that it might make sense, if we want to make it really clean, to have us pull the source into a new directory, under the objdir somewhere, with pulling everything needed via client.py and then package up the result of that. We'd have guaranteed pristine source there. Of course, that also means that we don't have the generated configure file(s) there.

Alternatively, we could actually use the normal srcdir and just run the source tarball generation before actually doing the builds.

Comment 7

[Robert Kaiser]

I think it's probably best if we create the makefile target to pack up a MOZ_PKG_SRCDIR or such, which can be set to any directory as an env var, so we can easily go and set it to a clean checkout we're doing. I'll default it to the $topsrcdir if the var hasn't been set so it always succeeds.

Comment 8

[Robert Kaiser]

Attachment: mozilla patch, v1

This patch is actually pretty simple, the build.mk addition is just for making it available from the toplevel, using the fact that packager.mk is included by browser/installer/Makefile.in - the same way, we make it flawlessly work for comm-central builds, using _their_ $topsrcdir by default (unless we can point MOZ_PKG_SRCDIR to a different directory where we pulled a clean copy of the source).
This also takes care of excluding .hg* and other vcs directories (--exclude-vcs takes care of CVS/ directories but not about .hg* on every currently used tar version).
The nice thing with this patch is also that people can easily do a tarball of their current development tree, even if it might contain patches, extensions pulled from elsewhere or such stuff.

For the release harness, we can make our buildbot stuff pull another clean tree somewhere and hand that one in as MOZ_PKG_SRCDIR if we fear that the tree we built from might be polluted somehow.

Comment 9

[(not currently active) Ted Mielczarek]

Comment on attachment 341950 [details] [diff] [review]
mozilla patch, v1

+CREATE_SOURCE_TAR = $(TAR) -c --owner=0 --group=0 --numeric-owner \
+  --mode="go-w" --exclude=".hg*" --exclude-vcs -f

Is this all standard tar, or GNU tar? Just worried that this will break on Mac or other BSD systems.

+	(cd $(MOZ_PKG_SRCDIR) && $(CREATE_SOURCE_TAR) - .) | bzip2 -vf > $(DIST)/$(PKG_SRCPACK_PATH)$(PKG_SRCPACK_BASENAME).tar.bz2

Is there a particular reason you need to pipe tar into bz2 here, or could you make this a single invocation of tar?

Comment 10

[Robert Kaiser]

(In reply to comment #9)
> (From update of attachment 341950 [details] [diff] [review])
> +CREATE_SOURCE_TAR = $(TAR) -c --owner=0 --group=0 --numeric-owner \
> +  --mode="go-w" --exclude=".hg*" --exclude-vcs -f
> 
> Is this all standard tar, or GNU tar? Just worried that this will break on Mac
> or other BSD systems.

Hmm, interesting objection ;-)
I believe we'll probably only do tarballs for releases on one box, which probably will be Linux. Still, it's good to be able to create source tarballs in other places, probably even including developer machines.
The tar I see on our Mac Tiger machines is actually GNU tar as well, but it's missing the --exclude-vcs option. I tested on the commandline that the following command does what we want (excluding hg and cvs special files/dirs):

+CREATE_SOURCE_TAR = $(TAR) -c --owner=0 --group=0 --numeric-owner \
+  --mode="go-w" --exclude=".hg*" --exclude="CVS" --exclude=".cvs*" -f

> +    (cd $(MOZ_PKG_SRCDIR) && $(CREATE_SOURCE_TAR) - .) | bzip2 -vf >
> $(DIST)/$(PKG_SRCPACK_PATH)$(PKG_SRCPACK_BASENAME).tar.bz2
> 
> Is there a particular reason you need to pipe tar into bz2 here, or could you
> make this a single invocation of tar?

I just copied what we are doing for normal packages, but I've wondered about the same thing myself when looking at what we're doing there - I just figured it's best to do it the same way in all places where we're packaging up stuff in tar.bz2 format.

Comment 11

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 341950 [details] [diff] [review]
mozilla patch, v1

a=beltzner (though this is NPOTB and doesn't need explicit approval)

Comment 12

[Robert Kaiser]

Pushed as http://hg.mozilla.org/mozilla-central/rev/c0e944ed5043 - this will still need a small comm-central patch but I'll do that in a followup.