Closed Bug 456373 Opened 11 years ago Closed 11 years ago

create a makefile target for packaging a source tarball


(Firefox Build System :: General, defect)

Not set


(Not tracked)



(Reporter: kairo, Assigned: kairo)




(1 file)

For release automation, it would be nice to have a makefile target that packages a source tarball of what we have in $topsrcdir with a standardized name in the style of other packages.

This should be implemented to work correctly on both comm-central and mozilla-central, I guess with a call from like the packages would probably be best there. The name of the target should be something like "make package-source".

Ted Mielczarek and Ben Hearsum agreed with that basic strategy on IRC, I'll implement it soon.

Ted also mentioned it should probably be smart and strip .hg so not ship hundreds of megs of history with every source tarball.
I'm not sure we should actually tar up topsrcdir... we still have some bugs where the srcdir can be contaminated during a build. Rather, I think the target should do an `hg archive` in a scratch directory and tar that.
A hg archive would badly fail for comm-central, I guess :(
Why? you know what repositories are involved, and you can archive each one into the same scratch directory.
Hrm, it's a mashup of hg repos and cvs modules, and it would mean duplicating the logic we already have in to pull those together correctly, which doesn't sound ideal.
Additionally, did we do something else than just packaging up $topsrcdir for cvs-based releases? SeaMonkey 1.1.x does that and I never heard complaints...
Can we just have do the tarball? doesn't exist for Firefox any more, does it?
Additionally, doesn't know about package name and path as that should be bundled in for everything, so we probably would need a makefile invoking it with those as parameters or so.

I can image that it might make sense, if we want to make it really clean, to have us pull the source into a new directory, under the objdir somewhere, with pulling everything needed via and then package up the result of that. We'd have guaranteed pristine source there. Of course, that also means that we don't have the generated configure file(s) there.

Alternatively, we could actually use the normal srcdir and just run the source tarball generation before actually doing the builds.
I think it's probably best if we create the makefile target to pack up a MOZ_PKG_SRCDIR or such, which can be set to any directory as an env var, so we can easily go and set it to a clean checkout we're doing. I'll default it to the $topsrcdir if the var hasn't been set so it always succeeds.
This patch is actually pretty simple, the addition is just for making it available from the toplevel, using the fact that is included by browser/installer/ - the same way, we make it flawlessly work for comm-central builds, using _their_ $topsrcdir by default (unless we can point MOZ_PKG_SRCDIR to a different directory where we pulled a clean copy of the source).
This also takes care of excluding .hg* and other vcs directories (--exclude-vcs takes care of CVS/ directories but not about .hg* on every currently used tar version).
The nice thing with this patch is also that people can easily do a tarball of their current development tree, even if it might contain patches, extensions pulled from elsewhere or such stuff.

For the release harness, we can make our buildbot stuff pull another clean tree somewhere and hand that one in as MOZ_PKG_SRCDIR if we fear that the tree we built from might be polluted somehow.
Attachment #341950 - Flags: review?(ted.mielczarek)
Comment on attachment 341950 [details] [diff] [review]
mozilla patch, v1

+CREATE_SOURCE_TAR = $(TAR) -c --owner=0 --group=0 --numeric-owner \
+  --mode="go-w" --exclude=".hg*" --exclude-vcs -f

Is this all standard tar, or GNU tar? Just worried that this will break on Mac or other BSD systems.


Is there a particular reason you need to pipe tar into bz2 here, or could you make this a single invocation of tar?
Attachment #341950 - Flags: review?(ted.mielczarek) → review+
(In reply to comment #9)
> (From update of attachment 341950 [details] [diff] [review])
> +CREATE_SOURCE_TAR = $(TAR) -c --owner=0 --group=0 --numeric-owner \
> +  --mode="go-w" --exclude=".hg*" --exclude-vcs -f
> Is this all standard tar, or GNU tar? Just worried that this will break on Mac
> or other BSD systems.

Hmm, interesting objection ;-)
I believe we'll probably only do tarballs for releases on one box, which probably will be Linux. Still, it's good to be able to create source tarballs in other places, probably even including developer machines.
The tar I see on our Mac Tiger machines is actually GNU tar as well, but it's missing the --exclude-vcs option. I tested on the commandline that the following command does what we want (excluding hg and cvs special files/dirs):

+CREATE_SOURCE_TAR = $(TAR) -c --owner=0 --group=0 --numeric-owner \
+  --mode="go-w" --exclude=".hg*" --exclude="CVS" --exclude=".cvs*" -f

> +    (cd $(MOZ_PKG_SRCDIR) && $(CREATE_SOURCE_TAR) - .) | bzip2 -vf >
> Is there a particular reason you need to pipe tar into bz2 here, or could you
> make this a single invocation of tar?

I just copied what we are doing for normal packages, but I've wondered about the same thing myself when looking at what we're doing there - I just figured it's best to do it the same way in all places where we're packaging up stuff in tar.bz2 format.
Attachment #341950 - Flags: approval1.9.1b2?
Target Milestone: --- → mozilla1.9.1b2
Comment on attachment 341950 [details] [diff] [review]
mozilla patch, v1

a=beltzner (though this is NPOTB and doesn't need explicit approval)
Attachment #341950 - Flags: approval1.9.1b2? → approval1.9.1b2+
Pushed as - this will still need a small comm-central patch but I'll do that in a followup.
Closed: 11 years ago
Resolution: --- → FIXED
Blocks: 464446
Blocks: 510770
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.