637551 - Reduce disk usage requirements for signing

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We have a ~100G partition for signing releases. It's easy to use that up when we have three active Firefox releases with multiple builds - we need about 23G for each Fx4 build, 18G for each Fx3.6 build. We should be able to use that space more efficiently which (at least) lets us hang on to all the builds for a release, and possibly the previous release, to allow for problems being discovered.

The main thing we can do is not copy the non-win32 dirs between unsigned-buildN/ and signed-buildN/. I've checked on the production box and we can hardlink all those files by making a call like this
 rsync -av --exclude '*unsigned*' --link-dest=`pwd`/unsigned-build1 \
   unsigned-build1/ signed-build1/

The --link-dest makes rsync do the magic of hardlinking as long as you give it an absolute path, and only takes a second instead of a few minutes. It does funny things to the disk usage as reported by du, as the space is assigned to the signed-build1 instead of unsigned-build1, but that's probably not a big deal.

We'd need to modify the signing Makefile (download target) and do a full signing run.

Comment 1

[Chris AtLee [:catlee]]

(In reply to comment #0)
> We have a ~100G partition for signing releases. It's easy to use that up when
> we have three active Firefox releases with multiple builds - we need about 23G
> for each Fx4 build, 18G for each Fx3.6 build. We should be able to use that
> space more efficiently which (at least) lets us hang on to all the builds for a
> release, and possibly the previous release, to allow for problems being
> discovered.
> 
> The main thing we can do is not copy the non-win32 dirs between
> unsigned-buildN/ and signed-buildN/. I've checked on the production box and we
> can hardlink all those files by making a call like this
>  rsync -av --exclude '*unsigned*' --link-dest=`pwd`/unsigned-build1 \
>    unsigned-build1/ signed-build1/
> 
> The --link-dest makes rsync do the magic of hardlinking as long as you give it
> an absolute path, and only takes a second instead of a few minutes. It does
> funny things to the disk usage as reported by du, as the space is assigned to
> the signed-build1 instead of unsigned-build1, but that's probably not a big
> deal.
> 
> We'd need to modify the signing Makefile (download target) and do a full
> signing run.

Wow, that works on windows?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

NTFS supports hardlinks, and Cygwin knows how to tickle the right API, so yay!

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

This style of signing is going away.