412454 - the Bootstrap Makefile should be more generic

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[[e-mail] (:bhearsum)]

Right now it _only_ works for 1.8 staging and requires bumping whenever the version changes. It would be nice to just be able to say "make clean_stage 3.0b3", or something like that. We could probably do something clever in the BuildStep that calls it so it would read the version string in from bootstrap.cfg to get rid of the bumping annoyance.

Comment 1

[[e-mail] (:bhearsum)]

Attachment: support different versions, products in Bootstrap Makefile

This patch to the Makefile will remove all hardcoded versions, products, and tags. It now reads in all of this information from bootstrap.cfg. I thought about having it passed in from Buildbot, but after looking at it I really don't see what difference it would make -- other than making the master.cfg uglier ;). We depend on bootstrap.cfg for every other part of Bootstrap, so it shouldn't hurt to depend on it here, too.

I've done a quick local test here and 'make stage' works as intended.

The only thing I'm unsure about here is in the 'tag -d' part. It's obviously necessary to clobber the tag for the current rc. But what happens in cases where the current rc is > 1? Should we be clobbering every tag for a specific version?

In any case, a master.cfg patch is upcoming...

Comment 2

[[e-mail] (:bhearsum)]

Attachment: update the prestage factory to download bootstrap.cfg, use makefile

At some point we should move the 'sync cvsmirror' logic to the Makefile. It's not dependent on bumping, though, so it's fine for now.

Comment 3

[Robert Helmer [:rhelmer]]

(In reply to comment #1)
> Created an attachment (id=297365) [details]
> support different versions, products in Bootstrap Makefile
> 
> This patch to the Makefile will remove all hardcoded versions, products, and
> tags. It now reads in all of this information from bootstrap.cfg. I thought
> about having it passed in from Buildbot, but after looking at it I really don't
> see what difference it would make -- other than making the master.cfg uglier
> ;). We depend on bootstrap.cfg for every other part of Bootstrap, so it
> shouldn't hurt to depend on it here, too.
> 
> I've done a quick local test here and 'make stage' works as intended.
> 
> The only thing I'm unsure about here is in the 'tag -d' part. It's obviously
> necessary to clobber the tag for the current rc. But what happens in cases
> where the current rc is > 1? Should we be clobbering every tag for a specific
> version?


Yeah, we have just hardcoded this in the Makefile, as-needed. Couldn't you use a for loop in the makefile to support this?

Comment 4

[[e-mail] (:bhearsum)]

(In reply to comment #3)
> (In reply to comment #1)
> > Created an attachment (id=297365) [details] [details]
> > support different versions, products in Bootstrap Makefile
> > 
> > This patch to the Makefile will remove all hardcoded versions, products, and
> > tags. It now reads in all of this information from bootstrap.cfg. I thought
> > about having it passed in from Buildbot, but after looking at it I really don't
> > see what difference it would make -- other than making the master.cfg uglier
> > ;). We depend on bootstrap.cfg for every other part of Bootstrap, so it
> > shouldn't hurt to depend on it here, too.
> > 
> > I've done a quick local test here and 'make stage' works as intended.
> > 
> > The only thing I'm unsure about here is in the 'tag -d' part. It's obviously
> > necessary to clobber the tag for the current rc. But what happens in cases
> > where the current rc is > 1? Should we be clobbering every tag for a specific
> > version?
> 
> 
> Yeah, we have just hardcoded this in the Makefile, as-needed. Couldn't you use
> a for loop in the makefile to support this?
> 

Yeah, I'm sure I can. I didn't want to look at Makefile looping unless it was necessary though...;-).

Comment 5

[[e-mail] (:bhearsum)]

Attachment: make sure all product tags are clobber (rc1 -> rcN)

Note the definition change of PRODUCT_RC, from, eg, 'rc3' -> '3'.

This is really ugly :(.

Comment 6

[[e-mail] (:bhearsum)]

Attachment: fix typo, use ftp:// instead of http:// for ftp.m.o

I've switched this to ftp://ftp.mozilla.org/... since there's http forwards for 3.0b2. They'll be one in place for 3.0b3 too, I bet, so better to catch this now.

Comment 7

[[e-mail] (:bhearsum)]

Attachment: [checked in] checkout bootstrap on 1.9 staging

Forgot to checkout Bootstrap on 1.9 staging.

Comment 8

[[e-mail] (:bhearsum)]

I've got a run going on staging-trunk-automation right now, the prestage factory has completed, so I'm reasonably sure that things are OK.

Comment 9

[Robert Helmer [:rhelmer]]

Comment on attachment 297407 [details] [diff] [review]
fix typo, use ftp:// instead of http:// for ftp.m.o

should tke out "-e robots=off" too.

Comment 10

[[e-mail] (:bhearsum)]

Attachment: remove robots option

Comment 11

[[e-mail] (:bhearsum)]

Attachment: [checked in] add robots option back into http download

This patch appears to have worked fine on staging-trunk-automation, I think it's good for check-in. There was a failure on win32 updateverify, but I believe it is unrelated.

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 297564 [details] [diff] [review]
[checked in] add robots option back into http download

Checking in Makefile;
/cvsroot/mozilla/tools/release/Makefile,v  <--  Makefile
new revision: 1.20; previous revision: 1.19
done

Comment 13

[[e-mail] (:bhearsum)]

Comment on attachment 297409 [details] [diff] [review]
[checked in] checkout bootstrap on 1.9 staging

Checking in staging/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging/master.cfg,v  <--  master.cfg
new revision: 1.17; previous revision: 1.16
done
Checking in staging-1.9/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v  <--  master.cfg
new revision: 1.14; previous revision: 1.13
done

Comment 14

[Robert Helmer [:rhelmer]]

Comment on attachment 297564 [details] [diff] [review]
[checked in] add robots option back into http download

>+	# download old builds, for l10n verify
>+	cd /data && wget -nv --cut-dirs=3 -np -r ftp://ftp.mozilla.org/pub/mozilla.org/${PRODUCT}/nightly/${OLD_VERSION}-candidates/rc1/ && mv ftp.mozilla.org/nightly/* /home/ftp/pub/${PRODUCT}/nightly/ && rm -rfv ftp.mozilla.org

Actually, for FTP the URL needs to have * at the end (just testing on staging-1.9-master) or updateverify fails, because *info.txt files are not present (should add that updateverify depends on this too in the comment).

This is needed for l10nverify, but the verify steps do not do a clean checkout, so it succeeded when really it should have failed. I am going to file a separate bug on asserting that the verify dirs are empty before beginning.

Comment 15

[[e-mail] (:bhearsum)]

Attachment: [checked in] fix the ftp URL

Comment 16

[[e-mail] (:bhearsum)]

Comment on attachment 298259 [details] [diff] [review]
[checked in] fix the ftp URL

Checking in Makefile;
/cvsroot/mozilla/tools/release/Makefile,v  <--  Makefile
new revision: 1.21; previous revision: 1.20
done