467310 - Run 1.9 staging before 3.0.7 to test recent changes

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We've moved several parts of Bootstrap out to stand-alone scripts, so it would be prudent to do an end-to-end test before starting on the next round of security releases.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Just after I hit submit on this bug I thought to go check the start date for 2.0.0.19 and 3.0.5. Turns out that's tomorrow (yay me), so there isn't time for any test runs. Also, 2.0.0.19 is scheduled to be the last 2.0.0.x release, so it doesn't make much sense to take any release automation changes for it. So we should stick with RELEASE_AUTOMATION_M11 for now, and do a staging run with all the new stuff before 3.0.6 comes along.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Stop checking for gu-IN

I already had this patch done, so here it is. This should make mac's update_verify go green - gu-IN only shipped on mac for b2 thru b5, so it doesn't make much sense to check that updates work.


Other notes for later:
* we should adjust update verify to not FAIL on softokn3.chk and freebl3.chk being different (bug 404340)
* need to sync up hg.mozilla.org/users/stage-ffxbld/tools at the same time as updating the cvs mirrors

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Chris is going to do 3.0.6 and kindly volunteered to do this test. Thanks!

Comment 4

[Chris Cooper [:coop] (he/him)]

(In reply to comment #2)
> * we should adjust update verify to not FAIL on softokn3.chk and freebl3.chk
> being different (bug 404340)

Is there anything we can put in moz19-firefox-win32.cfg to accomplish this? I don't remember the verify.sh code working that way, but it's not fresh in my mind.

> * need to sync up hg.mozilla.org/users/stage-ffxbld/tools at the same time as
> updating the cvs mirrors

Not sure what you mean...sync up from internal CVS? I need some more interpretation here.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #4)
> (In reply to comment #2)
> > * we should adjust update verify to not FAIL on softokn3.chk and freebl3.chk
> > being different (bug 404340)
> Is there anything we can put in moz19-firefox-win32.cfg to accomplish this? I
> don't remember the verify.sh code working that way, but it's not fresh in my
> mind.

Don't think we can do anything in the update verify config. I was meaning 'diff -x softokn3.chk -x freebl3.chk' at
 http://mxr.mozilla.org/mozilla/source/testing/release/common/check_updates.sh#60

Incidentally, bug 470811 will give us an UPDATE_PACKAGING_R7, so that we always replace these two files (rather than try to patch and fail miserably, as been hitting win32 nightlies for a while).

> > * need to sync up hg.mozilla.org/users/stage-ffxbld/tools at the same time > > as updating the cvs mirrors
> Not sure what you mean...sync up from internal CVS? I need some more
> interpretation here.

It's this line - 
http://mxr.mozilla.org/mozilla/source/tools/release/configs/fx-moz19-staging-bootstrap.cfg#87
ie we're using a copy of the stuff we split out to hg:build/tools for staging runs. Mostly pull, but looks like update verify configs are bumped and pushed them back to hg. Ben may have updated stage-ffxbld copy for his recent 3.1b3 test run, but should probably also sync from CVS to hg (to pick up history for the releases we did since landing in hg).

Lots of other things to update in this file (version, pullDate, etc; useBetaChannel should be 1).

Comment 6

[Chris Cooper [:coop] (he/him)]

(In reply to comment #5) 
> Incidentally, bug 470811 will give us an UPDATE_PACKAGING_R7, so that we always
> replace these two files (rather than try to patch and fail miserably, as been
> hitting win32 nightlies for a while).

What's the likelihood of bug 470811 landing before builds start for 3.0.6?
 
> It's this line - 
> http://mxr.mozilla.org/mozilla/source/tools/release/configs/fx-moz19-staging-bootstrap.cfg#87
> ie we're using a copy of the stuff we split out to hg:build/tools for staging
> runs. Mostly pull, but looks like update verify configs are bumped and pushed
> them back to hg. Ben may have updated stage-ffxbld copy for his recent 3.1b3
> test run, but should probably also sync from CVS to hg (to pick up history for
> the releases we did since landing in hg).

OK, I'll try to merge in any changes from CVS before I fire this up (likely tomorrow at this point).
 
> Lots of other things to update in this file (version, pullDate, etc;
> useBetaChannel should be 1).

Having never done a staging run of this before, what's the best testing procedure here? Should I use the same vars for 3.0.5 (with updated tool versions) or should I do a bogus 3.0.6 release?

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #6)
> What's the likelihood of bug 470811 landing before builds start for 3.0.6?

I don't think we need it for 3.0.6 given we haven't had any issues with windows partials failing there, and I need to figure out how we're going to manage CVS + 2x hg repos, so not likely.

> > Lots of other things to update in this file (version, pullDate, etc;
> > useBetaChannel should be 1).
> 
> Having never done a staging run of this before, what's the best testing
> procedure here? Should I use the same vars for 3.0.5 (with updated tool
> versions) or should I do a bogus 3.0.6 release?

Normally we do a dummy 3.0.6 release to get as close to we can. So the basic idea is to update the cvsmirror.clean on staging-1.9-master (looks you've done this), then choose pullDates a bit before to your update. Happy to review your changes to the confg file.

Comment 8

[Chris Cooper [:coop] (he/him)]

Attachment: Changes to Bootstrap config for staging

Comment 9

[Chris Cooper [:coop] (he/him)]

I've been trying to get this running all weekend. I think I finally found the root causes of some of the failures I was seeing.

First, there was a symlink in place /builds/cvsroot -> /builds/cvsmirror.clean, so the cvsmirror step was putting stuff back into the same directory structure. This was causing breakage until I removed the symlink.

Second, we were running out of space on the windows slave. This was causing the Repack part of the win32 build step to die, despite the problem actually occurring in the previous step when l10n build creation was silently failing due to the lack of space. Need to file a bug for this.

I'm currently re-running with dummy steps for everything prior to the windows build step, and with the TinderConfig portion of the windows builds step set to not haltOnFailure (tagging has already occurred).

Comment 10

[Chris Cooper [:coop] (he/him)]

We just got the cutoffs for 3.0.6, so I wasn't able to get this done in time. :/ 

I'll keep running it, and get versions tagged for the next release at least.

Comment 11

[Chris Cooper [:coop] (he/him)]

Nick: do you have any insight into the changes that I'm testing here? 

I see bug 457468 as the only blocker, and that seems to involve changes to updates only. Sadly, my staging runs haven't made it that far yet. The win32 build step keeps timing out trying to download builds, e.g.:

http://staging-1.9-master.build.mozilla.org:8810/builders/win32_build/builds/7/steps/shell_10/logs/stdio

Duplicating the same steps (downloading builds) by hand works fine though. :/

Comment 12

[Chris Cooper [:coop] (he/him)]

nagios reported socket timeouts today when connecting to fx-win32-1.9-slave1 today, which is some corroboration at least.

Comment 13

[Chris Cooper [:coop] (he/him)]

Nick: we finally made it to the update step, but the patcher-config-bump.pl step is failing. Here's the log:

http://staging-1.9-master.build.mozilla.org:8810/builders/update/builds/4/steps/shell_9/logs/stdio

Is this related to the changes in bug 457468 that I'm meant to be testing here?

Comment 14

[[e-mail] (:bhearsum)]

(In reply to comment #13)
> Nick: we finally made it to the update step, but the patcher-config-bump.pl
> step is failing. Here's the log:
> 
> http://staging-1.9-master.build.mozilla.org:8810/builders/update/builds/4/steps/shell_9/logs/stdio
> 
> Is this related to the changes in bug 457468 that I'm meant to be testing here?

It's probably more related to the bump script itself...I'll look into this tomorrow.

Comment 15

[[e-mail] (:bhearsum)]

I already mentioned this to coop, but for posterity, it looks like the staging CVS mirror already has a 3.0.6 patcher config file checked in. We can't yet re-bump patcher configs, so it will need to be reverted to 3.0.5 before the patcher config bump will work.

bug 466999 is related.

Comment 16

[Chris Cooper [:coop] (he/him)]

Attachment: Convert . to _ in tag name

Finally made it to update_verify and looks like we hit a legit bug:

http://staging-1.9-master.build.mozilla.org:8810/builders/linux_update_verify/builds/0/steps/shell_8/logs/stdio

This patch fixes up the tag name we try to pull.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 359126 [details] [diff] [review]
Convert . to _ in tag name

>+    $oldTagVersion +~ s/\./_/g;

You want =~ here, right ? r+ with that.

Comment 18

[Chris Cooper [:coop] (he/him)]

Comment on attachment 359126 [details] [diff] [review]
Convert . to _ in tag name

(In reply to comment #17)
> You want =~ here, right ? r+ with that.

Fixed.

Comment 19

[Chris Cooper [:coop] (he/him)]

I hit a bunch of cvs-related errors (missing tags) in the later steps. It turns out when I recreated the initial cvsmirror, I did so with the bootstrap.cfg for 3.0.5 *before* the staging config for 3.0.6 landed (to save some time, natch). However, this meant that the tags for the 3.0.5 release got stripped/untagged, and those tags are necessary for verification 3.0.5->3.0.6 later.

I recreated the cvsmirror using the 3.0.6 config and started a fresh run late last night. It's currently repacking on win32.

Comment 20

[Chris Cooper [:coop] (he/him)]

update_verify is failing to check the version bump back in (ssl required):

http://staging-1.9-master.build.mozilla.org:8810/builders/linux_update_verify/builds/2/steps/shell_8/logs/stdio

Expected? Or is this new?

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Looks like it needs to push to ssh://hg.m.o rather than https://hg.m.o

Comment 22

[Chris Cooper [:coop] (he/him)]

Attachment: Replace http(s) with ssh when doing an hg push

Didn't do an actually push, but did test that an hg clone works via ssh with the same ssh arguments as used for a push.

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 359605 [details] [diff] [review]
Replace http(s) with ssh when doing an hg push

Looks like we already have a GetPushRepo function, and use it, just fail to plug in the right var to the actual command:

http://mxr.mozilla.org/seamonkey/source/tools/release/Bootstrap/Step.pm#335

Comment 24

[Chris Cooper [:coop] (he/him)]

Attachment: Use pushRepo

Comment 25

[Chris Cooper [:coop] (he/him)]

I *think* I'm ready to declare an end-to-end run complete now. 

The update_verify step ran on all three platforms. We legit passed on linux, win32 failed on the two files we know about (freebl3.chk and softokn3.chk), and Mac failed on gu-IN.

Logs for failures:
http://staging-1.9-master.build.mozilla.org:8810/builders/win32_update_verify/builds/6/steps/shell_8/logs/stdio
http://staging-1.9-master.build.mozilla.org:8810/builders/macosx_update_verify/builds/4/steps/shell_8/logs/stdio

Nick: The Mac failure is a bit puzzling. Wasn't attachment 350713 [details] [diff] [review] meant to fix that?

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

Looks like I checked the gu-IN change into CVS but not Hg, so it probably helped for 3.0.6 but not here. I guess we'll need to sync from CVS to Hg anyway, once we're ready to make the switch. Maybe after 3.0.6 ?

Comment 27

[Chris Cooper [:coop] (he/him)]

Eep, this fell off my radar.

Ben: when you tagged M13 for releases automation, did you do a sync between CVS->Hg?

Comment 28

[[e-mail] (:bhearsum)]

I didn't do a sync, no. Good news is we're not yet done Repack for 3.0.7, we can probably sync over and update the tags before we hit an issue :).

Comment 29

[Chris Cooper [:coop] (he/him)]

Attachment: Stop checking for gu-IN (hg patch)

Comment 30

[Chris Cooper [:coop] (he/him)]

Comment on attachment 362761 [details] [diff] [review]
Stop checking for gu-IN (hg patch)

changeset:   226:e5a6060364bb