Closed Bug 467310 Opened 11 years ago Closed 11 years ago

Run 1.9 staging before 3.0.7 to test recent changes

Categories

(Release Engineering :: General, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nthomas, Assigned: coop)

References

Details

Attachments

(5 files, 1 obsolete file)

We've moved several parts of Bootstrap out to stand-alone scripts, so it would be prudent to do an end-to-end test before starting on the next round of security releases.
Status: NEW → ASSIGNED
Priority: -- → P2
Just after I hit submit on this bug I thought to go check the start date for 2.0.0.19 and 3.0.5. Turns out that's tomorrow (yay me), so there isn't time for any test runs. Also, 2.0.0.19 is scheduled to be the last 2.0.0.x release, so it doesn't make much sense to take any release automation changes for it. So we should stick with RELEASE_AUTOMATION_M11 for now, and do a staging run with all the new stuff before 3.0.6 comes along.
Priority: P2 → P3
Summary: Run 1.8 and 1.9 staging before 2.0.0.19 and 3.0.5 to test recent changes → Run 1.9 staging before 3.0.6 to test recent changes
I already had this patch done, so here it is. This should make mac's update_verify go green - gu-IN only shipped on mac for b2 thru b5, so it doesn't make much sense to check that updates work.


Other notes for later:
* we should adjust update verify to not FAIL on softokn3.chk and freebl3.chk being different (bug 404340)
* need to sync up hg.mozilla.org/users/stage-ffxbld/tools at the same time as updating the cvs mirrors
Attachment #350713 - Flags: review?(ccooper)
Attachment #350713 - Flags: review?(ccooper) → review+
Attachment #350713 - Flags: checked‑in+
Blocks: 457468
Chris is going to do 3.0.6 and kindly volunteered to do this test. Thanks!
Assignee: nthomas → ccooper
(In reply to comment #2)
> * we should adjust update verify to not FAIL on softokn3.chk and freebl3.chk
> being different (bug 404340)

Is there anything we can put in moz19-firefox-win32.cfg to accomplish this? I don't remember the verify.sh code working that way, but it's not fresh in my mind.

> * need to sync up hg.mozilla.org/users/stage-ffxbld/tools at the same time as
> updating the cvs mirrors

Not sure what you mean...sync up from internal CVS? I need some more interpretation here.
Priority: P3 → P2
(In reply to comment #4)
> (In reply to comment #2)
> > * we should adjust update verify to not FAIL on softokn3.chk and freebl3.chk
> > being different (bug 404340)
> Is there anything we can put in moz19-firefox-win32.cfg to accomplish this? I
> don't remember the verify.sh code working that way, but it's not fresh in my
> mind.

Don't think we can do anything in the update verify config. I was meaning 'diff -x softokn3.chk -x freebl3.chk' at
 http://mxr.mozilla.org/mozilla/source/testing/release/common/check_updates.sh#60

Incidentally, bug 470811 will give us an UPDATE_PACKAGING_R7, so that we always replace these two files (rather than try to patch and fail miserably, as been hitting win32 nightlies for a while).

> > * need to sync up hg.mozilla.org/users/stage-ffxbld/tools at the same time > > as updating the cvs mirrors
> Not sure what you mean...sync up from internal CVS? I need some more
> interpretation here.

It's this line - 
http://mxr.mozilla.org/mozilla/source/tools/release/configs/fx-moz19-staging-bootstrap.cfg#87
ie we're using a copy of the stuff we split out to hg:build/tools for staging runs. Mostly pull, but looks like update verify configs are bumped and pushed them back to hg. Ben may have updated stage-ffxbld copy for his recent 3.1b3 test run, but should probably also sync from CVS to hg (to pick up history for the releases we did since landing in hg).

Lots of other things to update in this file (version, pullDate, etc; useBetaChannel should be 1).
(In reply to comment #5) 
> Incidentally, bug 470811 will give us an UPDATE_PACKAGING_R7, so that we always
> replace these two files (rather than try to patch and fail miserably, as been
> hitting win32 nightlies for a while).

What's the likelihood of bug 470811 landing before builds start for 3.0.6?
 
> It's this line - 
> http://mxr.mozilla.org/mozilla/source/tools/release/configs/fx-moz19-staging-bootstrap.cfg#87
> ie we're using a copy of the stuff we split out to hg:build/tools for staging
> runs. Mostly pull, but looks like update verify configs are bumped and pushed
> them back to hg. Ben may have updated stage-ffxbld copy for his recent 3.1b3
> test run, but should probably also sync from CVS to hg (to pick up history for
> the releases we did since landing in hg).

OK, I'll try to merge in any changes from CVS before I fire this up (likely tomorrow at this point).
 
> Lots of other things to update in this file (version, pullDate, etc;
> useBetaChannel should be 1).

Having never done a staging run of this before, what's the best testing procedure here? Should I use the same vars for 3.0.5 (with updated tool versions) or should I do a bogus 3.0.6 release?
(In reply to comment #6)
> What's the likelihood of bug 470811 landing before builds start for 3.0.6?

I don't think we need it for 3.0.6 given we haven't had any issues with windows partials failing there, and I need to figure out how we're going to manage CVS + 2x hg repos, so not likely.

> > Lots of other things to update in this file (version, pullDate, etc;
> > useBetaChannel should be 1).
> 
> Having never done a staging run of this before, what's the best testing
> procedure here? Should I use the same vars for 3.0.5 (with updated tool
> versions) or should I do a bogus 3.0.6 release?

Normally we do a dummy 3.0.6 release to get as close to we can. So the basic idea is to update the cvsmirror.clean on staging-1.9-master (looks you've done this), then choose pullDates a bit before to your update. Happy to review your changes to the confg file.
Attachment #357356 - Flags: review?(bhearsum) → review+
I've been trying to get this running all weekend. I think I finally found the root causes of some of the failures I was seeing.

First, there was a symlink in place /builds/cvsroot -> /builds/cvsmirror.clean, so the cvsmirror step was putting stuff back into the same directory structure. This was causing breakage until I removed the symlink.

Second, we were running out of space on the windows slave. This was causing the Repack part of the win32 build step to die, despite the problem actually occurring in the previous step when l10n build creation was silently failing due to the lack of space. Need to file a bug for this.

I'm currently re-running with dummy steps for everything prior to the windows build step, and with the TinderConfig portion of the windows builds step set to not haltOnFailure (tagging has already occurred).
We just got the cutoffs for 3.0.6, so I wasn't able to get this done in time. :/ 

I'll keep running it, and get versions tagged for the next release at least.
No longer depends on: 471251
Nick: do you have any insight into the changes that I'm testing here? 

I see bug 457468 as the only blocker, and that seems to involve changes to updates only. Sadly, my staging runs haven't made it that far yet. The win32 build step keeps timing out trying to download builds, e.g.:

http://staging-1.9-master.build.mozilla.org:8810/builders/win32_build/builds/7/steps/shell_10/logs/stdio

Duplicating the same steps (downloading builds) by hand works fine though. :/
Summary: Run 1.9 staging before 3.0.6 to test recent changes → Run 1.9 staging before 3.0.7 to test recent changes
nagios reported socket timeouts today when connecting to fx-win32-1.9-slave1 today, which is some corroboration at least.
Nick: we finally made it to the update step, but the patcher-config-bump.pl step is failing. Here's the log:

http://staging-1.9-master.build.mozilla.org:8810/builders/update/builds/4/steps/shell_9/logs/stdio

Is this related to the changes in bug 457468 that I'm meant to be testing here?
(In reply to comment #13)
> Nick: we finally made it to the update step, but the patcher-config-bump.pl
> step is failing. Here's the log:
> 
> http://staging-1.9-master.build.mozilla.org:8810/builders/update/builds/4/steps/shell_9/logs/stdio
> 
> Is this related to the changes in bug 457468 that I'm meant to be testing here?

It's probably more related to the bump script itself...I'll look into this tomorrow.
I already mentioned this to coop, but for posterity, it looks like the staging CVS mirror already has a 3.0.6 patcher config file checked in. We can't yet re-bump patcher configs, so it will need to be reverted to 3.0.5 before the patcher config bump will work.

bug 466999 is related.
Finally made it to update_verify and looks like we hit a legit bug:

http://staging-1.9-master.build.mozilla.org:8810/builders/linux_update_verify/builds/0/steps/shell_8/logs/stdio

This patch fixes up the tag name we try to pull.
Attachment #359126 - Flags: review?(nthomas)
Comment on attachment 359126 [details] [diff] [review]
Convert . to _ in tag name

>+    $oldTagVersion +~ s/\./_/g;

You want =~ here, right ? r+ with that.
Attachment #359126 - Flags: review?(nthomas) → review+
Comment on attachment 359126 [details] [diff] [review]
Convert . to _ in tag name

(In reply to comment #17)
> You want =~ here, right ? r+ with that.

Fixed.
Attachment #359126 - Flags: checked‑in+
I hit a bunch of cvs-related errors (missing tags) in the later steps. It turns out when I recreated the initial cvsmirror, I did so with the bootstrap.cfg for 3.0.5 *before* the staging config for 3.0.6 landed (to save some time, natch). However, this meant that the tags for the 3.0.5 release got stripped/untagged, and those tags are necessary for verification 3.0.5->3.0.6 later.

I recreated the cvsmirror using the 3.0.6 config and started a fresh run late last night. It's currently repacking on win32.
update_verify is failing to check the version bump back in (ssl required):

http://staging-1.9-master.build.mozilla.org:8810/builders/linux_update_verify/builds/2/steps/shell_8/logs/stdio

Expected? Or is this new?
Looks like it needs to push to ssh://hg.m.o rather than https://hg.m.o
Didn't do an actually push, but did test that an hg clone works via ssh with the same ssh arguments as used for a push.
Attachment #359605 - Flags: review?(nthomas)
Attachment #359605 - Flags: review?(nthomas) → review-
Comment on attachment 359605 [details] [diff] [review]
Replace http(s) with ssh when doing an hg push

Looks like we already have a GetPushRepo function, and use it, just fail to plug in the right var to the actual command:

http://mxr.mozilla.org/seamonkey/source/tools/release/Bootstrap/Step.pm#335
Attached patch Use pushRepoSplinter Review
Attachment #359605 - Attachment is obsolete: true
Attachment #359630 - Flags: review?(nthomas)
Attachment #359630 - Flags: review?(nthomas) → review+
I *think* I'm ready to declare an end-to-end run complete now. 

The update_verify step ran on all three platforms. We legit passed on linux, win32 failed on the two files we know about (freebl3.chk and softokn3.chk), and Mac failed on gu-IN.

Logs for failures:
http://staging-1.9-master.build.mozilla.org:8810/builders/win32_update_verify/builds/6/steps/shell_8/logs/stdio
http://staging-1.9-master.build.mozilla.org:8810/builders/macosx_update_verify/builds/4/steps/shell_8/logs/stdio

Nick: The Mac failure is a bit puzzling. Wasn't attachment 350713 [details] [diff] [review] meant to fix that?
Looks like I checked the gu-IN change into CVS but not Hg, so it probably helped for 3.0.6 but not here. I guess we'll need to sync from CVS to Hg anyway, once we're ready to make the switch. Maybe after 3.0.6 ?
Eep, this fell off my radar.

Ben: when you tagged M13 for releases automation, did you do a sync between CVS->Hg?
I didn't do a sync, no. Good news is we're not yet done Repack for 3.0.7, we can probably sync over and update the tags before we hit an issue :).
Attachment #362761 - Flags: review?(bhearsum) → review+
Comment on attachment 362761 [details] [diff] [review]
Stop checking for gu-IN (hg patch)

changeset:   226:e5a6060364bb
Attachment #362761 - Flags: checked‑in+
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.