596259 - Tracking bug for build and release of Firefox 4.0beta7

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Comment 1

[Armen [:armenzg]]

Attachment: staging release changes

In preparation for tomorrow's staging release.

Comment 2

[[e-mail] (:bhearsum)]

Comment on attachment 481624 [details] [diff] [review]
staging release changes

Don't use the branch name as the sourceRepoRevision, that's not quite equivalent to what we'll be doing in production. Find a recent revision on that branch and use that instead, like 329afdc371fb.
Updating l10n-changesets_mozilla-2.0 with some recent changesets would be prudent.

Comment 3

[Armen [:armenzg]]

Attachment: beta7 staging release changes

As used. Addressed comments.
Reduced l10n changesets.
> buildbot sendchange --username=armenzg --master=localhost:9010 --branch=users/stage-ffxbld/mozilla-central -m "Firefox 4.0b7build1" doit

Comment 4

[Armen [:armenzg]]

Attachment: reduce difference between mozilla2{-staging}/config.py and mozilla/config.py

This change fixes the problem that I had for "make buildsymbols" for macosx64.
It also updates other variables to keep the files in sync with 0.8.x

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 482622 [details] [diff] [review]
reduce difference between mozilla2{-staging}/config.py and mozilla/config.py

Can you add the changes for the other platforms, too? Specifically:
- DISPLAY:2 for linux
- packageTests = True for windows
- buildSpace update for windows

Looks good otherwise.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

We should make this change to the release configs:
-enUSPlatforms       = ('linux', 'linux64', 'win32', 'macosx', 'macosx64')
+enUSPlatforms       = ('linux', 'linux64', 'win32', 'macosx64')

Comment 7

[Armen [:armenzg]]

Yeah we know that too. Thanks for checking.

Comment 8

[Armen [:armenzg]]

Attachment: syncing up config.py between mozilla2 and mozilla

Comment 9

[Armen [:armenzg]]

Here is what we will have to run manually after updates have been created:

> cd /opt/aus2/incoming/3/Firefox/4.0b6
> mkdir -p Darwin_x86-gcc3-u-ppc-i386/20100914072643
> rsync -av --exclude partial.txt Darwin_x86_64-gcc3/20100914073111/ Darwin_x86-gcc3-u-ppc-i386/20100914072643/

Ben sounds about right?

[cltbld@staging-stage 4.0b6]$ find Darwin_x86-gcc3-u-ppc-i386 -type f
Darwin_x86-gcc3-u-ppc-i386/20100914072643/be/betatest/complete.txt
Darwin_x86-gcc3-u-ppc-i386/20100914072643/be/releasetest/complete.txt
Darwin_x86-gcc3-u-ppc-i386/20100914072643/ca/betatest/complete.txt
Darwin_x86-gcc3-u-ppc-i386/20100914072643/ca/releasetest/complete.txt
Darwin_x86-gcc3-u-ppc-i386/20100914072643/cs/betatest/complete.txt
Darwin_x86-gcc3-u-ppc-i386/20100914072643/cs/releasetest/complete.txt
Darwin_x86-gcc3-u-ppc-i386/20100914072643/da/betatest/complete.txt
Darwin_x86-gcc3-u-ppc-i386/20100914072643/da/releasetest/complete.txt
Darwin_x86-gcc3-u-ppc-i386/20100914072643/de/betatest/complete.txt
Darwin_x86-gcc3-u-ppc-i386/20100914072643/de/releasetest/complete.txt
Darwin_x86-gcc3-u-ppc-i386/20100914072643/en-US/betatest/complete.txt
Darwin_x86-gcc3-u-ppc-i386/20100914072643/en-US/releasetest/complete.txt
Darwin_x86-gcc3-u-ppc-i386/20100914072643/zh-TW/betatest/complete.txt
Darwin_x86-gcc3-u-ppc-i386/20100914072643/zh-TW/releasetest/complete.txt
[cltbld@staging-stage 4.0b6]$ find Darwin_x86_64-gcc3 -type f
Darwin_x86_64-gcc3/20100914073111/be/betatest/complete.txt
Darwin_x86_64-gcc3/20100914073111/be/betatest/partial.txt
Darwin_x86_64-gcc3/20100914073111/be/releasetest/complete.txt
Darwin_x86_64-gcc3/20100914073111/be/releasetest/partial.txt
Darwin_x86_64-gcc3/20100914073111/ca/betatest/complete.txt
Darwin_x86_64-gcc3/20100914073111/ca/betatest/partial.txt
Darwin_x86_64-gcc3/20100914073111/ca/releasetest/complete.txt
Darwin_x86_64-gcc3/20100914073111/ca/releasetest/partial.txt
Darwin_x86_64-gcc3/20100914073111/cs/betatest/complete.txt
Darwin_x86_64-gcc3/20100914073111/cs/betatest/partial.txt
Darwin_x86_64-gcc3/20100914073111/cs/releasetest/complete.txt
Darwin_x86_64-gcc3/20100914073111/cs/releasetest/partial.txt
Darwin_x86_64-gcc3/20100914073111/da/betatest/complete.txt
Darwin_x86_64-gcc3/20100914073111/da/betatest/partial.txt
Darwin_x86_64-gcc3/20100914073111/da/releasetest/complete.txt
Darwin_x86_64-gcc3/20100914073111/da/releasetest/partial.txt
Darwin_x86_64-gcc3/20100914073111/de/betatest/complete.txt
Darwin_x86_64-gcc3/20100914073111/de/betatest/partial.txt
Darwin_x86_64-gcc3/20100914073111/de/releasetest/complete.txt
Darwin_x86_64-gcc3/20100914073111/de/releasetest/partial.txt
Darwin_x86_64-gcc3/20100914073111/en-US/betatest/complete.txt
Darwin_x86_64-gcc3/20100914073111/en-US/betatest/partial.txt
Darwin_x86_64-gcc3/20100914073111/en-US/releasetest/complete.txt
Darwin_x86_64-gcc3/20100914073111/en-US/releasetest/partial.txt
Darwin_x86_64-gcc3/20100914073111/zh-TW/betatest/complete.txt
Darwin_x86_64-gcc3/20100914073111/zh-TW/betatest/partial.txt
Darwin_x86_64-gcc3/20100914073111/zh-TW/releasetest/complete.txt
Darwin_x86_64-gcc3/20100914073111/zh-TW/releasetest/partial.txt

Comment 10

[Armen [:armenzg]]

After the previous change ppc-i386 users should be able to update to the new universal mac build. The update we have decided to make it a complete update rather than fiddling with partial.txt files. beltzner said it is fine.

Users running a ppc-i386 build on 10.5 after installing the complete update to the new universal will hit bug 600411 which beltzner said is fine to hit (Update applies, crash on start and then it works fine). Users running the ppc-i386 build on 10.6 won't have any issues.

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 483214 [details] [diff] [review]
syncing up config.py between mozilla2 and mozilla

r=me with the following (you can do it at check-in):
- don't land the production release config changes, it's icky to have 'TODO' type of things checked in.
- you make the linux changes in mozilla2-staging/config.py, too.

Comment 12

[[e-mail] (:bhearsum)]

(In reply to comment #9)
> Here is what we will have to run manually after updates have been created:
> 
> > cd /opt/aus2/incoming/3/Firefox/4.0b6
> > mkdir -p Darwin_x86-gcc3-u-ppc-i386/20100914072643
> > rsync -av --exclude partial.txt Darwin_x86_64-gcc3/20100914073111/ Darwin_x86-gcc3-u-ppc-i386/20100914072643/
> 
> Ben sounds about right?

Kindof...that's not the directory to do it in for the real thing, though. Once the real beta 7 snippets are generated, you should pop into /opt/aus2/snippets/staging and do that for all 3 directories:
- 201010??-Firefox-4.0b7
- 201010??-Firefox-4.0b7-beta
- 201010??-Firefox-4.0b7-test

I'm not 100% sure if we need to do more symlinking than that.....in the nightly directory we have 3 directories symlinked to Darwin_x86_64-gcc3. Probably not; at least for this release. Nick, can you confirm that?

Comment 13

[Armen [:armenzg]]

Comment on attachment 483214 [details] [diff] [review]
syncing up config.py between mozilla2 and mozilla

(In reply to comment #11)
> Comment on attachment 483214 [details] [diff] [review]
> changes needed for beta7
> 
> r=me with the following (you can do it at check-in):
> - don't land the production release config changes, it's icky to have 'TODO'
> type of things checked in.
> - you make the linux changes in mozilla2-staging/config.py, too.

Comments addressed.

As pushed:
http://hg.mozilla.org/build/buildbot-configs/rev/13898287789a

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #12)
There's a couple of issues we need to address here.

1) Bug 552924 changed the AUS query in b5 onwards, so we need symlinks for 
   b5 and b6 that map Darwin_x86-gcc3-u-ppc-i386 to the right place. We don't 
   need the ppc equivalent because those are no longer supported

2) Switching to the i386+x86_64 universal means we're not generating
   directories for Darwin_Universal-gcc3 any more, for all versions in the 
   patcher config

Combining those I think we want to do this:
a) For 3.7a1-5 and 4.0b1, which are en-US only, we can symlink/copy 
     4.0b1/Darwin_x86_64-gcc3  to  $version/Darwin_Universal-gcc3

b) For 4.0b2-b4, which have locales, symlink/copy from 
     $version/Darwin_x86_64-gcc3  to  $version/Darwin_Universal-gcc3 

c) For 4.0b5 and 4.0b6 symlink/copy from
     $version/Darwin_x86_64-gcc3  to  $version/Darwin_x86-gcc3-u-ppc-i386

That's for both the 201010DD-Firefox-4.0b7 and 201010DD-Firefox-4.0b7-test directories that get created automatically. I'd suggest letting the automation run, then duplicate the 201010DD-Firefox-4.0b7 to 201010D-Firefox-4.0b7-fixed, modifying 201010DD-Firefox-4.0b7-fixed and pushing it out. All the changes are additive.

Comment 15

[[e-mail] (:bhearsum)]

Makes sense to me. Are we making patcher changes or config changes in the future to avoid needing to futz around in later releases?

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

patcher changes I guess, and any follow on effects. AUS3 is still below the horizon.

Comment 17

[Armen [:armenzg]]

For now I will remove myself as I am going into holidays from EOD and be back next Tuesday.

Comment 18

[Armen [:armenzg]]

After today's meeting and the expected stabilization time I assume I will be back before the GO to build.

Comment 19

[Armen [:armenzg]]

Attachment: beta7 config changes

Comment 20

[Armen [:armenzg]]

Comment on attachment 487629 [details] [diff] [review]
beta7 config changes

TODO will be replaced by the right revision.

Comment 21

[[e-mail] (:bhearsum)]

Comment on attachment 487629 [details] [diff] [review]
beta7 config changes

You need an l10n-changesets_mozilla-2.0 update too, of course.
You need to bump the version in releaseNotesUrl.

Comment 22

[Armen [:armenzg]]

Attachment: beta7 config changes

Addressed release notes.

Can I get r+ conditional to do the following before commit?

* TODO to be replaced with changeset before commit (from release drivers).
* mozilla2/l10n-changesets_mozilla-2.0 to be replaced with contents from https://l10n-stage-sj.mozilla.org/shipping/confirm-ship?ms=fx4.0b7  after we hit "ship it" (if there are no pending sign offs then there is no need to ping Pike)

Comment 23

[[e-mail] (:bhearsum)]

Comment on attachment 487665 [details] [diff] [review]
beta7 config changes

This looks fine but I'd prefer to review the final diff too, if I'm around.

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Release bugs don't block releases, the number of bugs goes to zero then we build.

Comment 25

[Armen [:armenzg]]

Attachment: beta7 config changes (includes changeset and l10n-changesets)

Comment 26

[[e-mail] (:bhearsum)]

Comment on attachment 488262 [details] [diff] [review]
beta7 config changes (includes changeset and l10n-changesets)

Looks good to me.

Comment 27

[Armen [:armenzg]]

Comment on attachment 488262 [details] [diff] [review]
beta7 config changes (includes changeset and l10n-changesets)

Checked in as:
http://hg.mozilla.org/build/buildbot-configs/rev/23deb7d0556b

FTR, I hit bug 609717 during tagging. I recovered by manually login into the machine and pushing the change.

Comment 28

[Armen [:armenzg]]

I finally got this tested!
With few of the alphas, most of the betas including beta5 and beta6.

Could you please review the steps followed below?

BTW what are all these different "b6" directories in aus2?
# /opt/aus2/snippets/staging
20100913-Firefox-4.0b6
20100913-Firefox-4.0b6-test
20100914-Firefox-4.0b6
20100914-Firefox-4.0b6-less-partials
20100914-Firefox-4.0b6-test

I believe the steps to follow are the following:
>> BEGINNING <<
cd /opt/aus2/snippets/staging
cp -r 20100913-Firefox-4.0b6 20100913-Firefox-4.0b6-fixed
cd 20100913-Firefox-4.0b6-fixed

# back up the snippets
for n in {1..5}; do mv 3.7a$n/Darwin_Universal-gcc3 3.7a$n/Darwin_Universal-gcc3.bak; done
for n in {1..4}; do mv 4.0b$n/Darwin_Universal-gcc3 4.0b$n/Darwin_Universal-gcc3.bak; done
for n in {5..6}; do mv 4.0b$n/Darwin_x86-gcc3-u-ppc-i386 4.0b$n/Darwin_x86-gcc3-u-ppc-i386.bak; done

mkdir -p 3.7a1/Darwin_Universal-gcc3/20100208064157
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 3.7a1/Darwin_Universal-gcc3/20100208064157

mkdir -p 3.7a2/Darwin_Universal-gcc3/20100228155422
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 3.7a2/Darwin_Universal-gcc3/20100228155422/

mkdir -p 3.7a3/Darwin_Universal-gcc3/20100315065900
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 3.7a3/Darwin_Universal-gcc3/20100315065900/

mkdir -p 3.7a4/Darwin_Universal-gcc3/20100407105356
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 3.7a4/Darwin_Universal-gcc3/20100407105356/

mkdir -p 3.7a5/Darwin_Universal-gcc3/20100610053455
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 3.7a5/Darwin_Universal-gcc3/20100610053455

mkdir -p 4.0b1/Darwin_Universal-gcc3/20100630131607
rsync -av --exclude partial.txt 4.0b1/Darwin_x86_64-gcc3/20100630131936/ 4.0b1/Darwin_Universal-gcc3/20100630131607/

mkdir -p 4.0b2/Darwin_Universal-gcc3/20100720175703
rsync -av --exclude partial.txt 4.0b2/Darwin_x86_64-gcc3/20100720180051/ 4.0b2/Darwin_Universal-gcc3/20100720175703/

mkdir -p 4.0b3/Darwin_Universal-gcc3/20100805181943/
rsync -av --exclude partial.txt 4.0b3/Darwin_x86_64-gcc3/20100805182338/ 4.0b3/Darwin_Universal-gcc3/20100805181943/

mkdir -p 4.0b4/Darwin_Universal-gcc3/20100818121614
rsync -av --exclude partial.txt 4.0b4/Darwin_x86_64-gcc3/20100818121922/ 4.0b4/Darwin_Universal-gcc3/20100818121614/

mkdir -p 4.0b5/Darwin_x86-gcc3-u-ppc-i386/20100831070010
rsync -av --exclude partial.txt 4.0b5/Darwin_x86_64-gcc3/20100831070403/ 4.0b5/Darwin_x86-gcc3-u-ppc-i386/20100831070010/

mkdir -p 4.0b6/Darwin_x86-gcc3-u-ppc-i386/20100914072643
rsync -av --exclude partial.txt 4.0b6/Darwin_x86_64-gcc3/20100914073111/ 4.0b6/Darwin_x86-gcc3-u-ppc-i386/20100914072643/

>> END <<

I obtained the buildids like this:

# [e-mail]
cd /opt/aus2/snippets/staging/20100914-Firefox-4.0b6-test/Firefox
for dir in `ls`; do echo $dir; ls $dir/Darwin*; done

# after formatting the output I get this
3.7a1/Darwin_Universal-gcc3/20100208064157/
3.7a2/Darwin_Universal-gcc3/20100228155422/
3.7a3/Darwin_Universal-gcc3/20100315065900/
3.7a4/Darwin_Universal-gcc3/20100407105356/
3.7a5/Darwin_Universal-gcc3/20100610053455/
3.7a5/Darwin_x86_64-gcc3/20100610053411/
4.0b1/Darwin_Universal-gcc3/20100630131607/
4.0b1/Darwin_x86_64-gcc3/20100630131936/
4.0b2/Darwin_Universal-gcc3/20100720175703/
4.0b2/Darwin_x86_64-gcc3/20100720180051/
4.0b3/Darwin_Universal-gcc3/20100805181943/
4.0b3/Darwin_x86_64-gcc3/20100805182338/
4.0b4/Darwin_Universal-gcc3/20100818121614/
4.0b4/Darwin_x86_64-gcc3/20100818121922/
4.0b5/Darwin_Universal-gcc3/20100831070010/
4.0b5/Darwin_x86_64-gcc3/20100831070403/

Comment 29

[[e-mail] (:bhearsum)]

Please get rid of the .bak directories. Your backups are in the original snippet directory, you definitely don't want these to be pushed.

You should make the same modifications to the test snippets, and re-push them. QA should test them in various scenarios before we go to beta.

Don't exclude partials for 4.0b6, they're valid.

Where did you run this? I can't find the -fixed directory on aus2-staging nor staging-stage.

Comment 30

[Armen [:armenzg]]

I did it on staging-stage.build:/opt/aus2/incoming/3/Firefox/

I copied all 3.7* and 4.0* into backup_beta7 and the problem of the snippets in there is that they didn't have Darwin_Universal-gcc3 folders to be moved as we had regenerated the updates from scratch.(In reply to comment #29)

> Don't exclude partials for 4.0b6, they're valid.
> 
Why did I exclude them on comment 9? I believe that we exclude the partial because they are not generated from old universal to new universal but I can't recall.

> You should make the same modifications to the test snippets, and re-push them.
> QA should test them in various scenarios before we go to beta.
> 
Right. I will push them later today.

Comment 31

[[e-mail] (:bhearsum)]

(In reply to comment #30)
> I did it on staging-stage.build:/opt/aus2/incoming/3/Firefox/
> 
> I copied all 3.7* and 4.0* into backup_beta7 and the problem of the snippets in
> there is that they didn't have Darwin_Universal-gcc3 folders to be moved as we
> had regenerated the updates from scratch.(In reply to comment #29)

In the future, you should never be messing around with snippets inside of incoming/. Per Nick's instructions, you should be making copies of the directories in /opt/aus2/snippets/staging, adjusting them, and then pushing them.

> > Don't exclude partials for 4.0b6, they're valid.
> > 
> Why did I exclude them on comment 9? I believe that we exclude the partial
> because they are not generated from old universal to new universal but I can't
> recall.

Whoops, you're right about this.

> 
> > You should make the same modifications to the test snippets, and re-push them.
> > QA should test them in various scenarios before we go to beta.
> > 
> Right. I will push them later today.

Please make the adjustments in the correct directory and have me look over them again first.

Comment 32

[Armen [:armenzg]]

Now doing everything on aus2-staging but on "snippets/staging" (which is not live!):

>> BEGIN <<
#cltbld@aus-staging
cd /opt/aus2/snippets/staging/
cp -r 20101104-Firefox-4.0b7-test 20101104-Firefox-4.0b7-test-fixed
cd 20101104-Firefox-4.0b7-test-fixed/Firefox
# no need for moving "Darwin_Universal". The staging release was
# generated with "macosx" and "macosx64" and that is why I had 
# decided to move them on staging-stage
mkdir & rsync dance for each 3.7* and 4.0* directory as mentioned on comment 28.
~/bin/pushsnip 20101104-Firefox-4.0b7-test-fixed
cd -

cp -r 20101104-Firefox-4.0b7 20101104-Firefox-4.0b7-fixed
cd 20101104-Firefox-4.0b7-fixed/Firefox
mkdir & rsync dance for each 3.7* and 4.0* directory as mentioned on comment 28.
>> END <<

Once we get sign off from QA and release drivers we will push with the following command:
~/bin/pushsnip 20101104-Firefox-4.0b7-fixed

Comment 33

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Add missing ja-JP-mac to update verify

We have 'ja-JP-mac osx' in shipped locales, which isn't handled properly when osx means macosx64 elsewhere. This just corrects the gap. To land before final_verification.

Comment 34

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Add missing ja-JP-mac to update verify

... and also fix the path to the too build. 99% sure we don't use this for final verify but just in case (and for historical value).

Comment 35

[Armen [:armenzg]]

Comment on attachment 489440 [details] [diff] [review]
Add missing ja-JP-mac to update verify

http://hg.mozilla.org/build/tools/rev/326d283a0577