629509 - Update all OS X 10.6 test slaves to 10.6.6

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Joe Drew (not getting mail)]

Because of a bug in 10.6.2, which caused bug 629016, we need to update all our 10.6 OS X slaves to at least 10.6.3. My recommendation is to simply update to 10.6.6, to more accurately reflect our users' likely OS distribution.

This is quite important, because we need to block OpenGL on 10.6.2 and earlier, and if we do that without updating our 10.6 slaves, we won't be testing OpenGL at all.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Joe means testers, not builders, ie talos-r3-snow-*

Comment 2

[Armen [:armenzg]]

Attachment: Add 10.6.6 to the list of supported puppet platforms

shall I install also the iTunes update? :P

The 10.6.6 took 10mins to download and apply.
We can use "sudo softwareupdate --install MacOSXUpdCombo10.6.6-10.6.6" to install which means that we don't have to VNC.
A reboot was required for the change to take place.

We also need this puppet change to allow the machine to take updates.

I assume we will need a renaming of the builder name from 10.6.2 to 10.6.6 (patch coming).

I am running now unit tests and talos jobs on talos-r3-snow-001 to check what it does. The change I am testing is http://hg.mozilla.org/try/rev/e0dd77d96259 which re-enables again those changes.

You can see how they fail on production:
reftest 5626/5/230 - http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1296180493.1296181381.30447.gz
reftest 5622/9/250 -http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1296174673.1296179894.23329.gz
and how they succeed on staging after the 10.6.6 update (instead of 10.6.2).
reftest 5631/0/230 - http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1296241070.1296241767.22623.gz&fulltext=1

This shows that the update works.

The builds are on my people account: http://people.mozilla.com/~armenzg/builds




##### FOR THE RECORD

I trigger it with:
buildbot sendchange --master talos-staging-master02.build.mozilla.org:9010 --username [e-mail] --branch tryserver-macosx64-opt-unittest --revision e0dd77d96259 --property buildid:20110127143609  http://people.mozilla.com/~armenzg/builds/firefox-4.0b11pre.en-US.mac.dmg http://people.mozilla.com/~armenzg/builds/firefox-4.0b11pre.en-US.mac.tests.zip

Since the previous sendchange uses TryChooser (somewhere) it only triggers reftests. To fool the system I am pretending to be a mozilla-central change and therefore I can get the full run (yay for cheating!):
buildbot sendchange --master talos-staging-master02.build.mozilla.org:9010 --username sendchange-unittest --branch mozilla-central-macosx64-opt-unittest --revision e0dd77d96259 --property buildid: 20110127143609  http://people.mozilla.com/~armenzg/builds/firefox-4.0b11pre.en-US.mac.dmg http://people.mozilla.com/~armenzg/builds/firefox-4.0b11pre.en-US.mac.tests.zip


talos-r3-snow-001:talos-slave cltbld$ softwareupdate -l
Software Update Tool
Copyright 2002-2009 Apple

2011-01-28 08:16:34.054 softwareupdate[407:6517] PackageKit: Missing bundle path, skipping: <bundle id="com.apple.NetworkUtility"></bundle>
Software Update found the following new or updated software:
   * iPhoto_812-8.1.2
        iPhoto Update (8.1.2), 13259K [recommended]
   * MacOSXUpdCombo10.6.6-10.6.6
        Mac OS X Update Combined (10.6.6), 1034674K [recommended] [restart]
   * AirPortClientUpdate2009002-1.0
        AirPort Client Update 2009-002 (1.0), 13010K [recommended] [restart]
   * iMovie_805-8.0.6
        iMovie Update (8.0.6), 36271K [recommended]
   * iTunesX-10.1.2
        iTunes (10.1.2), 90946K [recommended]
   * RAWCamera27-2.7
        Digital Camera Raw Compatibility Update (2.7), 7110K [recommended]
   * AirPortUtility-5.5.2
        AirPort Utility (5.5.2), 11990K [recommended]
   * RemoteDesktopClient-3.4
        Remote Desktop Client Update (3.4), 5960K [recommended]
talos-r3-snow-001:talos-slave cltbld$ sudo time softwareupdate --install MacOSXUpdCombo10.6.6-10.6.6
Software Update Tool
Copyright 2002-2009 Apple

2011-01-28 08:17:53.295 softwareupdate[413:5803] PackageKit: Missing bundle path, skipping: <bundle id="com.apple.NetworkUtility"></bundle>

Downloading Mac OS X Update Combined
Waiting to install Mac OS X Update Combined
   Checking packages…
Installing 
   Waiting for other installations to complete…2011-01-28 08:19:37.327 softwareupdate[413:34a3] PackageKit: Missing bundle path, skipping: <bundle id="com.apple.NetworkUtility"></bundle>

   Validating packages…2011-01-28 08:19:37.634 softwareupdate[413:34a3] PackageKit: Missing bundle path, skipping: <bundle id="com.apple.NetworkUtility"></bundle>

   Writing files…
   Running package scripts…
   Removing old files…
   Optimizing system for installed software…
   Moving items into place…
   Registering updated components…
   Running package scripts…
   Registering updated applications…
   Writing package receipts…
Installed Mac OS X Update Combined
Done.

You have installed one or more updates that requires that you restart your
computer.  Please restart immediately.
      598.27 real       240.44 user        80.22 sys

Comment 3

[Armen [:armenzg]]

(In reply to comment #2)
> Since the previous sendchange uses TryChooser (somewhere) it only triggers
> reftests. To fool the system I am pretending to be a mozilla-central change and
> therefore I can get the full run (yay for cheating!):
> buildbot sendchange --master talos-staging-master02.build.mozilla.org:9010
> --username sendchange-unittest --branch mozilla-central-macosx64-opt-unittest
> --revision e0dd77d96259 --property buildid: 20110127143609 
> http://people.mozilla.com/~armenzg/builds/firefox-4.0b11pre.en-US.mac.dmg
> http://people.mozilla.com/~armenzg/builds/firefox-4.0b11pre.en-US.mac.tests.zip

Small typo:
buildbot sendchange --master talos-staging-master02.build.mozilla.org:9010 --username sendchange-unittest --branch mozilla-central-macosx64-opt-unittest --revision e0dd77d96259 http://people.mozilla.com/~armenzg/builds/firefox-4.0b11pre.en-US.mac.dmg http://people.mozilla.com/~armenzg/builds/firefox-4.0b11pre.en-US.mac.tests.zip

Comment 4

[Armen [:armenzg]]

I tested the builds and packages provided by joe's commit on the 10.6.6 machine and I got every test suite to be green.

My testing and analysis is now completed.

I will keep the machine running for the following changeset just to keep on doing jobs over the weekend:
http://tbpl.mozilla.org/?tree=Firefox&rev=5e576c6e3d75

Results:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1296248331.1296248723.20883.gz&fulltext=1
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1296248005.1296248227.18678.gz
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1296243216.1296244335.1320.gz
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1296246278.1296246951.13273.gz
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1296246950.1296247425.15351.gz
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1296245962.1296246279.10018.gz
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1296244973.1296245672.7658.gz
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1296245669.1296245963.8603.gz
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1296248721.1296248868.21223.gz
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1296247424.1296248006.17624.gz
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1296244334.1296244975.4425.gz

Comment 5

[Armen [:armenzg]]

Attachment: Rename from 10.6.2 to 10.6.6 for testing slaves

I recently noticed that we should update our builder name as well.
I have it running on staging so I don't know yet if it can has any problems.
I am afraid it might affect graph server posts.

Comment 6

[Joe Drew (not getting mail)]

I see no reason to avoid the upgrade to 10.6.6. I've verified that acceleration is enabled, and reftests are fine.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #5)
> I am afraid it might affect graph server posts.

Should check this carefully. For starters we have 'Mac OSX 10.6.2' as a platform option, lots of 10.6.2 in data.sql, eg
 http://hg.mozilla.org/graphs/file/ac18c882f3f5/sql/data.sql#l17

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 507924 [details] [diff] [review]
Add 10.6.6 to the list of supported puppet platforms

In this patch, the added non-comment line should be 10.6.0, not 10.2.0.

If we had a newer version of puppet, we could use regexes :(

However, the string 10.2.0 appears lots of places in the puppet configs, and you'll need to test/fix them all.

./manifests/base/os.pp:        "10.2.0" => "darwin10"
./manifests/includes/functions.pp:        "10.2.0": {
./manifests/os/osx.pp:        "10.2.0": {
./manifests/os/osx.pp:        "10.2.0": {
./manifests/os/talos_osx.pp:        "10.2.0": {
./manifests/os/talos_osx.pp:       "10.2.0": {
./manifests/os/talos_osx.pp:        "10.2.0": {
./manifests/packages/devtools.pp:                "10.2.0": {
./manifests/packages/repackaging-tools.pp:                "10.2.0": {

Comment 9

[Armen [:armenzg]]

Attachment: Add 10.6.6 to the list of supported puppet platforms and update

Thanks for catching that!
We would stop managing puppet with only my change.

I have checked on talos-r3-snow-001 and moz2-darwin10-slave03 that there is no fallouts on talking with puppet.

There is no need to address os/osx.pp, packages/devtools.pp packages/repackaging-tools.pp as they are just for builder machines.

Comment 10

[Armen [:armenzg]]

Attachment: Add 10.6.6 to the graph server

Nick how can I figure out why I don't get any problems on staging when I run talos jobs on staging?

AFAIK I have not landed anything on graphs-stage's DB.


TODO add scraping on tbox once this goes live

Comment 11

[Armen [:armenzg]]

I guess I can just check by looking at graph-stage's web UI.

Comment 12

[Armen [:armenzg]]

Attachment: Add 10.6.6 to the graph server [v1]

Interesting I can see the results:
http://graphs-stage.mozilla.org/graph.html#type=series&tests=[{%22test%22:%2236%22,%22branch%22:%221%22,%22machine%22:%22606%22,%22testrun%22:%221591819%22}]
but it says its a 10.6.2 machine even thought the builder name has 10.6.6 in it.
Tp4 on MacOSX 10.6.2
talos-r3-snow-001, Firefox branch
1/31/2011 8:14:19 AM
Build ID: 20110131035758

I think this is the right patch.
We move the snow slaves to point to the 21th os (10.6.6 - assuming that's the primary key we get) rather than the 17th (10.6.2).

I will try to get this into graphs-stage's DB before you wake up.

Comment 13

[Armen [:armenzg]]

Attachment: [sql statements] add 10.6.6 to talos

This is what catlee deployed for me on graphs-stage today.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Armen, I don't know the graph server enough to review these patches. Please catlee or alice.

Comment 15

[Armen [:armenzg]]

Comment on attachment 508484 [details]
[sql statements] add 10.6.6 to talos

You already did this for me.
Do you approve so I can ask IT to land it for me?

Comment 16

[Armen [:armenzg]]

joe we are scheduling this to happen on Thursday morning from 6AM to 8AM PST to tag along with another downtime going on.
We will bring it up on the Tuesday meeting for approval.
Sounds good?

Comment 17

[Joe Drew (not getting mail)]

Sounds lovely.

Comment 18

[Chris AtLee [:catlee]]

Comment on attachment 508484 [details]
[sql statements] add 10.6.6 to talos

update machines SET os_id=21 WHERE MATCH(name) AGAINST ('talos-r3-snow-'); 

this needs to be rewritten using LIKE.

also, the os_id on production may be different.

Comment 19

[Armen [:armenzg]]

Attachment: [sql statements] add 10.6.6 to talos [v1]

I can't make my local mysql to work right now.

Would this address the issues you mention?

Comment 20

[Armen [:armenzg]]

Comment on attachment 508410 [details] [diff] [review]
Add 10.6.6 to the list of supported puppet platforms and update

http://hg.mozilla.org/build/puppet-manifests/rev/46a59bcbf7a7

Comment 21

[Armen [:armenzg]]

Attachment: [sql statements] add 10.6.6 to talos [v2]

Is this ready for IT to apply it? Shall I wait until tomorrow just before the downtime?

Comment 22

[Armen [:armenzg]]

Attachment: [buildbotcustom] adjust tryserver to work with 10.6.6 renaming

Comment 23

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 509220 [details] [diff] [review]
[buildbotcustom] adjust tryserver to work with 10.6.6 renaming

Thanks for updating the try_parser tests too.

Comment 24

[Armen [:armenzg]]

We did not rename the slaves from 10.6.2 to 10.6.6 after all.
It would cause a discontinuity of the 10.6.2 posts on the graph server.

I have filed bug 631256 to keep track of the renaming work (I am proposing to do it after FF4) and talk with webdev on how to port forward the 10.6.2 posts.

All machines have been updated to 10.6.6 and I am using this changeset as the cut-line:
http://tbpl.mozilla.org/?tree=Firefox&rev=094a7967e171

Please ignore the set of talos jobs which were trying to run debug builds rather than opt builds (bug 631237). I triggered another 2 full sets to cover that up.

I have also updated and rebooted the ref image.

To determine if a machine has the change you can type:
> uname -r
> 10.6.0
instead of
> 10.2.0

Comment 25

[Armen [:armenzg]]

https://wiki.mozilla.org/ReferencePlatforms/Test/SnowLeopard#Update_from_10.6.2_to_10.6.6

Asked to update the ref image in https://bugzilla.mozilla.org/show_bug.cgi?id=617105#c16

Comment 26

[Armen [:armenzg]]

Nothing left to be done.

Comment 27

[Dustin J. Mitchell [:dustin] (he/him)]

Incidentally, this update is no longer available.  I'll update the wiki accordingly.

Comment 28

[Armen [:armenzg]]

I saw your changes. Thank you.

FTR it needs to be installed manually.
http://support.apple.com/kb/DL1349

Comment 29

[Dustin J. Mitchell [:dustin] (he/him)]

I couldn't get that update to work - it failed for me without any real description of why.