Last Comment Bug 629509 - Update all OS X 10.6 test slaves to 10.6.6
: Update all OS X 10.6 test slaves to 10.6.6
Status: RESOLVED FIXED
:
Product: Release Engineering
Classification: Other
Component: Other (show other bugs)
: other
: x86 Mac OS X
: P1 normal (vote)
: ---
Assigned To: Armen Zambrano [:armenzg] (EDT/UTC-4)
:
Mentors:
Depends on:
Blocks: 629016 631276
  Show dependency treegraph
 
Reported: 2011-01-27 14:17 PST by Joe Drew (not getting mail)
Modified: 2013-08-12 21:54 PDT (History)
5 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
Add 10.6.6 to the list of supported puppet platforms (500 bytes, patch)
2011-01-28 11:28 PST, Armen Zambrano [:armenzg] (EDT/UTC-4)
dustin: review-
Details | Diff | Splinter Review
Rename from 10.6.2 to 10.6.6 for testing slaves (1.02 KB, patch)
2011-01-28 13:59 PST, Armen Zambrano [:armenzg] (EDT/UTC-4)
bhearsum: review+
armenzg: checked‑in-
Details | Diff | Splinter Review
Add 10.6.6 to the list of supported puppet platforms and update (4.39 KB, patch)
2011-01-31 08:18 PST, Armen Zambrano [:armenzg] (EDT/UTC-4)
dustin: review+
bhearsum: review+
armenzg: checked‑in+
Details | Diff | Splinter Review
Add 10.6.6 to the graph server (3.88 KB, patch)
2011-01-31 08:27 PST, Armen Zambrano [:armenzg] (EDT/UTC-4)
no flags Details | Diff | Splinter Review
Add 10.6.6 to the graph server [v1] (14.73 KB, patch)
2011-01-31 08:39 PST, Armen Zambrano [:armenzg] (EDT/UTC-4)
catlee: review+
armenzg: checked‑in-
Details | Diff | Splinter Review
[sql statements] add 10.6.6 to talos (1.50 KB, text/plain)
2011-01-31 12:14 PST, Armen Zambrano [:armenzg] (EDT/UTC-4)
catlee: review-
Details
[sql statements] add 10.6.6 to talos [v1] (1.68 KB, text/plain)
2011-02-01 14:21 PST, Armen Zambrano [:armenzg] (EDT/UTC-4)
no flags Details
[sql statements] add 10.6.6 to talos [v2] (1.61 KB, text/plain)
2011-02-02 12:37 PST, Armen Zambrano [:armenzg] (EDT/UTC-4)
catlee: review+
armenzg: checked‑in-
Details
[buildbotcustom] adjust tryserver to work with 10.6.6 renaming (8.85 KB, patch)
2011-02-02 13:36 PST, Armen Zambrano [:armenzg] (EDT/UTC-4)
lukasblakk+bugs: review+
armenzg: checked‑in-
Details | Diff | Splinter Review

Description Joe Drew (not getting mail) 2011-01-27 14:17:58 PST
Because of a bug in 10.6.2, which caused bug 629016, we need to update all our 10.6 OS X slaves to at least 10.6.3. My recommendation is to simply update to 10.6.6, to more accurately reflect our users' likely OS distribution.

This is quite important, because we need to block OpenGL on 10.6.2 and earlier, and if we do that without updating our 10.6 slaves, we won't be testing OpenGL at all.
Comment 1 Nick Thomas [:nthomas] 2011-01-27 14:21:17 PST
Joe means testers, not builders, ie talos-r3-snow-*
Comment 2 Armen Zambrano [:armenzg] (EDT/UTC-4) 2011-01-28 11:28:53 PST
Created attachment 507924 [details] [diff] [review]
Add 10.6.6 to the list of supported puppet platforms

shall I install also the iTunes update? :P

The 10.6.6 took 10mins to download and apply.
We can use "sudo softwareupdate --install MacOSXUpdCombo10.6.6-10.6.6" to install which means that we don't have to VNC.
A reboot was required for the change to take place.

We also need this puppet change to allow the machine to take updates.

I assume we will need a renaming of the builder name from 10.6.2 to 10.6.6 (patch coming).

I am running now unit tests and talos jobs on talos-r3-snow-001 to check what it does. The change I am testing is http://hg.mozilla.org/try/rev/e0dd77d96259 which re-enables again those changes.

You can see how they fail on production:
reftest 5626/5/230 - http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1296180493.1296181381.30447.gz
reftest 5622/9/250 -http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1296174673.1296179894.23329.gz
and how they succeed on staging after the 10.6.6 update (instead of 10.6.2).
reftest 5631/0/230 - http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1296241070.1296241767.22623.gz&fulltext=1

This shows that the update works.

The builds are on my people account: http://people.mozilla.com/~armenzg/builds




##### FOR THE RECORD

I trigger it with:
buildbot sendchange --master talos-staging-master02.build.mozilla.org:9010 --username armenzg@mozilla.com --branch tryserver-macosx64-opt-unittest --revision e0dd77d96259 --property buildid:20110127143609  http://people.mozilla.com/~armenzg/builds/firefox-4.0b11pre.en-US.mac.dmg http://people.mozilla.com/~armenzg/builds/firefox-4.0b11pre.en-US.mac.tests.zip

Since the previous sendchange uses TryChooser (somewhere) it only triggers reftests. To fool the system I am pretending to be a mozilla-central change and therefore I can get the full run (yay for cheating!):
buildbot sendchange --master talos-staging-master02.build.mozilla.org:9010 --username sendchange-unittest --branch mozilla-central-macosx64-opt-unittest --revision e0dd77d96259 --property buildid: 20110127143609  http://people.mozilla.com/~armenzg/builds/firefox-4.0b11pre.en-US.mac.dmg http://people.mozilla.com/~armenzg/builds/firefox-4.0b11pre.en-US.mac.tests.zip


talos-r3-snow-001:talos-slave cltbld$ softwareupdate -l
Software Update Tool
Copyright 2002-2009 Apple

2011-01-28 08:16:34.054 softwareupdate[407:6517] PackageKit: Missing bundle path, skipping: <bundle id="com.apple.NetworkUtility"></bundle>
Software Update found the following new or updated software:
   * iPhoto_812-8.1.2
        iPhoto Update (8.1.2), 13259K [recommended]
   * MacOSXUpdCombo10.6.6-10.6.6
        Mac OS X Update Combined (10.6.6), 1034674K [recommended] [restart]
   * AirPortClientUpdate2009002-1.0
        AirPort Client Update 2009-002 (1.0), 13010K [recommended] [restart]
   * iMovie_805-8.0.6
        iMovie Update (8.0.6), 36271K [recommended]
   * iTunesX-10.1.2
        iTunes (10.1.2), 90946K [recommended]
   * RAWCamera27-2.7
        Digital Camera Raw Compatibility Update (2.7), 7110K [recommended]
   * AirPortUtility-5.5.2
        AirPort Utility (5.5.2), 11990K [recommended]
   * RemoteDesktopClient-3.4
        Remote Desktop Client Update (3.4), 5960K [recommended]
talos-r3-snow-001:talos-slave cltbld$ sudo time softwareupdate --install MacOSXUpdCombo10.6.6-10.6.6
Software Update Tool
Copyright 2002-2009 Apple

2011-01-28 08:17:53.295 softwareupdate[413:5803] PackageKit: Missing bundle path, skipping: <bundle id="com.apple.NetworkUtility"></bundle>

Downloading Mac OS X Update Combined
Waiting to install Mac OS X Update Combined
   Checking packages…
Installing 
   Waiting for other installations to complete…2011-01-28 08:19:37.327 softwareupdate[413:34a3] PackageKit: Missing bundle path, skipping: <bundle id="com.apple.NetworkUtility"></bundle>

   Validating packages…2011-01-28 08:19:37.634 softwareupdate[413:34a3] PackageKit: Missing bundle path, skipping: <bundle id="com.apple.NetworkUtility"></bundle>

   Writing files…
   Running package scripts…
   Removing old files…
   Optimizing system for installed software…
   Moving items into place…
   Registering updated components…
   Running package scripts…
   Registering updated applications…
   Writing package receipts…
Installed Mac OS X Update Combined
Done.

You have installed one or more updates that requires that you restart your
computer.  Please restart immediately.
      598.27 real       240.44 user        80.22 sys
Comment 3 Armen Zambrano [:armenzg] (EDT/UTC-4) 2011-01-28 11:34:54 PST
(In reply to comment #2)
> Since the previous sendchange uses TryChooser (somewhere) it only triggers
> reftests. To fool the system I am pretending to be a mozilla-central change and
> therefore I can get the full run (yay for cheating!):
> buildbot sendchange --master talos-staging-master02.build.mozilla.org:9010
> --username sendchange-unittest --branch mozilla-central-macosx64-opt-unittest
> --revision e0dd77d96259 --property buildid: 20110127143609 
> http://people.mozilla.com/~armenzg/builds/firefox-4.0b11pre.en-US.mac.dmg
> http://people.mozilla.com/~armenzg/builds/firefox-4.0b11pre.en-US.mac.tests.zip

Small typo:
buildbot sendchange --master talos-staging-master02.build.mozilla.org:9010 --username sendchange-unittest --branch mozilla-central-macosx64-opt-unittest --revision e0dd77d96259 http://people.mozilla.com/~armenzg/builds/firefox-4.0b11pre.en-US.mac.dmg http://people.mozilla.com/~armenzg/builds/firefox-4.0b11pre.en-US.mac.tests.zip
Comment 5 Armen Zambrano [:armenzg] (EDT/UTC-4) 2011-01-28 13:59:47 PST
Created attachment 507990 [details] [diff] [review]
Rename from 10.6.2 to 10.6.6 for testing slaves

I recently noticed that we should update our builder name as well.
I have it running on staging so I don't know yet if it can has any problems.
I am afraid it might affect graph server posts.
Comment 6 Joe Drew (not getting mail) 2011-01-28 19:22:21 PST
I see no reason to avoid the upgrade to 10.6.6. I've verified that acceleration is enabled, and reftests are fine.
Comment 7 Nick Thomas [:nthomas] 2011-01-29 00:30:14 PST
(In reply to comment #5)
> I am afraid it might affect graph server posts.

Should check this carefully. For starters we have 'Mac OSX 10.6.2' as a platform option, lots of 10.6.2 in data.sql, eg
 http://hg.mozilla.org/graphs/file/ac18c882f3f5/sql/data.sql#l17
Comment 8 Dustin J. Mitchell [:dustin] 2011-01-29 08:45:03 PST
Comment on attachment 507924 [details] [diff] [review]
Add 10.6.6 to the list of supported puppet platforms

In this patch, the added non-comment line should be 10.6.0, not 10.2.0.

If we had a newer version of puppet, we could use regexes :(

However, the string 10.2.0 appears lots of places in the puppet configs, and you'll need to test/fix them all.

./manifests/base/os.pp:        "10.2.0" => "darwin10"
./manifests/includes/functions.pp:        "10.2.0": {
./manifests/os/osx.pp:        "10.2.0": {
./manifests/os/osx.pp:        "10.2.0": {
./manifests/os/talos_osx.pp:        "10.2.0": {
./manifests/os/talos_osx.pp:       "10.2.0": {
./manifests/os/talos_osx.pp:        "10.2.0": {
./manifests/packages/devtools.pp:                "10.2.0": {
./manifests/packages/repackaging-tools.pp:                "10.2.0": {
Comment 9 Armen Zambrano [:armenzg] (EDT/UTC-4) 2011-01-31 08:18:09 PST
Created attachment 508410 [details] [diff] [review]
Add 10.6.6 to the list of supported puppet platforms and update

Thanks for catching that!
We would stop managing puppet with only my change.

I have checked on talos-r3-snow-001 and moz2-darwin10-slave03 that there is no fallouts on talking with puppet.

There is no need to address os/osx.pp, packages/devtools.pp packages/repackaging-tools.pp as they are just for builder machines.
Comment 10 Armen Zambrano [:armenzg] (EDT/UTC-4) 2011-01-31 08:27:33 PST
Created attachment 508412 [details] [diff] [review]
Add 10.6.6 to the graph server

Nick how can I figure out why I don't get any problems on staging when I run talos jobs on staging?

AFAIK I have not landed anything on graphs-stage's DB.


TODO add scraping on tbox once this goes live
Comment 11 Armen Zambrano [:armenzg] (EDT/UTC-4) 2011-01-31 08:29:05 PST
I guess I can just check by looking at graph-stage's web UI.
Comment 12 Armen Zambrano [:armenzg] (EDT/UTC-4) 2011-01-31 08:39:10 PST
Created attachment 508417 [details] [diff] [review]
Add 10.6.6 to the graph server [v1]

Interesting I can see the results:
http://graphs-stage.mozilla.org/graph.html#type=series&tests=[{%22test%22:%2236%22,%22branch%22:%221%22,%22machine%22:%22606%22,%22testrun%22:%221591819%22}]
but it says its a 10.6.2 machine even thought the builder name has 10.6.6 in it.
Tp4 on MacOSX 10.6.2
talos-r3-snow-001, Firefox branch
1/31/2011 8:14:19 AM
Build ID: 20110131035758

I think this is the right patch.
We move the snow slaves to point to the 21th os (10.6.6 - assuming that's the primary key we get) rather than the 17th (10.6.2).

I will try to get this into graphs-stage's DB before you wake up.
Comment 13 Armen Zambrano [:armenzg] (EDT/UTC-4) 2011-01-31 12:14:30 PST
Created attachment 508484 [details]
[sql statements] add 10.6.6 to talos

This is what catlee deployed for me on graphs-stage today.
Comment 14 Nick Thomas [:nthomas] 2011-01-31 12:59:18 PST
Armen, I don't know the graph server enough to review these patches. Please catlee or alice.
Comment 15 Armen Zambrano [:armenzg] (EDT/UTC-4) 2011-01-31 13:07:51 PST
Comment on attachment 508484 [details]
[sql statements] add 10.6.6 to talos

You already did this for me.
Do you approve so I can ask IT to land it for me?
Comment 16 Armen Zambrano [:armenzg] (EDT/UTC-4) 2011-01-31 13:25:12 PST
joe we are scheduling this to happen on Thursday morning from 6AM to 8AM PST to tag along with another downtime going on.
We will bring it up on the Tuesday meeting for approval.
Sounds good?
Comment 17 Joe Drew (not getting mail) 2011-01-31 13:37:59 PST
Sounds lovely.
Comment 18 Chris AtLee [:catlee] 2011-02-01 07:14:29 PST
Comment on attachment 508484 [details]
[sql statements] add 10.6.6 to talos

update machines SET os_id=21 WHERE MATCH(name) AGAINST ('talos-r3-snow-'); 

this needs to be rewritten using LIKE.

also, the os_id on production may be different.
Comment 19 Armen Zambrano [:armenzg] (EDT/UTC-4) 2011-02-01 14:21:44 PST
Created attachment 508898 [details]
[sql statements] add 10.6.6 to talos [v1]

I can't make my local mysql to work right now.

Would this address the issues you mention?
Comment 20 Armen Zambrano [:armenzg] (EDT/UTC-4) 2011-02-02 12:24:07 PST
Comment on attachment 508410 [details] [diff] [review]
Add 10.6.6 to the list of supported puppet platforms and update

http://hg.mozilla.org/build/puppet-manifests/rev/46a59bcbf7a7
Comment 21 Armen Zambrano [:armenzg] (EDT/UTC-4) 2011-02-02 12:37:58 PST
Created attachment 509197 [details]
[sql statements] add 10.6.6 to talos [v2]

Is this ready for IT to apply it? Shall I wait until tomorrow just before the downtime?
Comment 22 Armen Zambrano [:armenzg] (EDT/UTC-4) 2011-02-02 13:36:27 PST
Created attachment 509220 [details] [diff] [review]
[buildbotcustom] adjust tryserver to work with 10.6.6 renaming
Comment 23 Lukas Blakk [:lsblakk] use ?needinfo 2011-02-02 13:39:00 PST
Comment on attachment 509220 [details] [diff] [review]
[buildbotcustom] adjust tryserver to work with 10.6.6 renaming

Thanks for updating the try_parser tests too.
Comment 24 Armen Zambrano [:armenzg] (EDT/UTC-4) 2011-02-03 09:25:45 PST
We did not rename the slaves from 10.6.2 to 10.6.6 after all.
It would cause a discontinuity of the 10.6.2 posts on the graph server.

I have filed bug 631256 to keep track of the renaming work (I am proposing to do it after FF4) and talk with webdev on how to port forward the 10.6.2 posts.

All machines have been updated to 10.6.6 and I am using this changeset as the cut-line:
http://tbpl.mozilla.org/?tree=Firefox&rev=094a7967e171

Please ignore the set of talos jobs which were trying to run debug builds rather than opt builds (bug 631237). I triggered another 2 full sets to cover that up.

I have also updated and rebooted the ref image.

To determine if a machine has the change you can type:
> uname -r
> 10.6.0
instead of
> 10.2.0
Comment 26 Armen Zambrano [:armenzg] (EDT/UTC-4) 2011-02-03 12:40:49 PST
Nothing left to be done.
Comment 27 Dustin J. Mitchell [:dustin] 2011-05-05 21:26:49 PDT
Incidentally, this update is no longer available.  I'll update the wiki accordingly.
Comment 28 Armen Zambrano [:armenzg] (EDT/UTC-4) 2011-05-06 06:18:49 PDT
I saw your changes. Thank you.

FTR it needs to be installed manually.
http://support.apple.com/kb/DL1349
Comment 29 Dustin J. Mitchell [:dustin] 2011-05-06 08:16:09 PDT
I couldn't get that update to work - it failed for me without any real description of why.

Note You need to log in before you can comment on or make changes to this bug.