548768 - Tracking bug for running unittests on user desktop OS

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Until now, we've always run unittests on the same OS that we build on. This is because of an original unittest requirement of doing a build on the machine immediately before running any unittests. This meant that we could only run unittests on osx10.5, win2k3 and centos5, even though these are not typical user desktop OS. 

It would be better if we ran unittests on more typical user desktop OS. We already have Talos perf tests running on osx10.4, osx10.5, WinXP, Win7, fedora12, fedora12x64, and soon also osx10.6. Running unittests on those OS would be more representative of typical users.

One approach would have been to build out a separate pool of machines imaged with these same 7 identical OSs. However, that would require maintaining two different sets of 7 ref images, and also reducing our burst capability across the divided pools. Instead we decided to increase the pool-o-slaves to handle extra load and then run unittests on the same pool-o-slaves that already run Talos. This would be easier to maintain, scale better and handle burst loads better.

Some pre-req projects were:
1) separate out unittests from the builds. This allows us to run unittests on a pre-existing build, separate from the machine that actually generated the build. Details in bug#383136.
2) unify account/password/network configurations of talos machines to line up with what are used by the build+unittest machines. Details in bug#537065.
3) rebase talos machines on newer still-available hardware. This allowed us to build out a larger pool of identically spec'd slaves. Its unclear if this larger pool-o-slaves increased enough to handle the extra load of the newly added talos OS, and *also* handle the load of running unittests on those OS. However, because we're now using still-available hardware, we can further grow the pool-o-slaves if needed. Details in bug#537065


Whats left to do? 

This bug is to track:
- enabling running-unittests-on-preexisting-build on each talos OS - maybe each OS would be different dep bug?
- disabling unittests on the build machines

Comment 2

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

It's hard for me to understand how "run unit tests on different OSes" is a duplicate of "keep fixed performance characteristics for the unit test systems", even having read this bug a few times now.  Could someone point me at the context that explains how they're related?

Comment 3

[Aki Sasaki (not active)]

If we move our tests to hardware, VM settings will no longer be applicable ?

Comment 4

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Oh, OK -- one really has to read between the lines to see that part, I guess!  Thank you, I didn't manage to make that connection on my previous readings.

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

I am setting bug 548768 as blocking because I was unable to get gnuwin32's tar working on the slaves.  We don't want Cygwin and I was unable to get gnuwin32's tar or bzip working on the windows 7 test machine.

Comment 6

[alice nodelman [:alice] [:anode]]

Set a dependency on talos staging, as it needs to be in good shape to test this stuff.

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #0)
...
> This bug is to track:
> - enabling running-unittests-on-preexisting-build on each talos OS - maybe each
> OS would be different dep bug?
- running unittests manually first, on each OS, is in each of the dep bugs.
- patch master.cfg to run unittests on minis
- patch TBPL to handle unittests on different OS (bug#532560)
- run concurrently and watch for variances/problems. Also watch for wait
time/load problems.
> - disabling unittests on the build machines


I think thats it, but chime in if I missed anything?

Comment 9

[alice nodelman [:alice] [:anode]]

Have taken rev3 boxes offline to be used a proof of concept (ie, manual set up only of unit test on talos):

- talos-r3-w7-001
- talos-r3-fed-001
- talos-r3-fed64-001
- talos-r3-snow-001
- talos-r3-leopard-001

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Alice is going to run point on this.

Comment 11

[Armen [:armenzg]]

(In reply to comment #9)
> - talos-r3-fed-001
> - talos-r3-fed64-001
These have been put back to the pool. No tool chain changes have been done to them.
Nuke ~/armenzg-unittests if needed/wanted.

Comment 12

[alice nodelman [:alice] [:anode]]

Update:

- armenzg: unit test require no tool chain changes on fedora, will move on to looking at automating the process with a factory that inherits from TalosFactory, also look into getting sendchanges for packaged unit tests to talos-staging-master02
- lsblakk: possibly require buildbotcustom directory on mac machines, will have to investigate further as automation comes up
- jhford: w7 blocked on packaged unit tests as zips, starting on xp
- alice: bring up new talos staging using borrowed production rev3 machines

coming up:
- alice: need to hand off before going on long weekend so that we don't get blocked

Comment 13

[Chris AtLee [:catlee]]

(In reply to comment #12)
> Update:
> 
> - armenzg: unit test require no tool chain changes on fedora, will move on to
> looking at automating the process with a factory that inherits from
> TalosFactory, also look into getting sendchanges for packaged unit tests to
> talos-staging-master02

Why not use UnittestPackagedBuildFactory?  I expect it could be used with only minor modifications.

Comment 14

[John Ford [:jhford] CET/CEST Berlin Time]

since we are changing to zipped tests, it might be worthwhile to verify that unzip works as expected on all platforms.

Comment 15

[Armen [:armenzg]]

(In reply to comment #12)
> - armenzg: ... getting sendchanges for packaged unit tests to
> talos-staging-master02

The sendchange-unittest currently gets send to the localhost:
    master: localhost:9010
    branch: mozilla-central-linux-opt-unittest
    revision: b3e305eb173c
    comments: 
    user: sendchange-unittest
    files: ['http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1268302881/firefox-3.7a3pre.en-US.linux-i686.tar.bz2']

We should add another one for staging-master02 (sending to both places until we do the flip).
Something like:
    master: talos-staging-master02.build.mozilla.org:9010
    branch: mozilla-central-linux-opt-unittest

The code in question is:
http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py#l1322

We will also have to do this change for debug builds.

Comment 16

[Armen [:armenzg]]

Attachment: sendchange-unitests being sent to talos-staging-master02

This sendchanges happen as soon as the build (and other stuff) gets uploaded.
This will activate the sendchange for normal and debug builds.

Comment 17

[Armen [:armenzg]]

Attachment: [WIP] this adds the test builders for talos

Lukas this gets the job done with UnittestPackaged factories. As catlee mentioned this factory gets everything done.

This patch adds the builders, the schedulers and the factories for the test builders for Linux. I did not have time to do more and nothing with regards the debug test builders.

This is the sendchange I used locally:

buildbot sendchange --master='localhost:9010' --branch=mozilla-central-linux-opt-unittest --revision=85fe77e1b558 --comment='' --user=sendchange-unittest 'http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1268329194/firefox-3.7a3pre.en-US.linux-i686.tar.bz2'

Comment 18

[Armen [:armenzg]]

For the record, the previous patch added the test builders for Fedora 12 and my VM started doing packaged unit tests.

The patch can be easily modified to make it available for all other platforms and adding the debug test builders.

The most tedious thing will be to put just the right info in config.py

As I mentioned on IRC and on my previous comment that my only concern is to try to hack TalosUnittestFactory when UnittestPackagedUnittestFactory already gets the job done.

Comment 19

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: default packageTests factory argument to be true

Pro: tests packages by default, override when we do not want them
Con: disk space usage for things that don't run packaged unittests.  aiui, that is the current production try server.

I am going to test this in sm01 right now.

Comment 20

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 432156 [details] [diff] [review]
default packageTests factory argument to be true

Ok, tested this in staging and i had the packages upload to:
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1268425908/firefox-3.7a3pre.en-US.linux-x86_64.tar.bz2
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1268425908/firefox-3.7a3pre.en-US.linux-x86_64.tests.tar.bz2
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1268425908/firefox-3.7a3pre.en-US.linux-x86_64.txt
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1268425908/firefox-3.7a3pre.en-US.langpack.xpi

It failed to upload to AUS because it has the wrong private keys.

Comment 21

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 431932 [details] [diff] [review]
sendchange-unitests being sent to talos-staging-master02

I ran this in staging at the same time as my patch and i did verify that the sendchange went through.  It showed on talos-staging-master02 as

Changed by: sendchange
Changed at: Fri 12 Mar 2010 17:49:01
Branch: mozilla-central-linux64
Revision: a9b980fcb73a

Changed files:

    * http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1268444163/firefox-3.7a3pre.en-US.linux-x86_64.tar.bz2

Comments:

Properties:

Comment 22

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Comment on attachment 432156 [details] [diff] [review]
default packageTests factory argument to be true

Yes, we want this! *stamp*

Comment 23

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Comment on attachment 431932 [details] [diff] [review]
sendchange-unitests being sent to talos-staging-master02

>diff --git a/mozilla2-staging/config.py b/mozilla2-staging/config.py
>--- a/mozilla2-staging/config.py
>+++ b/mozilla2-staging/config.py
>@@ -56,17 +56,18 @@ GLOBAL_VARS = {
>     'talos_masters': [
>         ('talos-staging-master02.build.mozilla.org:9010', False),
>         ('talos-staging-master.mozilla.org:9010', False),
>         ('talos-master02.build.mozilla.org:9010', False),
>     ],
>     # List of unittest masters to notify of new builds to test,
>     # and if a failure to notify the master should result in a warning
>     'unittest_masters': [('localhost:9010', True, 0),
>-                         ('localhost:9011', True, 0)],
>+                         ('localhost:9011', True, 0),
>+                         ('talos-staging-master02.build.mozilla.org:9010', True, 0),],

nit: remove extra comma, to get " 0)],"

r+ with this fix.



>     'unittest_suites': [
>         ('mochitests', ['mochitest-plain']),
>         ('mochitest-other', ['mochitest-chrome', 'mochitest-browser-chrome',
>             'mochitest-a11y']),
>         ('reftest', ['reftest']),
>         ('crashtest', ['crashtest']),
>         ('xpcshell', ['xpcshell']),
>     ],
>diff --git a/mozilla2/config.py b/mozilla2/config.py
>--- a/mozilla2/config.py
>+++ b/mozilla2/config.py
>@@ -57,17 +57,18 @@ GLOBAL_VARS = {
>         ('talos-master.mozilla.org:9010', True),
>         ('talos-master.mozilla.org:9012', False),
>         ('talos-staging-master.mozilla.org:9010', False),
>         ('talos-staging-master02.build.mozilla.org:9010', False),
>         ('talos-master02.build.mozilla.org:9010', False),
>     ],
>     # List of unittest masters to notify of new builds to test,
>     # and if a failure to notify the master should result in a warning
>-    'unittest_masters': [('localhost:9010', False, 0)],
>+    'unittest_masters': [('localhost:9010', False, 0),
>+                         ('talos-staging-master02.build.mozilla.org:9010', False, 0)],
>     'unittest_suites': [
>         ('mochitests', ['mochitest-plain']),
>         ('mochitest-other', ['mochitest-chrome', 'mochitest-browser-chrome',
>             'mochitest-a11y']),
>         ('reftest', ['reftest']),
>         ('crashtest', ['crashtest']),
>         ('xpcshell', ['xpcshell']),
>     ],

Comment 24

[John Ford [:jhford] CET/CEST Berlin Time]

ok, this bug has been added to https://wiki.mozilla.org/ReleaseEngineering:Maintenance for landing when we are able to.

Comment 25

[Armen [:armenzg]]

Comment on attachment 432156 [details] [diff] [review]
default packageTests factory argument to be true

This patch is incorrect.
If you look at:
http://hg.mozilla.org/build/buildbotcustom/file/tip/misc.py#l598
> elif pf.get('enable_opt_unittests'):
>    packageTests = True
You will see that you are enabling optimized unittests in all branches and all platforms. We should only turn optimized unitests were needed.

Instead try adding "packageTests" in the platform you want:
http://hg.mozilla.org/build/buildbotcustom/file/tip/misc.py#l603
> packageTests = pf.get('packageTests', packageTests)

This should have the same effect but without triggering packaged unittests.

I have filed bug 552430 to keep track of the work involved on uploading linux 64 tests for staging testing of unit tests on talos for Linux 64.

Comment 26

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to comment #25)
> (From update of attachment 432156 [details] [diff] [review])
> This patch is incorrect.
> If you look at:
> http://hg.mozilla.org/build/buildbotcustom/file/tip/misc.py#l598
> > elif pf.get('enable_opt_unittests'):
> >    packageTests = True
> You will see that you are enabling optimized unittests in all branches and all
> platforms. We should only turn optimized unitests were needed.

Actually, the way I read that is if you have enabled optimized unit tests, then you will upload the packaged test file, not the reverse (i.e. opt_unittest depends on having packaged tests uploaded).  The packageTests factory argument just enables the make package-tests step from my reading of the code (http://hg.mozilla.org/build/buildbotcustom/file/aed4cb0172ca/process/factory.py#l824)

> Instead try adding "packageTests" in the platform you want:
> http://hg.mozilla.org/build/buildbotcustom/file/tip/misc.py#l603
> > packageTests = pf.get('packageTests', packageTests)

Yes, we could do that.  As the purpose this bug is to stop running unit tests on build machines, we will need to have packaged unit tests for all platforms on all branches unless we want to keep certain branches/platforms on build machine unit tests.  This seems like the most appropriate default under the circumstances.

> This should have the same effect but without triggering packaged unittests.
> 
> I have filed bug 552430 to keep track of the work involved on uploading linux
> 64 tests for staging testing of unit tests on talos for Linux 64.

sure, that is fine.  I will put new things and move the conversation over there.

Comment 27

[Lukas Blakk [:lsblakk] use ?needinfo]

In testing this on talos-staging, I've installed mercurial 1.3.1 on talos-r3-fed-001 with:
  yum install mercurial

Comment 28

[Lukas Blakk [:lsblakk] use ?needinfo]

Status update:
http://hg.mozilla.org/users/lsblakk_mozilla.com/buildbotcustom/rev/7af72111c5d0
http://hg.mozilla.org/users/lsblakk_mozilla.com/buildbot-configs/rev/128ec338de4b

I'm getting the builders created, with the right slaves attached - now just need to get the sendchanges triggering the right builders eg: 'macosx' triggers leopard/snowleopard builders.

Comment 29

[John Ford [:jhford] CET/CEST Berlin Time]

for the geriatric machines I created a dictionary to map build factory platforms to test running platforms.  that logic is in nightlybuildfactory at the end of doupload() iirc

Comment 30

[Lukas Blakk [:lsblakk] use ?needinfo]

I installed hg on talos-r3-snow-001 through vnc from :

http://mercurial.berkwood.com/binaries/Mercurial-1.5-py2.6-macosx10.6.zip

so that I didn't have to install anything else (xcode, macports)

Comment 31

[Armen [:armenzg]]

Comment on attachment 431932 [details] [diff] [review]
sendchange-unitests being sent to talos-staging-master02

http://hg.mozilla.org/build/buildbot-configs/rev/2e782932a01c

Comment 32

[Lukas Blakk [:lsblakk] use ?needinfo]

In testing this on talos-staging, I've installed mercurial 1.3.1 on
talos-r3-fed64-001 with:
  yum install mercurial

Comment 33

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Buildbotcustom patch to add unittests to talos

Tested in talos-staging.

NOTE:

@@ -187,17 +188,17 @@ class MozillaBuildFactory(BuildFactory):
+         command=['bash', '-l','-c',

added this so that snowleopard could grab the correct PYTHONPATH from a local .bash_profile since both leopard and snowleopard use the same env.

Comment 34

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: staging configs adding unittests to Talos

Comment 35

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: production configs adding unittests to Talos

Comment 36

[Chris AtLee [:catlee]]

Comment on attachment 435034 [details] [diff] [review]
Buildbotcustom patch to add unittests to talos

>-def generateTalosBranchObjects(branch, branch_config, PLATFORMS, SUITES,
>-        factory_class=TalosFactory):
>+def generateTalosBranchObjects(branch, branch_config, PLATFORMS, SUITES, 
>+        ACTIVE_UNITTEST_PLATFORMS, factory_class=TalosFactory):
>     branchObjects = {'schedulers': [], 'builders': [], 'status': []}
>     branch_builders = []
>+    test_builders = []
> 
>     branchName = branch_config['branch_name']
>     buildBranch = branch_config['build_branch']
>     talosCmd = branch_config['talos_command']
> 
>     for platform, platform_config in PLATFORMS.items():
>         for slave_platform in platform_config['slave_platforms']:
>             for suite, talosConfig in SUITES.items():
>@@ -1725,22 +1731,62 @@ def generateTalosBranchObjects(branch, b
>                               treeStableTimer=0,
>                               builderNames=[builder['name']],
>                               numberOfBuildsToTrigger=tests,
>                               )
>                 branchObjects['schedulers'].append(s)
>                 branchObjects['builders'].append(builder)
>                 branch_builders.append(builder['name'])
> 
>+            if platform in ACTIVE_UNITTEST_PLATFORMS.keys():
>+                testTypes = []
>+
>+                if branch_config['platforms'][platform].get('enable_opt_unittests'):
>+                    testTypes.append('opt')
>+                if branch_config['platforms'][platform].get('enable_debug_unittests'):
>+                    testTypes.append('debug')
>+
>+                for test_type in testTypes:
>+                    test_builders = []
>+                    triggeredUnittestBuilders = []
>+
>+                    # create builder names for TinderboxMailNotifier
>+                    for suites_name, suites in branch_config['unittest_suites']:
>+                        test_builders.extend(generateTestBuilderNames(
>+                            '%s %s %s test' % (platform_name, branch, test_type), suites_name, suites))
>+
>+                    triggeredUnittestBuilders.append(('%s-%s-%s-unittest' % (branch, slave_platform, test_type), test_builders, True))
>+
>+                    for suites_name, suites in branch_config['unittest_suites']:
>+                        # Remove mochitest-a11y from test suites since the packaged builds are not
>+                        # built with a11y enabled
>+                        if platform.startswith("macosx") and 'mochitest-a11y' in suites:
>+                            # Create a new factory that doesn't have mochitest-a11y
>+                           suites = suites[:]
>+                           suites.remove('mochitest-a11y')

So, I realize this a11y code is a mess, but won't this result in builder names getting generated for mochitest-a11y in the block above, and then no builders created below?

>+
>+                        # create the builders
>+                        branchObjects['builders'].extend(generateTestBuilder(
>+                                branch_config, branch, platform, "%s %s %s test" % (platform_name, branch, test_type),
>+                                "%s-%s-%s-unittest" % (branch, slave_platform, test_type),
>+                                suites_name, suites, branch_config.get('mochitest_leak_threshold', None),
>+                                branch_config.get('crashtest_leak_threshold', None),
>+                                platform_config[slave_platform]['slaves']))
>+
>+                    for scheduler_name, test_builders, merge in triggeredUnittestBuilders:
>+                        scheduler_branch = ('%s-%s-%s-unittest' % (branch, platform, test_type))
>+                        branchObjects['schedulers'].append(NoMergeScheduler(name=scheduler_name, 
>+                            branch=scheduler_branch, builderNames=test_builders, treeStableTimer=0))

>@@ -5255,19 +5256,22 @@ class WinmoBuildFactory(MobileBuildFacto
> 
> class UnittestPackagedBuildFactory(MozillaBuildFactory):
>     def __init__(self, platform, test_suites, env={}, productName='firefox',
>             mochitest_leak_threshold=None, crashtest_leak_threshold=None,
>             totalChunks=None, thisChunk=None, chunkByDir=None, downloadSymbols=True,
>             **kwargs):
>         self.platform = platform.split('-')[0]
> 
>-        self.env = MozillaEnvironments['%s-unittest' % self.platform].copy()
>+        self.env = deepcopy(MozillaEnvironments['%s-unittest' % self.platform])
>         self.env['MINIDUMP_STACKWALK'] = getPlatformMinidumpPath(self.platform)
>         self.env.update(env)
>+        # copy in overwritten env vars
>+        for key, value in env.items():
>+            self.env[key] = value
>         self.productName = productName
> 

What problem did you run into that required a deepcopy of the environment?
self.env should be a shallow dictionary, with only strings as values, so the
original .copy() should have been sufficient to get an isolated copy of the
environment.

You've also left in 'self.env.update(env)', which does exactly what your new
for loop does.

Comment 37

[Chris AtLee [:catlee]]

Comment on attachment 435037 [details] [diff] [review]
production configs adding unittests to Talos

Looks good here too.  Are you planning on pushing macosx out as soon as it's ready?

Comment 38

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to comment #37)
> (From update of attachment 435037 [details] [diff] [review])
> Looks good here too.  Are you planning on pushing macosx out as soon as it's
> ready?

Yes, we'll roll out Leopard and Snow Leopard right away. Fedora will follow when we've got all bugs known and tracked.

Comment 39

[Armen [:armenzg]]

Alice I would like to know if these machines will stay in the long-term on staging for my site-staging.pp changes.
> - talos-r3-fed-001
> - talos-r3-fed64-001
> - talos-r3-snow-001
> - talos-r3-leopard-001

Comment 40

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Buildbotcustom patch to add unittests to talos

so it turns out I didn't need the deepcopy, and I took the unittest_suites into the per-platform config so that the a11y block in generateTalosBranchObjects is no longer needed either.

Comment 41

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: staging configs adding unittests to Talos

just moved the unittest_suites into the branch[platforms] so we can take a11y out for macosx in the configs instead of in misc.py

Comment 42

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: production configs adding unittests to Talos

same changes as in staging - also set reboot to 1 build

Comment 43

[alice nodelman [:alice] [:anode]]

Comment on attachment 435290 [details] [diff] [review]
Buildbotcustom patch to add unittests to talos

Does it look like 'bash -l -c' has any effect on any other platform?  It might be totally overkill, but we could also limit only running the -l switch on mac platforms.

Comment 44

[alice nodelman [:alice] [:anode]]

Comment on attachment 435292 [details] [diff] [review]
production configs adding unittests to Talos

I find the deepcopy stuff a little weird, but it seems that you are avoiding having to re-write env and test names and such for unit test per-platform over and over - and that I do approve of.

Comment 45

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Buildbotcustom patch to add unittests to talos

carrying forward alice's r+ -- I just cleaned up an extra space and the import of deepcopy which is no longer needed here.

Comment 46

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: staging configs adding unittests to Talos

now with reboots set to 1 to match production

Comment 47

[alice nodelman [:alice] [:anode]]

Comment on attachment 435304 [details] [diff] [review]
staging configs adding unittests to Talos

Ship it!

Comment 48

[Chris AtLee [:catlee]]

Comment on attachment 435303 [details] [diff] [review]
Buildbotcustom patch to add unittests to talos

We need to be able to have different suites of tests for opt/debug on the same platform.  For example, right now we want to run a11y tests on macosx debug, but not on macosx opt.

Can you adjust the config.py patch so that there's a default set of test suites that gets copied in for each platform, and override macosx opt does that it doesn't run a11y?

Also, the TinderboxMailNotifier needs to be using errorparser="unittest" for the unittest builders.

Comment 49

[Armen [:armenzg]]

Lukas you can run now talos-r3-leopard-001 if you want through staging to see that it works after the puppet changes.

I will get you the snow leopard machine when the mounting /N bug gets resolved (bug 555790).

Comment 50

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Buildbotcustom patch to add unittests to talos

Tested in staging - adds ability to have different suites of tests for opt/debug on the same platform.  Configs patch (to be attached next) has a default set of test suites that gets copied in for each platform, mac opt unittests are overridden to take out mochitest-a11y

Also, added a separate TinderboxMailNotifier so as to use errorparser="unittest" for the unittest builders.

Finally, in the configs I had set up ACTIVE_BRANCHES to make it easier to turn branches on and off through master.cfg but I realized that I had attached that to the ACTIVE_UNITTEST_PLATFORMS so I refactored that and in doing so added ACTIVE_UNITTEST_PLATFORMS to the generateTalosReleaseBranchObjects params since that gets called within generateTalosBranchObjects and then in turn calls generateTalosBranchObjects. This way it just cycles the platform's active Unittest platforms back around but won't create any additional builders.

Comment 51

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: staging configs adding unittests to Talos

flagging catlee for feedback on the changes specific to the new buildbotcustom handling of default unittests for both opt/debug

Comment 52

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: production configs adding unittests to Talos

flagging catlee for feedback just on the changes specific to the new handling of default unittests (same as staging)

Comment 53

[alice nodelman [:alice] [:anode]]

Current status:

- pushing hard to get leopard/snow leopard in production by end of week, currently blocked on puppet working on snow leopard + finalization of buildbot-configs
- winxp/win7 blocked on unitest packages as zips, blocked on installation of hg and getting successful staging runs
- fed/fed64 blocked on people (needs investigation of failures so that they can be properly farmed out to appropriate teams)

Current plan:
1. leopard/snow leopard working in production by end of week
2. after leopard/snow leopard up move on to the issues found during linux staging and run them down
3. get unit test packages as zips pushed to production
4. staging of win7/winxp machine

Comment 54

[Chris AtLee [:catlee]]

Comment on attachment 435733 [details] [diff] [review]
Buildbotcustom patch to add unittests to talos

>+                    for scheduler_name, test_builders, merge in triggeredUnittestBuilders:
>+                        scheduler_branch = ('%s-%s-%s-unittest' % (branch, platform, test_type))
>+                        branchObjects['schedulers'].append(NoMergeScheduler(name=scheduler_name, 
>+                            branch=scheduler_branch, builderNames=test_builders, treeStableTimer=0))
>+

What if merge is True here?  We'll want to use a regular scheduler in that case, right?

>-         command=['bash', '-c',
>+         command=['bash', '-l','-c',

Why do we need '-l' here?

Comment 55

[Chris AtLee [:catlee]]

Comment on attachment 435734 [details] [diff] [review]
staging configs adding unittests to Talos

jsreftest should be run on all platforms, not just macosx.

and mochitest-a11y won't be running on macosx-debug if I'm reading this correctly.

Comment 56

[Chris AtLee [:catlee]]

Comment on attachment 435736 [details] [diff] [review]
production configs adding unittests to Talos

same as staging

Comment 57

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Buildbotcustom patch to add unittests to talos

> What if merge is True here?  We'll want to use a regular scheduler in that
> case, right?

Funny enough merge is set to True by default in generateBranchObjects as well as in generateTalosBranchObjects so thanks for catching it.  I put in a regular Scheduler option same as in the former.

> 
> >-         command=['bash', '-c',
> >+         command=['bash', '-l','-c',
> 
> Why do we need '-l' here?

This is to source the local bash_profile which is needed on snow leopard.  It has a different PYTHONPATH than the leopard slaves and yet the two share the same env since those are per platform (macosx) and not per slave_platform (leopard, snowleopard).  Adding the -l doesn't affect slaves with no bash_profile override of PYTHONPATH so it seems like a safe thing to do.

Comment 58

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: staging configs adding unittests to Talos

Added jsreftest to default unittest set.

In staging I have tested the configs and mac-debug builds do get a mochitest-a11y step in the mochitest-other builder with these configs.

Comment 59

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: production configs adding unittests to Talos

same as for staging - added jsreftest to default unittest suites.

Comment 60

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Buildbotcustom patch to add unittests to talos

Catlee, disregard the comment about 'bash' '-l'  because it appears that armenzg worked around this issue in his puppet installation of mercurial for leopard/snow leopard.  I've tested in staging on puppet slaves without the -l and hg clone worked fine.

Also, in this patch (which still handles the merge/nomerge) I shortened the builddir prefix to -u instead of -unittest because we were having installdmg.sh "failed to mount" on the longer builddir-named builder mozilla-central-leopard-debug-unittest-mochitest-other and with a shorter builddir name it no longer had an issue.  Now all builddirs are along the lines of mozilla-central-leopard-debug-u-mochitest-other.

Comment 61

[Armen [:armenzg]]

(In reply to comment #57)
> 
> This is to source the local bash_profile which is needed on snow leopard.  It
> has a different PYTHONPATH than the leopard slaves and yet the two share the
> same env since those are per platform (macosx) and not per slave_platform
> (leopard, snowleopard).  Adding the -l doesn't affect slaves with no
> bash_profile override of PYTHONPATH so it seems like a safe thing to do.
Lukas, the bash_profile is using PYTHONPATH=/Library/Python/2.5/site-packages instead of PYTHONPATH=/usr/local/lib/python2.5/site-packages (since it was non-existent).

You might want to run your patch without -l and it should probably fine since both platforms now use the same PYTHONPATH.

Comment 62

[Armen [:armenzg]]

Talos darwin slaves now have mercurial installed.
We are ready puppet wise to turn unit tests on them.

Comment 63

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: staging configs adding unittests to Talos

taking a11y out of mac debug unittest suites until bug 524775 is fixed.

Comment 64

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: production configs adding unittests to Talos

taking debug a11y out of mac unittest suites until bug 524775 is fixed.

Comment 65

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: talos-pool amendments to accomodate packaged unittest changes in talos-r3

http://hg.mozilla.org/build/buildbot-configs/rev/b198f460d3b1

Comment 66

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 436692 [details] [diff] [review]
talos-pool amendments to accomodate packaged unittest changes in talos-r3

r+ from alice in IRC

Comment 67

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 436235 [details] [diff] [review]
staging configs adding unittests to Talos

http://hg.mozilla.org/build/buildbot-configs/rev/62ce31923cc0

Comment 68

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 436236 [details] [diff] [review]
production configs adding unittests to Talos

http://hg.mozilla.org/build/buildbot-configs/rev/02cfb6f3932d

Comment 69

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 436086 [details] [diff] [review]
Buildbotcustom patch to add unittests to talos

http://hg.mozilla.org/build/buildbotcustom/rev/d5ba95f1dcb8

Comment 70

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Turns on sendchange notification for talos-master from production-master

the last, very important, piece of config update.

Comment 71

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Comment on attachment 436697 [details] [diff] [review]
Turns on sendchange notification for talos-master from production-master

*stamp*

Comment 72

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 436697 [details] [diff] [review]
Turns on sendchange notification for talos-master from production-master

http://hg.mozilla.org/build/buildbot-configs/rev/9f1626be3147

Comment 73

[Armen [:armenzg]]

I filed bug 558777 to track everything wrt to Leopard and Snow Leopard.

I have transferred all related dependent bugs to it.

Comment 74

[Armen [:armenzg]]

I have filed bug 558781 to track everything wrt to running unit tests for Windows XP and Windows 7.

I have transferred all related dependent bugs to it.

Comment 75

[Armen [:armenzg]]

I have filed bug 558782 to track everything wrt to running unit tests for
Fedora and Fedora 64.

I have transferred all related dependent bugs to it.

Comment 76

[Armen [:armenzg]]

Bug 557294 will get us more Rev3 machines (post delivery imaging is also needed) which we need to enable unit tests on more branches than just mozilla-central. This will help us put our wait times back to normal on production-talos.

Comment 77

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

afaict, all the comments, patches and depbugs here are done. Which makes this bug officially FIXED!

(let me know if I missed anything!)

Comment 78

[Armen [:armenzg]]

The only thing left is to add XP debug unit tests in bug 614955 but I don't want to mess anymore with the dependencies.