524775 - Enable a11y and tests on OSX debug builds on accessibility branch

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Comment 2

[(no longer active)]

Can the priority on this bug be bumped?

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

That depends on how much there is to do here, and how important it is compared to other work as we approach the end of the quarter. I'm pretty sure we won't be able to get it this month.

Note that accessibility is still disabled by default on mac, and we don't test it on opt builds either. If we knew that mochitest-a11y ran green on m-c opt and debug builds that would help a lot.

Comment 4

[David Bolter [:davidb] (NeedInfo me for attention)]

I'm aware of one failure in test_combobox.xul which I can disable (by sniffing Mac in the test file).

Comment 5

[David Bolter [:davidb] (NeedInfo me for attention)]

OK, landed 551548. Tests currently run green on Mac.

Comment 6

[Chris AtLee [:catlee]]

Attachment: Move a11y test flag into platform section of config.py

Comment 7

[Chris AtLee [:catlee]]

Attachment: Move a11y test flag into platform section of config.py, and enable for macosx-debug most branches

This, combined with the other patch, results in enabling a11y tests on macosx-debug on mozilla-central, places, tracemonkey, and addons.  It actually disables a11y tests on the refcounting electrolysis builds, but those are going away soon anyway.

I found this snippet helpful to add at the bottom of master?.cfg:

import re
for b in c['builders']:
    print b['name'], b['factory'].__class__.__name__
    for s in b['factory'].steps:
        step_class, args = s
        args = re.sub(' (instance )?at 0x[0-9a-f]+>', '>', str(args))
        print "   ", step_class.__name__, args

then buildbot checkconfig spits out the list of factories and steps, and you can compare the two.

Comment 8

[David Bolter [:davidb] (NeedInfo me for attention)]

(In reply to comment #7)
> Created an attachment (id=432939) [details]
> This, combined with the other patch, results in enabling a11y tests on
> macosx-debug on mozilla-central, places, tracemonkey, and addons.

Does this automagically include try server?

Comment 9

[Chris AtLee [:catlee]]

(In reply to comment #8)
> (In reply to comment #7)
> > Created an attachment (id=432939) [details] [details]
> > This, combined with the other patch, results in enabling a11y tests on
> > macosx-debug on mozilla-central, places, tracemonkey, and addons.
> 
> Does this automagically include try server?

It will, once bug 520227 lands.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 432939 [details] [diff] [review]
Move a11y test flag into platform section of config.py, and enable for macosx-debug most branches

>diff --git a/mozilla2-staging/config.py b/mozilla2-staging/config.py
>             },
>+            'enable_a11y_tests': False,
>         },
>         'win32': {

Overall this patch looks fine to me, but why is a11y False on opt builds ? Shouldn't we restore it to what we had for the refcounting builds ?

mozconfig patch to follow to set 'ac_add_options --enable-accessibility' on opt and debug builds mac ?

Comment 11

[Chris AtLee [:catlee]]

(In reply to comment #10)
> (From update of attachment 432939 [details] [diff] [review])
> >diff --git a/mozilla2-staging/config.py b/mozilla2-staging/config.py
> >             },
> >+            'enable_a11y_tests': False,
> >         },
> >         'win32': {
> 
> Overall this patch looks fine to me, but why is a11y False on opt builds ?
> Shouldn't we restore it to what we had for the refcounting builds ?
> 
> mozconfig patch to follow to set 'ac_add_options --enable-accessibility' on opt
> and debug builds mac ?

My understanding is that we don't want accessibility enabled in the nightly builds.  Maybe David Bolter can confirm this.

Comment 12

[David Bolter [:davidb] (NeedInfo me for attention)]

(In reply to comment #11)
> (In reply to comment #10)
> > (From update of attachment 432939 [details] [diff] [review] [details])
> > >diff --git a/mozilla2-staging/config.py b/mozilla2-staging/config.py
> > >             },
> > >+            'enable_a11y_tests': False,
> > >         },
> > >         'win32': {
> > 
> > Overall this patch looks fine to me, but why is a11y False on opt builds ?
> > Shouldn't we restore it to what we had for the refcounting builds ?
> > 
> > mozconfig patch to follow to set 'ac_add_options --enable-accessibility' on opt
> > and debug builds mac ?
> 
> My understanding is that we don't want accessibility enabled in the nightly
> builds.  Maybe David Bolter can confirm this.

Confirmed. Our Mac a11y is not useful at this time due to VoiceOver interop issues.

Comment 13

[Chris AtLee [:catlee]]

Attachment: Add enable-accessibility to osx debug builds

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 433329 [details]
Add enable-accessibility to osx debug builds

IIRC the plan was to break the symlink on the 1.9.1 configs so that we don't enable accessibility there.

Comment 15

[Chris AtLee [:catlee]]

Haven't had time to work on this in a while, back in the pool!

Comment 16

[Armen [:armenzg]]

I can enable a11y for 10.5 on my work on bug 581213 and bug 580410. I am going to be tackling these 2 bugs immediately.

Can I proceed and enable it? Who could have a look at the results once it is running on staging?

Comment 17

[alexander :surkov (:asurkov)]

(In reply to comment #16)
> I can enable a11y for 10.5 on my work on bug 581213 and bug 580410. I am going
> to be tackling these 2 bugs immediately.
> 
> Can I proceed and enable it? Who could have a look at the results once it is
> running on staging?

David, can you do this?

Comment 18

[Armen [:armenzg]]

From the staging run a11y does not even start:
INFO | runtests.py | Server pid: 342
INFO | runtests.py | Websocket server pid: 343
INFO | runtests.py | Running tests: start.

pk12util: PKCS12 IMPORT SUCCESSFUL
INFO | automation.py | SSL tunnel pid: 351
INFO | automation.py | Application pid: 352
TEST-UNEXPECTED-FAIL | automation.py | application timed out after 330 seconds with no output
Can't trigger Breakpad, just killing process
INFO | automation.py | Application ran for: 0:05:30.016987
INFO | automation.py | Reading PID log: /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpmDvP6kpidlog
WARNING | automationutils.processLeakLog() | refcount logging is off, so leaks can't be detected!

10.5's log:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1281973061.1281974055.23598.gz
10.6's log:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1281971865.1281972944.18468.gz

If I have time I will give it a shot to run it manually. If the output is the same I will not be enabling a11y test suite for mac.

From the logs is there anything I am doing wrong? any special environment variable that should be added?

Comment 19

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found in triage.

(In reply to comment #17)
> (In reply to comment #16)
> > I can enable a11y for 10.5 on my work on bug 581213 and bug 580410. I am going
> > to be tackling these 2 bugs immediately.
> > 
> > Can I proceed and enable it? Who could have a look at the results once it is
> > running on staging?
> 
> David, can you do this?

David/surkov: 

1) who is the dev. contact to verify these tests before we enable them? 

2) sanity check: we do already run these tests on opt linux, linux64, win32 builds. However, comment#12 makes me wonder if these tests are of any use on mac?

Comment 20

[David Bolter [:davidb] (NeedInfo me for attention)]

I'm your guy. Armen, just grab me and I'll check your setup.

Comment 21

[Armen [:armenzg]]

Taking.

Can we look at it in a couple of weeks? There is just a whole load on me.

If not, we will look for someone who can take this before me.

Comment 22

[David Bolter [:davidb] (NeedInfo me for attention)]

sure no rush yet.

Comment 23

[David Bolter [:davidb] (NeedInfo me for attention)]

OK it has been a couple of weeks, and this bug is becoming a strong want.

Comment 24

[Armen [:armenzg]]

Time to get to it.

Comment 25

[Armen [:armenzg]]

Attachment: [mozilla-tests/config.py] enable a11y for mac [not tested]

Comment 26

[(no longer active)]

FWIW, if we're not planning to ship those bits to our users, we should probably have two types of Mac builds on Mac (which would obviously suck, but we need to run tests on the same bits which we would ship to users anyways.)

Comment 27

[David Bolter [:davidb] (NeedInfo me for attention)]

As per in-person discussion, let's do this on a new accessibility branch. Filed as bug 649351.

Comment 28

[David Bolter [:davidb] (NeedInfo me for attention)]

To be clear, this bug is still valid and needed, but depends on the branch bug now.

Comment 29

[Chris Cooper [:coop] (he/him)]

Let's file a new bug for the trunk work when we get there then.

Comment 30

[David Bolter [:davidb] (NeedInfo me for attention)]

OK so next we need the accessibility branch nightlies to have accessibility built in.

Adding this to mozconfig is how we do it locally:
ac_add_options --enable-accessibility

Comment 31

[Armen [:armenzg]]

davidb, could you land a customized mozconfig on your repo? [1]

It probably should be called mozconfig-extra-macosx64 with the extra line you mention.

This is the generic one that is being used:
http://hg.mozilla.org/build/buildbot-configs/raw-file/7bf87d4f860d/mozilla2/macosx64/generic/nightly/mozconfig

[1] https://wiki.mozilla.org/DisposableProjectBranches#Using_a_custom_mozconfig

Comment 32

[David Bolter [:davidb] (NeedInfo me for attention)]

Yeah I need to figure out how to make sure the extra mozconfig never gets merged to central.

Comment 33

[David Bolter [:davidb] (NeedInfo me for attention)]

OK, I went ahead and added the extra mozconfig:
http://hg.mozilla.org/projects/accessibility/rev/3a7ec7ea5b78

... but I don't think it is working. Can you find out if the enable-accessibility option is hardcoded somewhere for mac builds?

Comment 34

[Chris AtLee [:catlee]]

(In reply to comment #33)
> OK, I went ahead and added the extra mozconfig:
> http://hg.mozilla.org/projects/accessibility/rev/3a7ec7ea5b78
> 
> ... but I don't think it is working. Can you find out if the
> enable-accessibility option is hardcoded somewhere for mac builds?

It might need a clobber first.

Comment 35

[David Bolter [:davidb] (NeedInfo me for attention)]

(In reply to comment #34)
> (In reply to comment #33)
> > OK, I went ahead and added the extra mozconfig:
> > http://hg.mozilla.org/projects/accessibility/rev/3a7ec7ea5b78
> > 
> > ... but I don't think it is working. Can you find out if the
> > enable-accessibility option is hardcoded somewhere for mac builds?
> 
> It might need a clobber first.

Actually, it does seem to be enabled after all :)
(I need to update my accessibility diagnostic addon)

Comment 36

[Armen [:armenzg]]

Attachment: [buildbot-configs] add ability to add test suites for project_branches plus some cleanup

Besides adding a11y tests for mac I am doing some clean up.
I know we could optimize DEFAULT_TALOS_VALUES but mixing those values on SUITES was making this patch too big.

Comment 37

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 531337 [details] [diff] [review]
[buildbot-configs] add ability to add test suites for project_branches plus some cleanup

>diff --git a/mozilla-tests/config.py b/mozilla-tests/config.py
>--- a/mozilla-tests/config.py
>+++ b/mozilla-tests/config.py
>@@ -136,16 +136,41 @@ ALL_PLATFORMS = PLATFORMS['linux']['slav
>                 PLATFORMS['macosx64']['slave_platforms']
> 
> NO_WIN = PLATFORMS['macosx64']['slave_platforms'] + PLATFORMS['linux']['slave_platforms'] + PLATFORMS['linux64']['slave_platforms']
> 
> NO_MAC = PLATFORMS['linux']['slave_platforms'] + PLATFORMS['linux64']['slave_platforms'] + PLATFORMS['win32']['slave_platforms'] + PLATFORMS['win64']['slave_platforms']
> 
> ANDROID = PLATFORMS['android']['slave_platforms']
> 
>+DEFAULT_TALOS_VALUES = {
>+    'chrome':               (True, {}, ALL_PLATFORMS),
>+    'nochrome':             (True, {}, ALL_PLATFORMS),
>+    'dirty':                (True, TALOS_DIRTY_OPTS, ALL_PLATFORMS),
>+    'tp4':                  (True, TALOS_TP4_OPTS, ALL_PLATFORMS),
>+    'cold':                 (True, TALOS_DIRTY_OPTS, NO_WIN),
>+    'v8':                   (True, {}, ALL_PLATFORMS),
>+    'svg':                  (True, {}, ALL_PLATFORMS),
>+    'scroll':               (True, {}, ALL_PLATFORMS),
>+    'dromaeo':              (True, {}, ALL_PLATFORMS),
>+    'addon':                (False, TALOS_ADDON_OPTS, ALL_PLATFORMS),
>+    'addon-baseline':       (False, TALOS_BASELINE_ADDON_OPTS, ALL_PLATFORMS),
>+    'a11y':                 (True, {}, NO_MAC),
>+    'paint':                (True, {}, ALL_PLATFORMS),
>+    'remote-ts':            (True, TALOS_REMOTE_FENNEC_OPTS, ANDROID),
>+    'remote-tdhtml':        (True, TALOS_REMOTE_FENNEC_OPTS, ANDROID),
>+    'remote-tsvg':          (True, TALOS_REMOTE_FENNEC_OPTS, ANDROID),
>+    'remote-tsspider':      (True, TALOS_REMOTE_FENNEC_OPTS, ANDROID),
>+    'remote-tpan':          (True, TALOS_REMOTE_FENNEC_OPTS, ANDROID),
>+    'remote-tp4m':          (True, TALOS_REMOTE_FENNEC_OPTS, ANDROID),
>+    'remote-tp4m_nochrome': (True, TALOS_REMOTE_FENNEC_OPTS, ANDROID),
>+    'remote-twinopen':      (True, TALOS_REMOTE_FENNEC_OPTS, ANDROID),
>+    'remote-tzoom':         (True, TALOS_REMOTE_FENNEC_OPTS, ANDROID),
>+}
>+
> # these three are for mozilla-1.9.1 and mozilla-1.9.2
> OLD_BRANCH_ALL_PLATFORMS = PLATFORMS['linux']['slave_platforms'] + \
>                 PLATFORMS['win32']['slave_platforms'] + \
>                 PLATFORMS['macosx']['slave_platforms']
> 
> OLD_BRANCH_NO_WIN = PLATFORMS['macosx']['slave_platforms'] + PLATFORMS['linux']['slave_platforms']
> 
> OLD_BRANCH_NO_MAC = PLATFORMS['linux']['slave_platforms'] + PLATFORMS['win32']['slave_platforms']
>@@ -219,16 +244,34 @@ def removeSuite(suiteName, suiteList):
>         name, suites = info
>         # Let's see if suiteName is on this list of suites
>         if suiteName in suites:
>             suites = suites[:]
>             suites.remove(suiteName)
>             suiteList[i] = (name, suites)
>     return suiteList
> 
>+def addSuite(suiteGroupName, newSuiteName, suiteList):
>+    # In UNITTEST_SUITES we have opt, debug and mobile unit tests keys.
>+    # Each one of these have a list of tuples of test suites.
>+    #     e.g. suiteGroup = ('reftest', ['reftest])
>+    newSuiteList = []
>+    added = False
>+    for tuple in suiteList:
>+        name, suites = tuple
>+        if suiteGroupName == name:
>+            suites.append(newSuiteName)
>+            added = True
>+        newSuiteList.append((name, suites))

Check your indenting here, a non-match gets added twice ^ and below too.

>+
>+    if not added:
>+        newSuiteList.append((name, suites))
>+
>+    return newSuiteList
>+



> # You must define opt_unittest_suites when enable_opt_unittests is True for a 
> # platform. Likewise debug_unittest_suites for enable_debug_unittests
> PLATFORM_UNITTEST_VARS = {
>         'linux': {
>             'builds_before_reboot': 1,
>             'unittest-env' : {'DISPLAY': ':0'},
>             'enable_opt_unittests': True,
>             'enable_debug_unittests': True,
>@@ -869,61 +912,69 @@ BRANCHES['try']['platforms']['macosx64']
> BRANCHES['try']['platforms']['macosx64']['snowleopard']['opt_unittest_suites'] += [('jetpack', ['jetpack'])]
> BRANCHES['try']['platforms']['macosx64']['leopard']['opt_unittest_suites'] += [('jetpack', ['jetpack'])]
> BRANCHES['try']['platforms']['win32']['xp']['opt_unittest_suites'] += [('jetpack', ['jetpack'])]
> BRANCHES['try']['platforms']['win32']['xp']['debug_unittest_suites'] += [('jetpack', ['jetpack'])]
> BRANCHES['try']['platforms']['win32']['win7']['opt_unittest_suites'] += [('jetpack', ['jetpack'])]
> BRANCHES['try']['platforms']['win32']['win7']['debug_unittest_suites'] += [('jetpack', ['jetpack'])]
> BRANCHES['try']['platforms']['android']['enable_opt_unittests'] = True
> 
>-######## generic branch variables for project branches
>-for branch in ACTIVE_PROJECT_BRANCHES:
>-    branchConfig = PROJECT_BRANCHES[branch]
>+def loadDefaultValues(BRANCHES, branch, branchConfig):

I'd like this to still be called PROJECT_BRANCHES so that it's not confused with something that would run on all branches.

>     BRANCHES[branch]['repo_path'] = branchConfig.get('repo_path', 'projects/' + branch) 
>     BRANCHES[branch]['branch_name'] = branchConfig.get('branch_name', branch.title())
>     BRANCHES[branch]['mobile_branch_name'] = branchConfig.get('mobile_branch_name', branch.title())
>     BRANCHES[branch]['build_branch'] = branchConfig.get('build_branch', branch.title())
>     BRANCHES[branch]['talos_command'] = branchConfig.get('talos_cmd', TALOS_CMD)
>     BRANCHES[branch]['fetch_symbols'] = branchConfig.get('fetch_symbols', True)
>     BRANCHES[branch]['support_url_base'] = branchConfig.get('support_url_base', 'http://build.mozilla.org/talos')
>     BRANCHES[branch]['enable_unittests'] = branchConfig.get('enable_unittests', True)
>     # Check if Talos is enabled, if False, set 0 runs for all suites
>     if branchConfig.get('enable_talos') == False:
>         branchConfig['talos_suites'] = {}
>         for suite in SUITES.keys():
>             branchConfig['talos_suites'][suite]  = 0
>+
>+def loadCustomTalosSuites(DEFAULT_TALOS_VALUES, branchConfig):
>     # Want to turn on/off a talos suite? Set it in the PROJECT_BRANCHES[branch]['talos_suites'] otherwise, defaults below
>     if branchConfig.get('talos_suites'):
>         talosConfig = branchConfig['talos_suites']
>     else:
>+        # This is the default and will make all talosConfig.get(key,0) calls
>+        # to default to 0 a.k.a. disabled suite
>         talosConfig = {}
>+
>     for suite in SUITES.keys():
>-        if suite.startswith('remote-'):
>-            BRANCHES[branch][suite + '_tests'] = (talosConfig.get(suite, 0), True, TALOS_REMOTE_FENNEC_OPTS, ANDROID)
>-        else:
>-            # 0 runs by default
>-            if suite in ('v8', 'addon', 'addon-baseline', 'cold'):
>-                if suite == 'cold':
>-                    BRANCHES[branch][suite + '_tests'] = (talosConfig.get(suite, 0), True, TALOS_DIRTY_OPTS, ALL_PLATFORMS)
>-                elif suite == 'addon':
>-                    BRANCHES[branch][suite + '_tests'] = (talosConfig.get(suite, 0), False, TALOS_ADDON_OPTS, ALL_PLATFORMS)
>-                elif suite == 'addon-baseline':
>-                     BRANCHES[branch][suite + '_tests'] = (talosConfig.get(suite, 0), False, TALOS_BASELINE_ADDON_OPTS, ALL_PLATFORMS)
>-                else:
>-                    BRANCHES[branch][suite + '_tests'] = (talosConfig.get(suite, 0), True, {}, ALL_PLATFORMS)
>-            else:
>-                # default is 1 run
>-                if suite == 'dirty':
>-                    BRANCHES[branch][suite + '_tests'] = (talosConfig.get(suite, 1), True, TALOS_DIRTY_OPTS, ALL_PLATFORMS)
>-                elif suite == 'tp4':
>-                    BRANCHES[branch][suite + '_tests'] = (talosConfig.get(suite, 1), True, TALOS_TP4_OPTS, ALL_PLATFORMS)
>-                elif suite == 'a11y':
>-                    BRANCHES[branch][suite + '_tests'] = (talosConfig.get(suite, 1), True, {}, NO_MAC)
>-                else:
>-                    BRANCHES[branch][suite + '_tests'] = (talosConfig.get(suite, 1), True, {}, ALL_PLATFORMS)
>+        BRANCHES[branch][suite + '_tests'] = (talosConfig.get(suite, 0), ) + DEFAULT_TALOS_VALUES[suite]
>+
>+def loadCustomUnittestSuites(BRANCHES, branch, branchConfig):
>+    # If you want a project branch to have a different set of unit tests you can
>+    # do the following:
>+    #  - add a key called "add_test_suites"
>+    #  - add a tuple for each test suite with the following format:
>+    #      ('OS_nick', 'platform', 'opt|debug', 'new or existing group', 'suite name')
>+    #      e.g. ('macosx64', 'snowleopard', 'debug', 'mochitest-other', 'a11y')
>+    #
>+    # Old way of adding suites but still the same format
>+    #    BRANCHES['mozilla-central']['platforms']['win32']['win7']['debug_unittest_suites'] \
>+    #        += [('jetpack', ['jetpack'])]
>+    #
>+    for suiteToAdd in branchConfig.get('add_test_suites', []):
>+        type = 'opt_unittest_suites' if suiteToAdd[2] == 'opt' else 'debug_unittest_suites'
>+        # 'debug_unittest_suites' or 'opt_unittest_suites' is a list of tuple
>+        # addSuite() modifies that list and returns a new one with the added suite
>+        BRANCHES[branch]['platforms'][suiteToAdd[0]][suiteToAdd[1]][type] = \
>+            addSuite( suiteGroupName=suiteToAdd[3], newSuiteName=suiteToAdd[4],
>+                      suiteList=BRANCHES[branch]['platforms'][suiteToAdd[0]][suiteToAdd[1]][type])

Can you attach a dump_masters output to show that this works?

>+
>+######## generic branch variables for project branches
>+for branch in ACTIVE_PROJECT_BRANCHES:
>+    branchConfig = PROJECT_BRANCHES[branch]
>+    loadDefaultValues(BRANCHES, branch, branchConfig)
>+    loadCustomTalosSuites(DEFAULT_TALOS_VALUES, branchConfig)
>+    loadCustomUnittestSuites(BRANCHES, branch, branchConfig)
> 
> if __name__ == "__main__":
>     import sys, pprint, re
> 
>     class BBPrettyPrinter(pprint.PrettyPrinter):
>         def format(self, object, context, maxlevels, level):
>             if isinstance(object, WithProperties):
>                 return pprint.PrettyPrinter.format(self, object.fmtstring, context, maxlevels, level)
>diff --git a/mozilla/project_branches.py b/mozilla/project_branches.py
>--- a/mozilla/project_branches.py
>+++ b/mozilla/project_branches.py
>@@ -11,16 +11,19 @@ PROJECT_BRANCHES = {
>             'tp4': 0,
>             'chrome': 0,
>             'nochrome': 0,
>             'dromaeo': 0,
>             'svg': 0,
>             'scroll': 0,
>             'paint': 0,
>         },
>+        'add_test_suites': [
>+            ('macosx64', 'snowleopard', 'opt', 'mochitest-other', 'mochitest-a11y'),
>+        ]
>     },
>     'build-system': {
>         'enable_talos': False,
>     },
>     'devtools':{
>         'enable_nightly': True,
>         # need both of these to turn off mobile completely because of key in generic config.py
>         'enable_mobile': False,

Comment 38

[Armen [:armenzg]]

The only changes I have done is to move the functions way at the top and make changed a variable from 'branch' to 'projectBranch' to make it more obvious.

(In reply to comment #37)
> >+def addSuite(suiteGroupName, newSuiteName, suiteList):
> >+    # In UNITTEST_SUITES we have opt, debug and mobile unit tests keys.
> >+    # Each one of these have a list of tuples of test suites.
> >+    #     e.g. suiteGroup = ('reftest', ['reftest])
> >+    newSuiteList = []
> >+    added = False
> >+    for tuple in suiteList:
> >+        name, suites = tuple
> >+        if suiteGroupName == name:
> >+            suites.append(newSuiteName)
> >+            added = True
> >+        newSuiteList.append((name, suites))
> 
> Check your indenting here, a non-match gets added twice ^ and below too.
> 
> >+
> >+    if not added:
> >+        newSuiteList.append((name, suites))
> >+
> >+    return newSuiteList
> >+
I am looking at the indenting and I don't see it wrong.
If the suite gets added on the loop then we don't execute the following if statement.
If it has not been added then we run the if statement to append it at the end. 

> Can you attach a dump_masters output to show that this works?
For sure :) I will get it tomorrow.

Comment 39

[Armen [:armenzg]]

Attachment: [buildbot-configs] add ability to add test suites for project_branches plus some cleanup

And the new attachment! :)

Comment 40

[Armen [:armenzg]]

Comment on attachment 531750 [details] [diff] [review]
[buildbot-configs] add ability to add test suites for project_branches plus some cleanup

I am fixing a couple of things. I will attach the newest in an hour or early tomorrow.

Comment 41

[Armen [:armenzg]]

davidb, how does this look?

There are hundreds of oranges there.

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1305320283.1305322036.6286.gz&fulltext=1

Comment 42

[David Bolter [:davidb] (NeedInfo me for attention)]

Current plan (based on discussion with Armen and Raphael):
Investigate cost of building accessibility into our Mac releases, but pref'ed off.
Then for a11y mochitests the pref would be flipped on. Armen is that last part already easily doable with our test infrastructure?

Comment 43

[Armen [:armenzg]]

(In reply to comment #42)
> Then for a11y mochitests the pref would be flipped on. Armen is that last
> part already easily doable with our test infrastructure?
Easily doable. It has been done several times for other teams.

Comment 44

[Armen [:armenzg]]

Attachment: [buildbot-configs] add ability to add test suites for project_branches plus some cleanup

I used dump_master.py to verify that only a11y gets added to mochitest-other on the a11y branch for both opt and debug builds.

I know it is a big patch but it is pretty much a no-op + ability to add unit tests from project branches.

I could optimize this patch even more but I feel that this step can be taken and let it sink without bit rotting other people's work.

Comment 45

[Armen [:armenzg]]

(In reply to comment #42)
> Current plan (based on discussion with Armen and Raphael):
> Investigate cost of building accessibility into our Mac releases, but
> pref'ed off.
> Then for a11y mochitests the pref would be flipped on. Armen is that last
> part already easily doable with our test infrastructure?

dbolter are you tracking that work on any bug? I would like to add it to the dependency.

Comment 46

[David Bolter [:davidb] (NeedInfo me for attention)]

Done. Bug 661740. Thanks for the ping Armen.

Comment 47

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 536996 [details] [diff] [review]
[buildbot-configs] add ability to add test suites for project_branches plus some cleanup

awesome - i'm glad we'll have more control over project branch unittests with this and thanks for the documentation in there as well.

Comment 48

[Armen [:armenzg]]

Comment on attachment 536996 [details] [diff] [review]
[buildbot-configs] add ability to add test suites for project_branches plus some cleanup

http://hg.mozilla.org/build/buildbot-configs/rev/fb8a29ea0773
http://hg.mozilla.org/build/buildbot-configs/rev/9047ccca3717

I landed adding the ability to add test suites from project_branches from using the feature for accessibility branch.
Just for the sake of making it clear.

I also added "tp" to the list of valid suites.

Comment 49

[Armen [:armenzg]]

Attachment: [mozconfigs] enable accessibility for debug builds

We are not releasing the debug bits so it would not matter to get debug builds with accessibility enabled by default.

This gives us the ability to run a11y tests on debug builds.

This is in parallel to bug 661740.

Comment 50

[David Bolter [:davidb] (NeedInfo me for attention)]

Comment on attachment 539218 [details] [diff] [review]
[mozconfigs] enable accessibility for debug builds

r=me thanks!

Comment 51

[Armen [:armenzg]]

Comment on attachment 539218 [details] [diff] [review]
[mozconfigs] enable accessibility for debug builds

This has now landed.

The next scheduled reconfigure should get this picked up.
http://hg.mozilla.org/build/buildbot-configs/rev/9b927148d652

Comment 52

[Armen [:armenzg]]

Attachment: add accessibility mozconfigs

Right now davidb has a mozconfig extra file on his branch to get accessibility enabled but if he merges to mozilla-central he would be enabling it eventually for every branch when he is not quite ready.

Shall we use this patch? or wait for bug 558180?

Comment 53

[Armen [:armenzg]]

Attachment: add accessibility mozconfigs

I had added the wrong patch.

I have updated the nightly channel updates and I have left the mobile desktop mozconfigs out as they are not in use.

Comment 54

[Chris AtLee [:catlee]]

Comment on attachment 542824 [details] [diff] [review]
add accessibility mozconfigs

You need corresponding mozconfigs in mozilla2-staging, right?

Comment 55

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to comment #54)
> Comment on attachment 542824 [details] [diff] [review] [review]
> add accessibility mozconfigs
> 
> You need corresponding mozconfigs in mozilla2-staging, right?

didn't we deprecate those mozconfigs?

Comment 56

[Armen [:armenzg]]

(In reply to comment #55)
> (In reply to comment #54)
> > Comment on attachment 542824 [details] [diff] [review] [review] [review]
> > add accessibility mozconfigs
> > 
> > You need corresponding mozconfigs in mozilla2-staging, right?
> 
> didn't we deprecate those mozconfigs?

I believe we stopped using them since we don't have staging masters but preproduction ones (and we use the mozconfigs under mozilla2/ for that).

catlee, r+ under that light?

Comment 57

[Chris AtLee [:catlee]]

Comment on attachment 542824 [details] [diff] [review]
add accessibility mozconfigs

yeah, looks fine otherwise.

Comment 58

[Armen [:armenzg]]

Comment on attachment 542824 [details] [diff] [review]
add accessibility mozconfigs

This will be live on the next scheduled reconfig.
http://hg.mozilla.org/build/buildbot-configs/rev/28fbbfce7c17

Comment 59

[Armen [:armenzg]]

Comment on attachment 542824 [details] [diff] [review]
add accessibility mozconfigs

This is now on production:
http://hg.mozilla.org/build/buildbot-configs/rev/2b1e7533a8c1

We now have mozconfig files exclusive to the accessibility project [1], hence, we won't need the mozconfig-extra-macosx64 file anymore [2].

I believe everything on this bug is done.

[1]
armenzg-laptop $ grep -r accessibility mozilla2/*/accessibility
mozilla2/linux/accessibility/nightly/mozconfig:ac_add_options --enable-update-channel=nightly-accessibility
mozilla2/linux/accessibility/nightly-rpm/mozconfig:ac_add_options --enable-update-channel=nightly-accessibility
mozilla2/linux64/accessibility/nightly-rpm/mozconfig:ac_add_options --enable-update-channel=nightly-accessibility
mozilla2/macosx/accessibility/debug/mozconfig:ac_add_options --enable-accessibility
mozilla2/macosx/accessibility/unittest/mozconfig:ac_add_options --enable-accessibility
mozilla2/macosx64/accessibility/debug/mozconfig:ac_add_options --enable-accessibility
mozilla2/macosx64/accessibility/nightly/mozconfig:ac_add_options --enable-update-channel=nightly-accessibility
mozilla2/macosx64/accessibility/nightly/mozconfig:ac_add_options --enable-accessibility

[2] http://hg.mozilla.org/projects/accessibility/file/3a7ec7ea5b78/mozconfig-extra-macosx64

Comment 60

[Armen [:armenzg]]

davidb could you strip the mozconfig-extra-macosx64 from your repository?

I belive "hg rm mozconfig-extra-macosx64 && hg commit" should be good enough.

Comment 61

[Armen [:armenzg]]

Thanks davidb!
http://hg.mozilla.org/projects/accessibility/rev/201aa2b03a65

I will verify the results in the morning and close this bug.

Comment 62

[Armen [:armenzg]]

I see mochitest-other running with a11y.

I see "ac_add_options --enable-accessibility" in both Mac64 logs.

I also see:
cp configs/mozilla2/macosx64/accessibility/debug/mozconfig .mozconfig
cp configs/mozilla2/macosx64/accessibility/nightly/mozconfig .mozconfig

I believe all issues have been dealt with.
You can merge to mozilla-central without pushing any mozconfig changes to it.

Also note that accessibility is enabled in *all* mac debug builds for *all* project branches.
For optimized builds *only* the accessibility branch has accessibility enabled for Mac optimized builds. This could cause problems when merging back to mozilla-central but you also have the ability of pushing first to the try server to double check.

Let me know if you have any questions.