1359595 - enable CI tests for DevEdition on mozilla-beta

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Description

[[e-mail] (:bhearsum)]

We'll be enabling builds for DevEdition on mozilla-beta by setting them up as separate "platforms" in buildbot and Taskcluster (eg: win32-devedition). We'll also need unit tests and talos for them. Some notable things:
- Builds will upload to https://archive.mozilla.org/pub/devedition/tinderbox-builds/
- We want to run all the same tests on DevEdition that we do on Firefox on mozilla-beta
- They need to show up separately on Treeherder

Builds are being worked on in bug 1358601.

Comment 1

[[e-mail] (:bhearsum)]

Kim, you said that you might be able to find someone to help with this? I'm hoping the builds will be ready in a day or two.

Comment 2

[Kim Moir [:kmoir] ET]

Sebastian, is this something that you might be interested in working on?  I can meet with you on vidyo to discuss the scope of the work if you are interested.

Comment 3

[Sebastian Pacurar [:spacurar]]

Yes, I am interested. We can talk on irc after the Releng AllTeam meeting to discuss when we can make the call. Since the bug needs to be done in a day or two as stated by :bhearsum we might be able to make the call right after the AllTeam meeting.

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Sebastian Pacurar [:spacurar] from comment #3)
> Yes, I am interested. We can talk on irc after the Releng AllTeam meeting to
> discuss when we can make the call. Since the bug needs to be done in a day
> or two as stated by :bhearsum we might be able to make the call right after
> the AllTeam meeting.

Works for me!

Comment 5

[Kim Moir [:kmoir] ET]

Sebastian, so as a first step you need to add new platforms Ben added in 

https://bug1358601.bmoattachments.org/attachment.cgi?id=8861601

to buildbot/mozilla-tests/config.py

for the platforms the tests run on (slave_platforms), they can use the existing ones for win32, win64 and macosx64

You will also have to update mozilla-tests/production_config.py

You might also need to update this file in puppet 
modules/buildmaster/templates/BuildSlaves-tests.py.erb 

Test those patches on your dev-master and ensure that they work.  The next step is to enable the same tests for these new platforms as exist for the regular builds on mozilla-beta.

This bug is an an example of similar work but for talos only and the builds run on taskcluster, not buildbot. 

https://bugzilla.mozilla.org/show_bug.cgi?id=1338871

Comment 6

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_adding-macox64-devedition.patch

The patch applied on the dev test-master returns an error: KeyError: 'snowleopard_devedition'. It happens at line 2617:

if platform not in ('linux', 'linux64'):
->              branch['platforms'][platform][slave_platform]['debug_unittest_suites'] += \
                    MOCHITEST_DT_8_E10S + MOCHITEST_E10S + REFTEST_E10S_TWO_CHUNKS
                branch['platforms'][platform][slave_platform]['opt_unittest_suites'] += REFTEST_E10S
                if slave_platform != 'yosemite_r7':
                    branch['platforms'][platform][slave_platform]['debug_unittest_suites'] += \
                        REFTEST_NOACCEL_E10S_TWO_CHUNKS
                    branch['platforms'][platform][slave_platform]['opt_unittest_suites'] += \
                        REFTEST_NOACCEL_E10S


Note that I also added macosx64-devedition at mozilla-esr45 branch, otherwise I was receiving the same KeyError here:

## Tests that only run on ESR45 ###
for branch in BRANCHES.keys():
    if branch == 'mozilla-esr45':
        for platform in PLATFORMS.keys():
            if platform not in ['linux', 'linux64', 'linux64-asan']:
                continue
            for slave_platform in PLATFORMS[platform]['slave_platforms']:
                if slave_platform in BRANCHES[branch]['platforms'][platform]:
                    BRANCHES[branch]['platforms'][platform][slave_platform]['debug_unittest_suites'] += \
                        REFTEST_NOACCEL_FOUR_CHUNKS + REFTEST_FOUR_CHUNKS
                    BRANCHES[branch]['platforms'][platform][slave_platform]['opt_unittest_suites'] += \
                        REFTEST_NOACCEL_TWO_CHUNKS + REFTEST_TWO_CHUNKS
        for platform in PLATFORMS.keys():
->          for slave_platform in PLATFORMS[platform]['slave_platforms']:
                if slave_platform in BRANCHES[branch]['platforms'][platform]:
                    # Use the unchunked mochitest-gl on ESR45 due to leaks when chunked
                    for test_type in ('debug_unittest_suites', 'opt_unittest_suites'):
                        for item in BRANCHES[branch]['platforms'][platform][slave_platform][test_type]:
                            if item[0] == 'mochitest-gl':
                                BRANCHES[branch]['platforms'][platform][slave_platform][test_type].remove(item)
                        BRANCHES[branch]['platforms'][platform][slave_platform][test_type]+= MOCHITEST_WEBGL

Comment 7

[Kim Moir [:kmoir] ET]

Comment on attachment 8862503 [details] [diff] [review]
bug_1359595_adding-macox64-devedition.patch

Looks good so far.

You don't need 
"Rev4 MacOSX Snow Leopard 10.6 devedition"

It's only used for esr

We will only run tests on yosemite_r7 for this bug

I think you need to update this file as well
mozilla-tests/BuildSlaves.py.template

Comment 8

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_v2_adding-macox64-devedition-buildbot.patch

I finally made it work, and got some diffs from it, even though the diffs look totally different from how it should be. Looking forward to refactor the code to work only for mozilla-beta.

I still can't find a way to stop recieving a KeyError regarding mozilla-esr45, so I left 'macosx64-devedition' there, in the "platforms" until I find a way to avoid that.

Comment 9

[Sebastian Pacurar [:spacurar]]

Attachment: diff for bug_1359595_v2_adding-macox64-devedition-buildbot.diff

Comment 10

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_adding-macox64-devedition-puppet.patch

Comment 11

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_v3_adding-macox64-devedition-buildbot.patch

Found a way to get rid of the KeyError: 'macosx64-devedition' regarding mozilla-esr45. They are out of the diff now.

Comment 12

[Sebastian Pacurar [:spacurar]]

Attachment: diff for bug_1359595_v3_adding-macox64-devedition-buildbot.diff

Comment 13

[Kim Moir [:kmoir] ET]

Comment on attachment 8862905 [details] [diff] [review]
bug_1359595_v3_adding-macox64-devedition-buildbot.patch

To avoid the key error you need to update buildbot-configs/mozilla/BuildSlaves.py.template

So I think the next steps are do to that and change the patch so the tests only run on beta.

Comment 14

[Alin Selagea [:aselagea]]

Comment on attachment 8862857 [details] [diff] [review]
bug_1359595_adding-macox64-devedition-puppet.patch

Landed on default: https://hg.mozilla.org/build/puppet/rev/9897e078ee88

Comment 15

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_v4_adding-macox64-devedition-buildbot.patch

Comment 16

[Sebastian Pacurar [:spacurar]]

Attachment: diff for bug_1359595_v4_adding-macox64-devedition-buildbot.diff

Comment 17

[Kim Moir [:kmoir] ET]

Comment on attachment 8864093 [details] [diff] [review]
bug_1359595_v4_adding-macox64-devedition-buildbot.patch

looking better :-)

-# we only need to load SETA config on test scheduler master
-# it doesn't impact other masters
+#we only need to load SETA config on test scheduler master
+#it doesn't impact other masters

I don't know why the space is removed, we need the space for lint tests

why is this change required?

# Copy project branches into BRANCHES keys
@@ -2517,6 +2669,8 @@ for branch in BRANCHES.keys():
                         REFTEST_NOACCEL_TWO_CHUNKS + REFTEST_TWO_CHUNKS
         for platform in PLATFORMS.keys():
             for slave_platform in PLATFORMS[platform]['slave_platforms']:
+                if platform not in BRANCHES[branch]['platforms'].keys():
+                    continue
                 if slave_platform in BRANCHES[branch]['platforms'][platform]:
                     # Use the unchunked mochitest-gl on ESR45 due to leaks when chunked
                     for test_type in ('debug_unittest_suites', 'opt_unittest_suites'):
@@ -3081,6 +3235,15 @@ for branch in BRANCHES.keys():
     BRANCHES[branch]['platforms']['linux64-stylo']['talos_slave_platforms'] = []


mozilla-tests/BuildSlaves.py.template should be updated instead of mozilla/BuildSlaves.py.template

Comment 18

[Sebastian Pacurar [:spacurar]]

(In reply to Kim Moir [:kmoir] from comment #17)
> Comment on attachment 8864093 [details] [diff] [review]
> bug_1359595_v4_adding-macox64-devedition-buildbot.patch
> 
> looking better :-)
> 
> -# we only need to load SETA config on test scheduler master
> -# it doesn't impact other masters
> +#we only need to load SETA config on test scheduler master
> +#it doesn't impact other masters
> 
> I don't know why the space is removed, we need the space for lint tests

Sorry about the comments, that can be easily fixed.

> 
> why is this change required?
> 
> # Copy project branches into BRANCHES keys
> @@ -2517,6 +2669,8 @@ for branch in BRANCHES.keys():
>                          REFTEST_NOACCEL_TWO_CHUNKS + REFTEST_TWO_CHUNKS
>          for platform in PLATFORMS.keys():
>              for slave_platform in PLATFORMS[platform]['slave_platforms']:
> +                if platform not in BRANCHES[branch]['platforms'].keys():
> +                    continue
>                  if slave_platform in
> BRANCHES[branch]['platforms'][platform]:
>                      # Use the unchunked mochitest-gl on ESR45 due to leaks
> when chunked
>                      for test_type in ('debug_unittest_suites',
> 'opt_unittest_suites'):
> @@ -3081,6 +3235,15 @@ for branch in BRANCHES.keys():
>      BRANCHES[branch]['platforms']['linux64-stylo']['talos_slave_platforms']
> = []
> 
> 
> mozilla-tests/BuildSlaves.py.template should be updated instead of
> mozilla/BuildSlaves.py.template

I added the conditional in order to avoid the KeyError for macosx64-devedition regarding mozilla-esr45. The problem is here -> BRANCHES[branch]['platforms'][platform]  
This happens for win32/64-devedition as well, so the conditional avoids the KeyError for windows as well. I also modified the BuildSlaves.py.template but the error keeps being thrown. 

Will update mozilla-tests/BuildSlaves.py.template.

Comment 19

[Kim Moir [:kmoir] ET]

The key error might be occurring because you need to update

your_path_to_your_build_master/master/BuildSlaves.py

with the new platform key

The template file allows the master to be updated automatically by puppet. However, if you are running a development master and add a new platform, it needs to be updated manually

Comment 20

[Sebastian Pacurar [:spacurar]]

If you look here: https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla-tests/config.py#45-54  you will notice that there are several platform types listed there. The reason I used that conditional is to avoid adding macosx64/win32/win64-devedition platforms to mozilla-esr45. I tested it that way and indeed it works because there won't be any KeyError anymore, but that means macosx64-devedition will be enabled for mozilla-esr45 branch as well and we want it only on mozilla-beta.

I also tried to add yosemite_r7_devedition to Buildslaves.py but the error is because macosx64-devedition is not present in the mozilla-esr45 platforms snippet I added above.

Comment 21

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_v5_adding-macox64-devedition-buildbot.patch

Fixed the comment spacing lines, and updated mozilla-tests/BuildSlaves.py.template.

mozilla/BuildSlaves.py.template has been reverted. Shouldn't we add yosemite_r7_devedition there as well?

Comment 22

[Kim Moir [:kmoir] ET]

Comment on attachment 8864197 [details] [diff] [review]
bug_1359595_v5_adding-macox64-devedition-buildbot.patch

These look better.

Did your run ./test-masters.sh?  

I added these patches to my master and the new builders don't appear on it. Not sure why, will look again in the morning.

Are you planning to add the other two platforms before deploying?

As a side note, if these new tests require treeherder changes it would be good to get those changes on file since treeherder is not deployed everyday, more like every week or so.

Comment 23

[Sebastian Pacurar [:spacurar]]

Yes I already tested it with windows 64 and it works. I keep having some problems with windows 32. There seems to be KeyErrors regarding win7_vm_devedition. Right now I backed-up the patch with windows 32 and will add "devedition" to tests from devedition plaforms only for macosx64. After I see this working well I will update it for windows 32 and windows 64.

Comment 24

[[e-mail] (:bhearsum)]

Thanks for all your work so far Sebastian!

We moved our build set-up work to the "jamun" project branch for now. When you're ready, it would be good to verify the tests there, too.

Comment 25

[Kim Moir [:kmoir] ET]

Sebastian, you could update your patches to work on jamun in addition to mozilla-beta.  Then we could verify that the tests work there before running them on production in mozilla-beta.

https://treeherder.mozilla.org/#/jobs?repo=jamun

Comment 26

[Sebastian Pacurar [:spacurar]]

After enabling macosx64/win32/win64 DevEdition for jamun, it seems I ran into an error about the number of builders for t-w864-ix-279 exceeding the limit:

Traceback (most recent call last):
  File "../braindump/buildbot-related/builder_list.py", line 24, in <module>
    execfile(os.path.basename(sys.argv[1]), g)
  File "master.cfg", line 203, in <module>
    assert count <= builderLimit, "%s has %i builders; limit is %i" % (s, count, builderLimit)
AssertionError: t-w864-ix-279 has 4181 builders; limit is 4084

About that win7_vm_devedition KeyError it seems it was a typo in production_config.py, it's fixed now.

Comment 27

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_adding-macox64-win32-win64-devedition-platforms-for-beta-and-jamun.patch

This is the patch related to the error I get at https://bugzilla.mozilla.org/show_bug.cgi?id=1359595#c26

Comment 28

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_adding-win32-win64-devedition-puppet.patch

Added win32/win64 devedition slave_platforms in puppet.

Comment 29

[Kim Moir [:kmoir] ET]

Sebastian, can you please attach an updated builder diff?  Thanks!

Comment 30

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_v2_adding-macox64-win32-win64-devedition-platforms-for-beta-and-jamun.patch

This one is the right patch. I still receive this error when using builder_list.py script to create diffs:

Traceback (most recent call last):
  File "../braindump/buildbot-related/builder_list.py", line 24, in <module>
    execfile(os.path.basename(sys.argv[1]), g)
  File "master.cfg", line 203, in <module>
    assert count <= builderLimit, "%s has %i builders; limit is %i" % (s, count, builderLimit)

I will attach the diff with builder limit modified from master/master.cfg just to avoid the error and see the differneces

test-masters.sh passed

Comment 31

[Sebastian Pacurar [:spacurar]]

Attachment: diff for bug_1359595_v2_adding-macox64-win32-win64-devedition-platforms-for-beta-and-jamun.diff

This is the diff requested. I had to increase builder limit to 5000 in order to get this result.

Comment 32

[[e-mail] (:bhearsum)]

(In reply to Sebastian Pacurar [:spacurar] from comment #31)
> Created attachment 8865360 [details] [diff] [review]
> diff for
> bug_1359595_v2_adding-macox64-win32-win64-devedition-platforms-for-beta-and-
> jamun.diff
> 
> This is the diff requested. I had to increase builder limit to 5000 in order
> to get this result.

I don't think it's safe just to increase that limit here - it appears that it has to be updated in many places: https://bugzilla.mozilla.org/show_bug.cgi?id=712244

If it will be enough, I think we're better off disabling one or more of the twigs to get under the current limit.

Comment 33

[Kim Moir [:kmoir] ET]

Sebastian, you can write patches to disable the holly branch if that helps.  I don't think I need it for testing anymore.

Comment 34

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_adding-macox64-win32-win64-devedition-platforms-for-beta-and-jamun_disable-elm-cypress-holly.patch

Comment 35

[Kim Moir [:kmoir] ET]

Comment on attachment 8865449 [details] [diff] [review]
bug_1359595_adding-macox64-win32-win64-devedition-platforms-for-beta-and-jamun_disable-elm-cypress-holly.patch

I think this looks good with two exceptions.

We are first going to enable the tests on jamun to test before enabling on beta

so this part should be jamun only, and we can add beta later
+# Bug 1359595 - enable CI tests for DevEdition on mozilla-beta and jamun
+for branch in BRANCHES.keys():
+    if branch in ['mozilla-beta', 'jamun']:

mozilla/project_branches.py doesn't need to be modified because it is covered in bug 1363047

removing review flag to allow for updated patches

Comment 36

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_adding-macox64-win32-win64-devedition-platforms-for-jamun.patch

Comment 37

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_v2_adding-macox64-win32-win64-devedition-platforms-for-jamun.patch

Sorry, this is the right patch not the one before.

Comment 38

[Sebastian Pacurar [:spacurar]]

Attachment: diff for bug_1359595_v2_adding-macox64-win32-win64-devedition-platforms-for-jamun.diff

This is the builder diff. It looks like if the tests are enabled only on jamun branch there won't be any builder limit error thrown.

Comment 39

[Kim Moir [:kmoir] ET]

Comment on attachment 8865831 [details] [diff] [review]
bug_1359595_v2_adding-macox64-win32-win64-devedition-platforms-for-jamun.patch

Looks good.  Let's land and try to run tests on jamun. You'll need to land the other patch to disable the project branches as well so we have enough builders.

Comment 40

[Alin Selagea [:aselagea]]

Comment on attachment 8865831 [details] [diff] [review]
bug_1359595_v2_adding-macox64-win32-win64-devedition-platforms-for-jamun.patch

Previously landed https://bug1363047.bmoattachments.org/attachment.cgi?id=8865780

and then this patch:

https://hg.mozilla.org/build/buildbot-configs/rev/b5a54cef9cd1
https://hg.mozilla.org/build/buildbot-configs/rev/8bb6b154a7d0

Comment 41

[Alin Selagea [:aselagea]]

@Kim: do you know how we could trigger these new tests on jamun if we don't have any changes to land there?

Comment 42

[Kim Moir [:kmoir] ET]

I'm not sure if bhearsum is merging from m-c on a regular basis onto jamun.  You could ask him if he is doing that, a merge would trigger builds + tests.

Comment 43

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_tools.patch

Added devedition tests to /buildfarm/maintenance/production-masters.json

Comment 44

[Alin Selagea [:aselagea]]

Comment on attachment 8866288 [details] [diff] [review]
bug_1359595_tools.patch

There are some nits that need to be fixed when adding macosx64-devedition entries.
e.g. https://bugzilla.mozilla.org/attachment.cgi?id=8866288&action=diff#a/buildfarm/maintenance/production-masters.json_sec16

r+ with those changes.

Comment 45

[Alin Selagea [:aselagea]]

Comment on attachment 8866288 [details] [diff] [review]
bug_1359595_tools.patch

Fixed the nits and pushed:
https://hg.mozilla.org/build/tools/rev/9de932394e04

Comment 46

[Alin Selagea [:aselagea]]

Comment on attachment 8865832 [details] [diff] [review]
diff for bug_1359595_v2_adding-macox64-win32-win64-devedition-platforms-for-jamun.diff

Note: we stopped doing XP tests starting with Firefox 53 (see bug 1310836), so we don't need them here either.

That'll probably mean touching https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla-tests/config.py#3950

Comment 47

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_bb-configs.patch

Fixed the typo in win7_vm_devedition from production_config.py

Comment 48

[Kim Moir [:kmoir] ET]

Attachment: bug1359595puppet.patch

fix puppet

Comment 49

[Kim Moir [:kmoir] ET]

Attachment: bug1359595rmix.patch

patch to remove xp_ix_devedition platform since it's not needed and causing reconfig to fail

Comment 50

[Kim Moir [:kmoir] ET]

Comment on attachment 8866525 [details] [diff] [review]
bug1359595rmix.patch

I landed this so the reconfig on the windows masters could proceed.

Comment 51

[Nick Thomas [:nthomas] (UTC+12)]

Typo fix to allocate slaves to win7_vm_devedition, r+ from kmoir on IRC
https://hg.mozilla.org/build/buildbot-configs/rev/0894835927f8d0b0bb2489a579f2f69247674f9b

Comment 52

[Alin Selagea [:aselagea]]

We don't have any devedition tests on Windows for the most recent push to jamun since we were missing several things at that point [1]. They still appear as pending in TH, but I can't see them in the list of pending jobs.

I manually triggered one of those tests today after the landed the latest patches and it seems it completed fine. [2]

Could we do another push to confirm Windows tests work fine as well?

[1] treeherder.mozilla.org/#/jobs?repo=jamun&revision=2bf70a420d7f15ede51d2e23af599b6a76e6f2cf&filter-searchStr=devedition
[2] https://treeherder.mozilla.org/#/jobs?repo=jamun&revision=2bf70a420d7f15ede51d2e23af599b6a76e6f2cf&filter-searchStr=devedition&selectedJob=98314360

Comment 53

[Alin Selagea [:aselagea]]

err, wrong link for [1]
https://treeherder.mozilla.org/#/jobs?repo=jamun&revision=2bf70a420d7f15ede51d2e23af599b6a76e6f2cf&filter-searchStr=devedition

Comment 54

[[e-mail] (:bhearsum)]

(In reply to Alin Selagea [:aselagea][:buildduty] from comment #52)
> We don't have any devedition tests on Windows for the most recent push to
> jamun since we were missing several things at that point [1]. They still
> appear as pending in TH, but I can't see them in the list of pending jobs.
> 
> I manually triggered one of those tests today after the landed the latest
> patches and it seems it completed fine. [2]
> 
> Could we do another push to confirm Windows tests work fine as well?
> 
> [1]
> treeherder.mozilla.org/#/
> jobs?repo=jamun&revision=2bf70a420d7f15ede51d2e23af599b6a76e6f2cf&filter-
> searchStr=devedition
> [2]
> https://treeherder.mozilla.org/#/
> jobs?repo=jamun&revision=2bf70a420d7f15ede51d2e23af599b6a76e6f2cf&filter-
> searchStr=devedition&selectedJob=98314360

I just pushed a merge from beta -> jamun, let's see how it goes! https://treeherder.mozilla.org/#/jobs?repo=jamun&revision=ab077434ec6f6ac7cad3a4a4fa05f136e5cd72d5

Comment 55

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_enable-talos.patch

Enable talos tests for macosx64, win32, win64 devedition platforms.

Comment 56

[Alin Selagea [:aselagea]]

Comment on attachment 8867244 [details] [diff] [review]
bug_1359595_enable-talos.patch

That was easy :-)

Comment 57

[Sebastian Pacurar [:spacurar]]

Attachment: diff for bug_1359595_enable-talos.diff

Comment 58

[Alin Selagea [:aselagea]]

Comment on attachment 8866406 [details] [diff] [review]
bug_1359595_bb-configs.patch

That's in production.

Comment 59

[Alin Selagea [:aselagea]]

Comment on attachment 8867244 [details] [diff] [review]
bug_1359595_enable-talos.patch

https://hg.mozilla.org/build/buildbot-configs/rev/86f42a196d3b
https://hg.mozilla.org/build/buildbot-configs/rev/da5b8bc21292

Comment 60

[Phil Ringnalda (:philor)]

Right now you are running every (or maybe it's nearly-every, looks like a few non-e10s suites are on t-w732-spot but not also duplicated on g-w732-spot) test suite on g-w732-spot and on t-w732-spot and on t-w732-ix instead of what you want, the suites that can run on t-w732-spot only running there, the ones that can't but can run on g-w732-spot only running there, and the ones that can't tolerate AWS at all running on t-w732-ix.

Comment 61

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_v2_bb-configs.patch

I managed to make jamun a clone of mozilla-beta by copying the final list of tests for the regular edition to the dev-edition at the end of the file once they are defined.

I also would like to know if we should enable PGO for jamun as well, since it is enabled on mozilla-beta

Comment 62

[Sebastian Pacurar [:spacurar]]

Attachment: diff for bug_1359595_v2_bb-configs.diff

The builder diff

Comment 63

[Kim Moir [:kmoir] ET]

Comment on attachment 8867770 [details] [diff] [review]
bug_1359595_v2_bb-configs.patch

Do you still need to define the tests for devedition now that you are just copying them from beta

i.e. can parts like this be deleted?

https://hg.mozilla.org/build/buildbot-configs/file/96d29d2e7abc/mozilla-tests/config.py#l2007

I wouldn't worry about pgo

Comment 64

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_v3_bb-configs.patch

I managed to delete the suite_configs and leave an opt/debug_unittest_suites as empty lists in the json for the devedition slave platforms. 

The builder diff looks the same as the one from version 2 patch.

Comment 65

[Alin Selagea [:aselagea]]

Comment on attachment 8868077 [details] [diff] [review]
bug_1359595_v3_bb-configs.patch

https://hg.mozilla.org/build/buildbot-configs/rev/04fefd582166
https://hg.mozilla.org/build/buildbot-configs/rev/b17e9c891ace

Okay, we should be good with this.

Comment 66

[Sebastian Pacurar [:spacurar]]

Comment on attachment 8866452 [details] [diff] [review]
bug1359595puppet.patch

I noticed this patch landed in production so I marked it as obsolete.

Comment 67

[Sebastian Pacurar [:spacurar]]

Comment on attachment 8864844 [details] [diff] [review]
bug_1359595_adding-win32-win64-devedition-puppet.patch

I noticed that kmoir made a patch like this and excluded xp_ix_devedition since we don't need it. That patch landed in production so I will mark this one as obsolete.

Comment 68

[Kim Moir [:kmoir] ET]

Comment on attachment 8868077 [details] [diff] [review]
bug_1359595_v3_bb-configs.patch

backed out because non devedition jobs were not working on jamun

Comment 69

[Kim Moir [:kmoir] ET]

Attachment: bug1359595bb.patch

I updated your patch since bhearsum asked how the test work was going and I know they would like them running on beta very soon

Some notes
- the regular tests are not currently configured to run on jamun right now.  If you look at buildbot-configs/mozilla-tests/project_branches.py they are disabled.  So that was intentional, not the problem of the previous patch.
-we don't need to run any devedition tests on try so I changed some defaults to false
-we don't need to run debug test for dev edition since they are opt builds not debug builds
-I re-initialized the list of tests to [] in the loop you wrote because I noticed the dev-edition tests were not empty when updating them, probably from some of the many loops above
-I didn't run into any builder limit issues with this patch

I'll attach my builder diffs for win and mac for you to confirm.  It would be a good idea to compare the list of builders on beta with those for devedition for each platform on jamun and make sure they match

Comment 70

[Kim Moir [:kmoir] ET]

Attachment: mac.diff

Comment 71

[Kim Moir [:kmoir] ET]

Attachment: win.diff

Comment 72

[Sebastian Pacurar [:spacurar]]

Attachment: diff_mac.diff

I notice my mac diff is different from yours after I patched my buildbot-configs with your last patch. Windows diff looks the same.

The win diff is the same with mine.

Comment 73

[Kim Moir [:kmoir] ET]

Okay, does the diff match the list of tests that you would expect on beta?

Comment 74

[Jordan Lund (:jlund)]

Comment on attachment 8869211 [details] [diff] [review]
bug1359595bb.patch

Review of attachment 8869211 [details] [diff] [review]:
-----------------------------------------------------------------

looks good. one question about deepcopy below.

::: mozilla-tests/config.py
@@ +140,4 @@
>  PLATFORMS['win32-devedition']['win7_ix_devedition'] = {'name': "Windows 7 32-bit DevEdition"}
>  PLATFORMS['win32-devedition']['win7_vm_devedition'] = {'name': "Windows 7 VM 32-bit DevEdition"}
>  PLATFORMS['win32-devedition']['win7_vm_gfx_devedition'] = {'name': "Windows 7 VM-GFX 32-bit DevEdition",
> +                                     'try_by_default': False}

only this slave_platform has try_by_default set?

@@ +3266,5 @@
> +                # re-initializing the values of the opt tests since they seem to be set for a list above 
> +                BRANCHES[branch]['platforms'][str(platform + '-devedition')][str(slave_platform + '_devedition')]['opt_unittest_suites'] = []
> +                # same since we don't need debug tests for devedition
> +                BRANCHES[branch]['platforms'][str(platform + '-devedition')][str(slave_platform + '_devedition')]['debug_unittest_suites'] = []
> +                BRANCHES[branch]['platforms'][str(platform + '-devedition')][str(slave_platform + '_devedition')]['opt_unittest_suites'] = BRANCHES['mozilla-beta']['platforms'][platform][slave_platform]['opt_unittest_suites']

I'm torn as to whether this should be a deepcopy or point the same list.

On the one hand, if you mutate the list of the non-devedition platform below this loop, you get the benefit of updating the devedition equivalent so they stay in sync.

On the other hand, we might not always want that. In the future, it's easy to imagine test jobs being specific to devedition or vice versa. Say you want to have the same base of test jobs but then later add specific devedition ones. And this could cause unexpected results to the untrained eye.

Given all of that, I think I am in favor of keeping this loop at the bottom of the file and using deepcopy instead.

sigh... oh, loops and mutable objects, how you cause so much fun.

Kim, what do you think is best?

Comment 75

[Kim Moir [:kmoir] ET]

Attachment: bug1359595-bb2.patch

added try on default to false to all and switched to deepcopy. Thanks for the suggestions Jordan.

Comment 76

[Jordan Lund (:jlund)]

Comment on attachment 8869516 [details] [diff] [review]
bug1359595-bb2.patch

Review of attachment 8869516 [details] [diff] [review]:
-----------------------------------------------------------------

look great to me :)

Comment 77

[Kim Moir [:kmoir] ET]

Attachment: bug1359595config.patch

buildbot jobs failed on jamun due to missing config file, this will fix that

Comment 78

[Jordan Lund (:jlund)]

Comment on attachment 8869577 [details] [diff] [review]
bug1359595config.patch

Review of attachment 8869577 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +3266,5 @@
>              for slave_platform in PLATFORMS[platform]['slave_platforms']:
>                  if slave_platform not in BRANCHES['mozilla-beta']['platforms'][platform].keys():
>                      continue
> +                # re-initializing the values of the opt tests since they seem
> +                # to be set for a list above

s/for/from/ ?

@@ +3271,1 @@
>                  BRANCHES[branch]['platforms'][str(platform + '-devedition')][str(slave_platform + '_devedition')]['opt_unittest_suites'] = []

do we need this line?

@@ +3271,3 @@
>                  BRANCHES[branch]['platforms'][str(platform + '-devedition')][str(slave_platform + '_devedition')]['opt_unittest_suites'] = []
> +                # copy from beta
> +                BRANCHES[branch]['platforms'][str(platform + '-devedition')][str(slave_platform + '_devedition')] = deepcopy(BRANCHES['mozilla-beta']['platforms'][platform][slave_platform])

I'm slow, how does this help?

the ..[str(slave_platform + '_devedition')] value was missing something from [slave_platform] equivalent?

Comment 79

[Kim Moir [:kmoir] ET]

Attachment: bug1359595config2.patch

I removed the line we didn't need.

The tests failed earlier because we didn't copy the 'suite_config' part of the tests and thus when they ran on jamun they were missing the config to use

 'win8_64': {
            'opt_unittest_suites': UNITTEST_SUITES['opt_unittest_suites'] + JITTEST + MARIONETTE + \
                                   REFTEST_NOACCEL + REFTEST_ONE_CHUNK + WEB_PLATFORM_REFTESTS + \
                                   WEB_PLATFORM_TESTS_CHUNKED,
            'debug_unittest_suites': UNITTEST_SUITES['debug_unittest_suites'] + JITTEST + \
                                     MOCHITEST_5 + MOCHITEST_DT_8 + REFTEST_NOACCEL_TWO_CHUNKS + \
                                     REFTEST_TWO_CHUNKS,
            'suite_config': {
                'mochitest-gpu': {
                    'config_files': ["unittests/win_unittest.py"],
                },

Comment 80

[Jordan Lund (:jlund)]

Comment on attachment 8869589 [details] [diff] [review]
bug1359595config2.patch

Review of attachment 8869589 [details] [diff] [review]:
-----------------------------------------------------------------

ah, makes sense!

Comment 81

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1359595_bb-configs_m-b.patch

I enabled DeveEition tests for mozilla-beta. There are the same tests which run on jamun. And I did not receive any limited builders errors.

There seemed to be an indentation problem: originally there were only 3 spaces instead of 4 between each lines and I added one space where it had to be added. This is why the same lines look like they were deleted and re-added (after the "else:" part).

Comment 82

[Sebastian Pacurar [:spacurar]]

Attachment: diff for bug_1359595_bb-configs_m-b.diff

The builder diff after DevEdition tests were enabled on mozilla-beta.

Comment 83

[Kim Moir [:kmoir] ET]

Comment on attachment 8870035 [details] [diff] [review]
bug_1359595_bb-configs_m-b.patch

looks good, would be a good idea to verify with bhearsum that tests are ready to enable on beta before landing

Comment 84

[[e-mail] (:bhearsum)]

(In reply to Kim Moir [:kmoir] from comment #83)
> Comment on attachment 8870035 [details] [diff] [review]
> bug_1359595_bb-configs_m-b.patch
> 
> looks good, would be a good idea to verify with bhearsum that tests are
> ready to enable on beta before landing

Woohoo! I don't see any reason why we can't enable them.

Comment 85

[Kim Moir [:kmoir] ET]

Comment on attachment 8870035 [details] [diff] [review]
bug_1359595_bb-configs_m-b.patch

https://hg.mozilla.org/build/buildbot-configs/rev/e571f9d54af7

Comment 86

[Phil Ringnalda (:philor)]

The tests don't realize that they are running on PGO builds, having names like "Windows 7 VM-GFX 32-bit DevEdition mozilla-beta opt test reftest" rather than "Windows 7 VM-GFX 32-bit DevEdition mozilla-beta pgo test reftest", which I'm pretty sure means we forgot to set pgo_strategy: per-checkin, and we are instead relying on the mozharness fallback of forcing PGO on any build on a release branch. We should set pgo_strategy, just in case there are other side effects of not setting it besides just the test naming.

Comment 87

[Kim Moir [:kmoir] ET]

Looks like tests are working on dev edition on beta. 

https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=f7c942b561d57638df5c1c2542dd3d946dc71907

The only remaining work is 
1) Write bb patches to enable talos jobs for linux64. Renable talos for linux64 in taskcluster.  Add this job to treeherder.
2) address comment 86 above

Comment 88

[Phil Ringnalda (:philor)]

Ugh, must be more complicated than just pgo_strategy, since it's BRANCHES[branch] that has a pgo_strategy.

Comment 89

[Kim Moir [:kmoir] ET]

Sebastian, 

I looked on your master that you were having problems with.  Have you tried configuring your local master's master/master_config.json with a similar config to what a real production master would look like with your changes? i.e. from production_masters.json

"limit_fx_platforms": [
      "linux",
      "linux64",
      "linux64-stylo",
      "linux64-stylosequential",
      "linux64-devedition"
    ],
    "limit_fx_slave_platforms": {
      "linux64": [
        "ubuntu64_hw"
      ],
      "linux64-stylo": [
        "ubuntu64_hw_stylo"
      ],
      "linux64-stylo-seq": [
        "ubuntu64_hw_styloseq"
      ],
      "linux64-devedition": [
        "ubuntu64_hw_devedition"
      ]
    },


Do the new builders appear on your master with these changes?

If you get stuck, please reach out to other people in channel if I'm not online :-)

Comment 90

[Sebastian Pacurar [:spacurar]]

I tried it this way as well but still same "linux64-devedition" KeyError.

Comment 91

[Alin Selagea [:aselagea]]

Attachment: enable_devedition_talos_linux64.patch

I simply wanted to unblock things here, so I did a bit of research to see what we can do. 

From what I could see, the key error on the build masters was due to [1], which in turn was looking at the code here:[2] - we had no 'linux64-devedition' entry there, so I added it similar to how we did with the other devedition platforms.
That resulted in a new jamun build, which I removed since it's not needed.

I then restricted the talos tests to m-b and jamun. Running '/buildbot-configs/test-masters.sh' finishes without any error now. 

[1] https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/config.py#2981
[2] https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/config.py#137

Comment 92

[Alin Selagea [:aselagea]]

Here's the builder diff:

--- old_sorted  2017-06-13 09:13:02.215327462 -0700
+++ new_sorted  2017-06-13 09:28:03.315589153 -0700
@@ -2379,0 +2380,30 @@ Ubuntu HW 12.04 x64 cedar talos tp5o-e10
+Ubuntu HW 12.04 x64 DevEdition jamun talos chromez-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos chromez ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos dromaeojs-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos dromaeojs ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos g1-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos g1 ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos g2-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos g2 ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos g3-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos g3 ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos g4-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos g4 ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos other-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos other ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos svgr-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos svgr ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos tp5o-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos tp5o ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos chromez-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos dromaeojs-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos g1-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos g2-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos g3-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos g4-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos g5-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos other-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos perf-reftest-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos quantum-pageload-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos svgr-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos tp5o-e10s ScriptFactory

Comment 93

[Kim Moir [:kmoir] ET]

Comment on attachment 8877222 [details] [diff] [review]
enable_devedition_talos_linux64.patch

So we don't need to enable the build for Linux64 since these builds run in taskcluster.  I just remembered what I did for Linux64 talos which was to modify the tests so they don't fail if there aren't upstream builders, see

https://hg.mozilla.org/build/buildbot-configs/file/tip/test/test_no_testers_without_builders.py#l27

Comment 94

[Kim Moir [:kmoir] ET]

Comment on attachment 8877222 [details] [diff] [review]
enable_devedition_talos_linux64.patch

I was thinking about this patch this morning.  Fixing the tests won't address the issue because the root cause is a missing key, not dependent builders.

You might be able to get away with just a definition for
linux64-devedition in mozilla/config.py as just an empty dict because the builds are just deleted anyways, not sure.

diff --git a/mozilla/config.py b/mozilla/config.py
--- a/mozilla/config.py
+++ b/mozilla/config.py
@@ -73,6 +73,7 @@ GLOBAL_VARS = {
         'linux64-asan-debug': {},
         'linux64-av': {},
         'linux64-add-on-devel': {},
+        'linux64-devedition': {},
         'macosx64-debug': {},
         'macosx64-add-on-devel': {},
         'win32-debug': {},
@@ -657,6 +658,107 @@ PLATFORM_VARS = {
             # algorithm for nightlies
             'consider_for_nightly': False,
         },
+        'linux64-devedition': {},

Comment 95

[Sebastian Pacurar [:spacurar]]

Attachment: enable_devedition_talos_linux64_v2.patch

This is the version you mentioned about in https://bugzilla.mozilla.org/show_bug.cgi?id=1359595#c94

It seems to be working and the builder diff looks the same.

Comment 96

[Sebastian Pacurar [:spacurar]]

Attachment: enable_devedition_talos_linux64_taskcluster.patch

Comment 97

[Sebastian Pacurar [:spacurar]]

Attachment: enable_devedition_talos_linux64_taskcluster.diff

The builder diff after I added talos tests on jamun. As you can see linux64-nightly/opt-talos is also included since the devedition is a copy of the devedition ones. After I added jamun on run-on-projects in taskcluster/ci/test/tests.yml, both of them have been enabled.

Comment 98

[Sebastian Pacurar [:spacurar]]

Comment on attachment 8878402 [details] [diff] [review]
enable_devedition_talos_linux64_taskcluster.patch

This applies for jamun branch.

Comment 99

[Alin Selagea [:aselagea]]

Comment on attachment 8877671 [details] [diff] [review]
enable_devedition_talos_linux64_v2.patch

https://hg.mozilla.org/build/buildbot-configs/rev/93d1e0cc159a
https://hg.mozilla.org/build/buildbot-configs/rev/455096219e44

Comment 100

[Kim Moir [:kmoir] ET]

Comment on attachment 8878402 [details] [diff] [review]
enable_devedition_talos_linux64_taskcluster.patch

When I ran this locally, I got 

 File "/Users/kmoir/hg/mozilla-inbound/taskcluster/taskgraph/transforms/job/__init__.py", line 209, in configure_taskdesc_for_run
    func(config, job, taskdesc)
  File "/Users/kmoir/hg/mozilla-inbound/taskcluster/taskgraph/transforms/job/mozharness_test.py", line 459, in mozharness_test_buildbot_bridge
    prefix=BUILDER_NAME_PREFIX[platform],
KeyError: u'linux64-devedition-nightly'

Did you test this patch against m-i using 

./mach taskgraph target -p ~/Downloads/parameters-m-i.yml 

where parameters-m-i.yml is a push from a recent run on mozilla-inbound?

Also, you can remove mozilla-aurora from the list of branches, it is removed in your other patch to deprecate mozilla-aurora

Comment 101

[Sebastian Pacurar [:spacurar]]

Attachment: enable_devedition_talos_linux64_taskcluster_v2.patch

This works now. linux64-devedition-nightly was missing from BUILDER_NAME_PREFIX in mozharness_test.py

I was using a broken parameters.yml file before, I grabbed a new one and now it works.

Comment 102

[Kim Moir [:kmoir] ET]

Sebastian, it looks like the last patch was not landed on inbound.  Can you please arrange to have it checked in and ensure that the test results are green when it lands.  After it is merged to mozilla-central by the sheriffs, you will need to arrange to land the patch on beta.  If you land something on beta you need to include a=release in addition to r=your_reviewer.  This is because beta is a special branch where you need special permission to land.  We have approval from relman to land changes for dawn and devedition.  Once the patch lands on beta, you need to ensure that the linux64 talos jobs run and are green.

Comment 103

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/e8dd56285144
enable linux64 devedition talos tests on m-b and jamun, r=kmoir (patch by spacurar)

Comment 104

[Alin Selagea [:aselagea]]

Comment on attachment 8879070 [details] [diff] [review]
enable_devedition_talos_linux64_taskcluster_v2.patch

I've adjusted this patch a bit because the aurora entries were already removed (see bug 1363742). I also added a "DevEdition" suffix to the builder name in order to match the similar Mac tests.

Note: no linux64 devedition talos tests will run on m-i or m-c since they use the "default" filter, which has 'include_nightly': False. They should be available on jamun and mozilla-beta when the merges are done.

Comment 105

[Alin Selagea [:aselagea]]

(In reply to Alin Selagea [:aselagea][:buildduty] from comment #104)

> Note: no linux64 devedition talos tests will run on m-i or m-c since they
> use the "default" filter, which has 'include_nightly': False. They should be
> available on jamun and mozilla-beta when the merges are done.

Well, ignore this part. I was simply referring to the fact that no linux64 devedition talos tests will show up when using a 'parameter.yml' file from m-i, due to above mentioned reason.

Comment 106

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/e8dd56285144

Comment 107

[Kim Moir [:kmoir] ET]

This patch still needs rebased and  to be uplifted to beta, so reopening to allow spacurar to do that.

Comment 108

[Sebastian Pacurar [:spacurar]]

Attachment: enable_devedition_talos_linux64_taskcluster_m-b.patch

This is the rebase done for mozilla-beta.

Comment 109

[Sebastian Pacurar [:spacurar]]

Attachment: diff for enable_devedition_talos_linux64_taskcluster_m-b.diff

The builder diff for mozilla-beta.

Comment 110

[Kim Moir [:kmoir] ET]

https://hg.mozilla.org/releases/mozilla-beta/rev/cbcc1cbefa4b

Comment 111

[Kim Moir [:kmoir] ET]

Sebastian, it looks like the talos tests for linux64 devedition are failing with malformed payload exceptions.  This can sometimes occur when the name of job on taskcluster and buildbot do not match.  Can you look at the names and correspond that they match?

https://tools.taskcluster.net/task-group-inspector/#/W9JbvkwDSY6wUkS5h7rhxw/B_DTgjjGRpWkqFOQQt8ZtQ?_k=o463a9

Comment 112

[Alin Selagea [:aselagea]]

Yeah, that seems to be the case:
e.g.

TC: Ubuntu HW 12.04 x64 DevEdition mozilla-beta devedition talos chromez-e10s
BB: Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos chromez-e10s ScriptFactory

We'll need to fix that.

Comment 113

[Sebastian Pacurar [:spacurar]]

Attachment: talos_name_bb.patch

Comment 114

[Sebastian Pacurar [:spacurar]]

Attachment: talos_name_tc.patch

Comment 115

[Alin Selagea [:aselagea]]

I'll land these on Monday. We'll also need to rebase the TC patch and uplift it to beta.

Comment 116

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/fd4a832c33ec
fix linux64 devedition builder name, r=aselagea (patch by spacurar)

Comment 117

[Alin Selagea [:aselagea]]

Attachment: beta_rebase-talos_name_tc.patch

I rebased the patch in order to be able to uplift it to beta.
The previous push to m-i went fine: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=fd4a832c33eceb4d642e4c38b26eeefa3dea5a24

@Ryan: do you think you can help with this uplift?

Comment 118

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/fd4a832c33ec

Comment 119

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8884839 [details] [diff] [review]
beta_rebase-talos_name_tc.patch

https://hg.mozilla.org/releases/mozilla-beta/rev/c7b2725a5992

For future reference, you can use |hg commit -u| to change the patch author when committing.

Comment 120

[Alin Selagea [:aselagea]]

Thanks Ryan!

So we have the new buildernames:
TC: Ubuntu HW 12.04 x64 devedition mozilla-beta talos chromez-e10s
BB: Ubuntu HW 12.04 x64 devedition mozilla-beta talos chromez-e10s ScriptFactory

However, the corresponding tasks still end up with exceptions:
e.g. https://tools.taskcluster.net/groups/dHRL5y1nTMmAHvGq32KOlg/tasks/BQV4opjcRxKA2f-xtO3wAg/runs/0

Comment 121

[Alin Selagea [:aselagea]]

Looking at the logs from a bbb master, I got the following error:
"python: buildbot_bridge_tclistener bbb.services - task BQV4opjcRxKA2f-xtO3wAg: run 0: buildername: Ubuntu HW 12.04 x64 devedition mozilla-beta talos chromez-e10s: Payload is invalid (invalid buildername), refusing to create BuildRequest"

Again, the builder name seems correct. The problem here is these devedition talos tests *do not* appear on the corresponding bb-masters, so we'll need to update [1]
in order for this to work.

@Sebastian: could you please create a patch for that?

[1] https://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/production-masters.json

Comment 122

[Sebastian Pacurar [:spacurar]]

Attachment: tools_talos-name.patch

Comment 123

[Alin Selagea [:aselagea]]

Comment on attachment 8886199 [details] [diff] [review]
tools_talos-name.patch

There's some indentation that needs to be fixed.
r+ with those changes.

Comment 124

[Mihai Tabara [:mtabara]⌚️GMT]

Most of the work for but 1362387 seems to be tracked here so I think it's safe to close that as well when work is done here.

Comment 125

[Alin Selagea [:aselagea]]

Comment on attachment 8886199 [details] [diff] [review]
tools_talos-name.patch

Fixed indentation issues and pushed:
https://hg.mozilla.org/build/tools/rev/438415f41189

I can now see Linux64 Devedition talos tests on the corresponding buildot masters.

Comment 126

[Nick Thomas [:nthomas] (UTC+12)]

Unfortunately jobs are not begin run because there are no slaves assigned to the builders, eg:
http://buildbot-master52.build.mozilla.org:8201/builders/Ubuntu%20HW%2012.04%20x64%20devedition%20mozilla-beta%20talos%20g3-e10s

These were pending on m-b 0d6599e2269d96ec841687d1d482d8904b685f8b, until I cancelled them to shush the nagios alert for backlog:
Ubuntu HW 12.04 x64 devedition mozilla-beta talos dromaeojs-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos g3-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos svgr-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos g5-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos tp5o-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos g1-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos g4-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos other-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos g2-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos chromez-e10s

Comment 127

[Nick Thomas [:nthomas] (UTC+12)]

Ah I see, a simple typo. Fixed in 
https://hg.mozilla.org/build/buildbot-configs/rev/98ca745db1cebd6590480feb20f27fd8b02ce952
https://hg.mozilla.org/build/buildbot-configs/rev/1e878099eb13345d4864acd30c557aeabc6894d4

Comment 128

[Nick Thomas [:nthomas] (UTC+12)]

Not so fast! That breaks the travis tests - https://travis-ci.org/mozilla-releng/build-buildbot-configs/jobs/254288086

I suspect that's because Buildslaves.py needs to learn about ubuntu64_hw_devedition (via BuildSlaves-tests.py.erb in puppet). Going to leave you to that but please bear in mind bug 1381269.

Comment 129

[Nick Thomas [:nthomas] (UTC+12)]

And we might want to think about making travis go red when a job has no slaves attached to it.

Comment 130

[Alin Selagea [:aselagea]]

Attachment: fix_devedition_typo.patch

Thanks for your investigation here, Nick!

Rather than touching that puppet manifest(which doesn't seem up to date anyway) [1], I think we'll need to update Buildslaves.py.template here: [2].
I also fixed the typo you mentioned above. Ran /buildbot-configs/test-masters.sh on my test master and went fine.

[1] https://hg.mozilla.org/build/puppet/file/9155129eb0d3/modules/buildmaster/templates/BuildSlaves-tests.py.erb
[2] https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla-tests/BuildSlaves.py.template

Comment 131

[Alin Selagea [:aselagea]]

Attachment: fix_production_masters.patch

Also, I've noticed that we use scl3 bb masters (bm103, bm104, bm105) for linux64 stylo talos jobs, so I think we should do the same for linux64 devedition. 

Here's the patch to update those.

Comment 132

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Comment on attachment 8887010 [details] [diff] [review]
fix_production_masters.patch

looks good

Comment 133

[Alin Selagea [:aselagea]]

Comment on attachment 8887006 [details] [diff] [review]
fix_devedition_typo.patch

https://hg.mozilla.org/build/buildbot-configs/rev/48dda3309e9e
https://hg.mozilla.org/build/buildbot-configs/rev/f9c7a44b5691

Comment 134

[Alin Selagea [:aselagea]]

Comment on attachment 8887010 [details] [diff] [review]
fix_production_masters.patch

https://hg.mozilla.org/build/tools/rev/0b8788e405bd

Comment 135

[Alin Selagea [:aselagea]]

(In reply to Alin Selagea [:aselagea][:buildduty] from comment #130)

> Rather than touching that puppet manifest(which doesn't seem up to date
> anyway) [1], I think we'll need to update Buildslaves.py.template here: [2].

Disregard this. Puppet changes are also needed, otherwise Buildslaves.py won't get updated on the bb-masters and the reconfigs will fail. I'll upload a patch for this shortly.

Comment 136

[Alin Selagea [:aselagea]]

Attachment: add_ubuntu64_hw_devedition.patch

Comment 137

[Alin Selagea [:aselagea]]

Comment on attachment 8887056 [details] [diff] [review]
add_ubuntu64_hw_devedition.patch

https://hg.mozilla.org/build/puppet/rev/7b130d18f0b7
https://hg.mozilla.org/build/puppet/rev/398673363c4c

Comment 138

[Alin Selagea [:aselagea]]

Okay, so all reconfigs completed successfully and the Linux64 devedition talos jobs are now visible on the three scl3 bb-masters. Travis errors are also gone. On top of that, we have the first beta push for which these tests have actually started and..they're all green: \o/

https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=e26b1f5d635e2b62ba99f5ec90a0c10509a6eee9&filter-searchStr=devedition%20talos%20linux

I think we now have all the desired devedition tests in place, so we can go ahead and close this bug. If anything else is needed, we should file a new bug to avoid adding more load to this one.

Comment 139

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.