enable CI tests for DevEdition on mozilla-beta

RESOLVED FIXED

Status

enhancement
RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: bhearsum, Assigned: spacurar)

Tracking

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox56 fixed)

Details

Attachments

(32 attachments, 17 obsolete attachments)

811 bytes, patch
kmoir
: review+
aselagea
: checked-in+
Details | Diff | Splinter Review
16.82 KB, patch
Details | Diff | Splinter Review
173.49 KB, patch
Details | Diff | Splinter Review
44.32 KB, patch
kmoir
: review+
aselagea
: checked-in+
Details | Diff | Splinter Review
84.91 KB, patch
Details | Diff | Splinter Review
5.77 KB, patch
aselagea
: review+
aselagea
: checked-in+
Details | Diff | Splinter Review
587 bytes, patch
kmoir
: review+
aselagea
: checked-in+
Details | Diff | Splinter Review
8.29 KB, patch
kmoir
: checked-in+
Details | Diff | Splinter Review
465 bytes, patch
aselagea
: review+
aselagea
: checked-in+
Details | Diff | Splinter Review
3.53 KB, patch
Details | Diff | Splinter Review
50.11 KB, patch
Details | Diff | Splinter Review
4.52 KB, text/plain
Details
46.86 KB, text/plain
Details
9.10 KB, patch
Details | Diff | Splinter Review
34.59 KB, patch
jlund
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
1.90 KB, patch
jlund
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
1.17 KB, patch
kmoir
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
27.01 KB, patch
Details | Diff | Splinter Review
11.48 KB, patch
kmoir
: review+
kmoir
: feedback-
Details | Diff | Splinter Review
6.25 KB, patch
kmoir
: review+
aselagea
: checked-in+
Details | Diff | Splinter Review
6.78 KB, patch
Details | Diff | Splinter Review
1.65 KB, patch
Details | Diff | Splinter Review
7.85 KB, patch
kmoir
: review+
aselagea
: checked-in+
Details | Diff | Splinter Review
1.75 KB, patch
kmoir
: review+
Details | Diff | Splinter Review
601 bytes, patch
Details | Diff | Splinter Review
773 bytes, patch
aselagea
: review+
Details | Diff | Splinter Review
1.97 KB, patch
aselagea
: review+
Details | Diff | Splinter Review
1.97 KB, patch
RyanVM
: checked-in+
Details | Diff | Splinter Review
1.99 KB, patch
aselagea
: review+
aselagea
: checked-in+
Details | Diff | Splinter Review
1.85 KB, patch
aobreja
: review+
aselagea
: checked-in+
Details | Diff | Splinter Review
6.00 KB, patch
aobreja
: review+
aselagea
: checked-in+
Details | Diff | Splinter Review
1.69 KB, patch
aobreja
: review+
aselagea
: checked-in+
Details | Diff | Splinter Review
We'll be enabling builds for DevEdition on mozilla-beta by setting them up as separate "platforms" in buildbot and Taskcluster (eg: win32-devedition). We'll also need unit tests and talos for them. Some notable things:
- Builds will upload to https://archive.mozilla.org/pub/devedition/tinderbox-builds/
- We want to run all the same tests on DevEdition that we do on Firefox on mozilla-beta
- They need to show up separately on Treeherder

Builds are being worked on in bug 1358601.
(Reporter)

Comment 1

2 years ago
Kim, you said that you might be able to find someone to help with this? I'm hoping the builds will be ready in a day or two.
Sebastian, is this something that you might be interested in working on?  I can meet with you on vidyo to discuss the scope of the work if you are interested.
Flags: needinfo?(spacurar)
(Assignee)

Comment 3

2 years ago
Yes, I am interested. We can talk on irc after the Releng AllTeam meeting to discuss when we can make the call. Since the bug needs to be done in a day or two as stated by :bhearsum we might be able to make the call right after the AllTeam meeting.
Flags: needinfo?(spacurar)
(Reporter)

Comment 4

2 years ago
(In reply to Sebastian Pacurar [:spacurar] from comment #3)
> Yes, I am interested. We can talk on irc after the Releng AllTeam meeting to
> discuss when we can make the call. Since the bug needs to be done in a day
> or two as stated by :bhearsum we might be able to make the call right after
> the AllTeam meeting.

Works for me!
Sebastian, so as a first step you need to add new platforms Ben added in 

https://bug1358601.bmoattachments.org/attachment.cgi?id=8861601

to buildbot/mozilla-tests/config.py

for the platforms the tests run on (slave_platforms), they can use the existing ones for win32, win64 and macosx64

You will also have to update mozilla-tests/production_config.py

You might also need to update this file in puppet 
modules/buildmaster/templates/BuildSlaves-tests.py.erb 

Test those patches on your dev-master and ensure that they work.  The next step is to enable the same tests for these new platforms as exist for the regular builds on mozilla-beta.

This bug is an an example of similar work but for talos only and the builds run on taskcluster, not buildbot. 

https://bugzilla.mozilla.org/show_bug.cgi?id=1338871
(Assignee)

Comment 6

2 years ago
The patch applied on the dev test-master returns an error: KeyError: 'snowleopard_devedition'. It happens at line 2617:

if platform not in ('linux', 'linux64'):
->              branch['platforms'][platform][slave_platform]['debug_unittest_suites'] += \
                    MOCHITEST_DT_8_E10S + MOCHITEST_E10S + REFTEST_E10S_TWO_CHUNKS
                branch['platforms'][platform][slave_platform]['opt_unittest_suites'] += REFTEST_E10S
                if slave_platform != 'yosemite_r7':
                    branch['platforms'][platform][slave_platform]['debug_unittest_suites'] += \
                        REFTEST_NOACCEL_E10S_TWO_CHUNKS
                    branch['platforms'][platform][slave_platform]['opt_unittest_suites'] += \
                        REFTEST_NOACCEL_E10S


Note that I also added macosx64-devedition at mozilla-esr45 branch, otherwise I was receiving the same KeyError here:

## Tests that only run on ESR45 ###
for branch in BRANCHES.keys():
    if branch == 'mozilla-esr45':
        for platform in PLATFORMS.keys():
            if platform not in ['linux', 'linux64', 'linux64-asan']:
                continue
            for slave_platform in PLATFORMS[platform]['slave_platforms']:
                if slave_platform in BRANCHES[branch]['platforms'][platform]:
                    BRANCHES[branch]['platforms'][platform][slave_platform]['debug_unittest_suites'] += \
                        REFTEST_NOACCEL_FOUR_CHUNKS + REFTEST_FOUR_CHUNKS
                    BRANCHES[branch]['platforms'][platform][slave_platform]['opt_unittest_suites'] += \
                        REFTEST_NOACCEL_TWO_CHUNKS + REFTEST_TWO_CHUNKS
        for platform in PLATFORMS.keys():
->          for slave_platform in PLATFORMS[platform]['slave_platforms']:
                if slave_platform in BRANCHES[branch]['platforms'][platform]:
                    # Use the unchunked mochitest-gl on ESR45 due to leaks when chunked
                    for test_type in ('debug_unittest_suites', 'opt_unittest_suites'):
                        for item in BRANCHES[branch]['platforms'][platform][slave_platform][test_type]:
                            if item[0] == 'mochitest-gl':
                                BRANCHES[branch]['platforms'][platform][slave_platform][test_type].remove(item)
                        BRANCHES[branch]['platforms'][platform][slave_platform][test_type]+= MOCHITEST_WEBGL
Attachment #8862503 - Flags: review?(kmoir)
(Assignee)

Updated

2 years ago
Assignee: nobody → spacurar
Comment on attachment 8862503 [details] [diff] [review]
bug_1359595_adding-macox64-devedition.patch

Looks good so far.

You don't need 
"Rev4 MacOSX Snow Leopard 10.6 devedition"

It's only used for esr

We will only run tests on yosemite_r7 for this bug

I think you need to update this file as well
mozilla-tests/BuildSlaves.py.template
Attachment #8862503 - Flags: review?(kmoir)
(Assignee)

Comment 8

2 years ago
I finally made it work, and got some diffs from it, even though the diffs look totally different from how it should be. Looking forward to refactor the code to work only for mozilla-beta.

I still can't find a way to stop recieving a KeyError regarding mozilla-esr45, so I left 'macosx64-devedition' there, in the "platforms" until I find a way to avoid that.
Attachment #8862503 - Attachment is obsolete: true
Attachment #8862855 - Flags: review?(kmoir)
(Assignee)

Comment 11

2 years ago
Found a way to get rid of the KeyError: 'macosx64-devedition' regarding mozilla-esr45. They are out of the diff now.
Attachment #8862855 - Attachment is obsolete: true
Attachment #8862855 - Flags: review?(kmoir)
Attachment #8862905 - Flags: review?
Attachment #8862857 - Flags: review?(kmoir) → review+
Comment on attachment 8862905 [details] [diff] [review]
bug_1359595_v3_adding-macox64-devedition-buildbot.patch

To avoid the key error you need to update buildbot-configs/mozilla/BuildSlaves.py.template

So I think the next steps are do to that and change the patch so the tests only run on beta.
Comment on attachment 8862857 [details] [diff] [review]
bug_1359595_adding-macox64-devedition-puppet.patch

Landed on default: https://hg.mozilla.org/build/puppet/rev/9897e078ee88
Attachment #8862857 - Flags: checked-in+
(Assignee)

Comment 15

2 years ago
Attachment #8862906 - Attachment is obsolete: true
Attachment #8864093 - Flags: review?(kmoir)
Comment on attachment 8864093 [details] [diff] [review]
bug_1359595_v4_adding-macox64-devedition-buildbot.patch

looking better :-)

-# we only need to load SETA config on test scheduler master
-# it doesn't impact other masters
+#we only need to load SETA config on test scheduler master
+#it doesn't impact other masters

I don't know why the space is removed, we need the space for lint tests

why is this change required?

# Copy project branches into BRANCHES keys
@@ -2517,6 +2669,8 @@ for branch in BRANCHES.keys():
                         REFTEST_NOACCEL_TWO_CHUNKS + REFTEST_TWO_CHUNKS
         for platform in PLATFORMS.keys():
             for slave_platform in PLATFORMS[platform]['slave_platforms']:
+                if platform not in BRANCHES[branch]['platforms'].keys():
+                    continue
                 if slave_platform in BRANCHES[branch]['platforms'][platform]:
                     # Use the unchunked mochitest-gl on ESR45 due to leaks when chunked
                     for test_type in ('debug_unittest_suites', 'opt_unittest_suites'):
@@ -3081,6 +3235,15 @@ for branch in BRANCHES.keys():
     BRANCHES[branch]['platforms']['linux64-stylo']['talos_slave_platforms'] = []


mozilla-tests/BuildSlaves.py.template should be updated instead of mozilla/BuildSlaves.py.template
Attachment #8864093 - Flags: review?(kmoir)
(Assignee)

Comment 18

2 years ago
(In reply to Kim Moir [:kmoir] from comment #17)
> Comment on attachment 8864093 [details] [diff] [review]
> bug_1359595_v4_adding-macox64-devedition-buildbot.patch
> 
> looking better :-)
> 
> -# we only need to load SETA config on test scheduler master
> -# it doesn't impact other masters
> +#we only need to load SETA config on test scheduler master
> +#it doesn't impact other masters
> 
> I don't know why the space is removed, we need the space for lint tests

Sorry about the comments, that can be easily fixed.

> 
> why is this change required?
> 
> # Copy project branches into BRANCHES keys
> @@ -2517,6 +2669,8 @@ for branch in BRANCHES.keys():
>                          REFTEST_NOACCEL_TWO_CHUNKS + REFTEST_TWO_CHUNKS
>          for platform in PLATFORMS.keys():
>              for slave_platform in PLATFORMS[platform]['slave_platforms']:
> +                if platform not in BRANCHES[branch]['platforms'].keys():
> +                    continue
>                  if slave_platform in
> BRANCHES[branch]['platforms'][platform]:
>                      # Use the unchunked mochitest-gl on ESR45 due to leaks
> when chunked
>                      for test_type in ('debug_unittest_suites',
> 'opt_unittest_suites'):
> @@ -3081,6 +3235,15 @@ for branch in BRANCHES.keys():
>      BRANCHES[branch]['platforms']['linux64-stylo']['talos_slave_platforms']
> = []
> 
> 
> mozilla-tests/BuildSlaves.py.template should be updated instead of
> mozilla/BuildSlaves.py.template

I added the conditional in order to avoid the KeyError for macosx64-devedition regarding mozilla-esr45. The problem is here -> BRANCHES[branch]['platforms'][platform]  
This happens for win32/64-devedition as well, so the conditional avoids the KeyError for windows as well. I also modified the BuildSlaves.py.template but the error keeps being thrown. 

Will update mozilla-tests/BuildSlaves.py.template.
The key error might be occurring because you need to update

your_path_to_your_build_master/master/BuildSlaves.py

with the new platform key

The template file allows the master to be updated automatically by puppet. However, if you are running a development master and add a new platform, it needs to be updated manually
(Assignee)

Comment 20

2 years ago
If you look here: https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla-tests/config.py#45-54  you will notice that there are several platform types listed there. The reason I used that conditional is to avoid adding macosx64/win32/win64-devedition platforms to mozilla-esr45. I tested it that way and indeed it works because there won't be any KeyError anymore, but that means macosx64-devedition will be enabled for mozilla-esr45 branch as well and we want it only on mozilla-beta.

I also tried to add yosemite_r7_devedition to Buildslaves.py but the error is because macosx64-devedition is not present in the mozilla-esr45 platforms snippet I added above.
(Assignee)

Comment 21

2 years ago
Fixed the comment spacing lines, and updated mozilla-tests/BuildSlaves.py.template.

mozilla/BuildSlaves.py.template has been reverted. Shouldn't we add yosemite_r7_devedition there as well?
Attachment #8864093 - Attachment is obsolete: true
Attachment #8864197 - Flags: review?(kmoir)
Comment on attachment 8864197 [details] [diff] [review]
bug_1359595_v5_adding-macox64-devedition-buildbot.patch

These look better.

Did your run ./test-masters.sh?  

I added these patches to my master and the new builders don't appear on it. Not sure why, will look again in the morning.

Are you planning to add the other two platforms before deploying?

As a side note, if these new tests require treeherder changes it would be good to get those changes on file since treeherder is not deployed everyday, more like every week or so.
(Assignee)

Comment 23

2 years ago
Yes I already tested it with windows 64 and it works. I keep having some problems with windows 32. There seems to be KeyErrors regarding win7_vm_devedition. Right now I backed-up the patch with windows 32 and will add "devedition" to tests from devedition plaforms only for macosx64. After I see this working well I will update it for windows 32 and windows 64.
(Reporter)

Comment 24

2 years ago
Thanks for all your work so far Sebastian!

We moved our build set-up work to the "jamun" project branch for now. When you're ready, it would be good to verify the tests there, too.
Sebastian, you could update your patches to work on jamun in addition to mozilla-beta.  Then we could verify that the tests work there before running them on production in mozilla-beta.

https://treeherder.mozilla.org/#/jobs?repo=jamun
(Assignee)

Comment 26

2 years ago
After enabling macosx64/win32/win64 DevEdition for jamun, it seems I ran into an error about the number of builders for t-w864-ix-279 exceeding the limit:

Traceback (most recent call last):
  File "../braindump/buildbot-related/builder_list.py", line 24, in <module>
    execfile(os.path.basename(sys.argv[1]), g)
  File "master.cfg", line 203, in <module>
    assert count <= builderLimit, "%s has %i builders; limit is %i" % (s, count, builderLimit)
AssertionError: t-w864-ix-279 has 4181 builders; limit is 4084

About that win7_vm_devedition KeyError it seems it was a typo in production_config.py, it's fixed now.
(Assignee)

Comment 27

2 years ago
This is the patch related to the error I get at https://bugzilla.mozilla.org/show_bug.cgi?id=1359595#c26
Attachment #8864197 - Attachment is obsolete: true
Attachment #8864197 - Flags: review?(kmoir)
Attachment #8864843 - Flags: review?(kmoir)
(Assignee)

Comment 28

2 years ago
Added win32/win64 devedition slave_platforms in puppet.
Attachment #8864844 - Flags: review?(kmoir)
Sebastian, can you please attach an updated builder diff?  Thanks!
Flags: needinfo?(spacurar)
(Assignee)

Comment 30

2 years ago
This one is the right patch. I still receive this error when using builder_list.py script to create diffs:

Traceback (most recent call last):
  File "../braindump/buildbot-related/builder_list.py", line 24, in <module>
    execfile(os.path.basename(sys.argv[1]), g)
  File "master.cfg", line 203, in <module>
    assert count <= builderLimit, "%s has %i builders; limit is %i" % (s, count, builderLimit)

I will attach the diff with builder limit modified from master/master.cfg just to avoid the error and see the differneces

test-masters.sh passed
Attachment #8865359 - Flags: review?(kmoir)
(Assignee)

Updated

2 years ago
Attachment #8864843 - Attachment is obsolete: true
Attachment #8864843 - Flags: review?(kmoir)
(Assignee)

Comment 31

2 years ago
This is the diff requested. I had to increase builder limit to 5000 in order to get this result.
Flags: needinfo?(spacurar)
(Reporter)

Comment 32

2 years ago
(In reply to Sebastian Pacurar [:spacurar] from comment #31)
> Created attachment 8865360 [details] [diff] [review]
> diff for
> bug_1359595_v2_adding-macox64-win32-win64-devedition-platforms-for-beta-and-
> jamun.diff
> 
> This is the diff requested. I had to increase builder limit to 5000 in order
> to get this result.

I don't think it's safe just to increase that limit here - it appears that it has to be updated in many places: https://bugzilla.mozilla.org/show_bug.cgi?id=712244

If it will be enough, I think we're better off disabling one or more of the twigs to get under the current limit.
Sebastian, you can write patches to disable the holly branch if that helps.  I don't think I need it for testing anymore.
Attachment #8864844 - Flags: review?(kmoir) → review+
(Assignee)

Updated

2 years ago
Depends on: 1363047
Comment on attachment 8865449 [details] [diff] [review]
bug_1359595_adding-macox64-win32-win64-devedition-platforms-for-beta-and-jamun_disable-elm-cypress-holly.patch

I think this looks good with two exceptions.

We are first going to enable the tests on jamun to test before enabling on beta

so this part should be jamun only, and we can add beta later
+# Bug 1359595 - enable CI tests for DevEdition on mozilla-beta and jamun
+for branch in BRANCHES.keys():
+    if branch in ['mozilla-beta', 'jamun']:

mozilla/project_branches.py doesn't need to be modified because it is covered in bug 1363047

removing review flag to allow for updated patches
Attachment #8865449 - Flags: review?(kmoir)
(Assignee)

Comment 36

2 years ago
Attachment #8865449 - Attachment is obsolete: true
Attachment #8865803 - Flags: review?(kmoir)
(Assignee)

Comment 37

2 years ago
Sorry, this is the right patch not the one before.
Attachment #8865803 - Attachment is obsolete: true
Attachment #8865803 - Flags: review?(kmoir)
Attachment #8865831 - Flags: review?(kmoir)
(Assignee)

Comment 38

2 years ago
This is the builder diff. It looks like if the tests are enabled only on jamun branch there won't be any builder limit error thrown.
Comment on attachment 8865831 [details] [diff] [review]
bug_1359595_v2_adding-macox64-win32-win64-devedition-platforms-for-jamun.patch

Looks good.  Let's land and try to run tests on jamun. You'll need to land the other patch to disable the project branches as well so we have enough builders.
Attachment #8865831 - Flags: review?(kmoir) → review+
@Kim: do you know how we could trigger these new tests on jamun if we don't have any changes to land there?
Flags: needinfo?(kmoir)
I'm not sure if bhearsum is merging from m-c on a regular basis onto jamun.  You could ask him if he is doing that, a merge would trigger builds + tests.
Flags: needinfo?(kmoir)
(Assignee)

Comment 43

2 years ago
Added devedition tests to /buildfarm/maintenance/production-masters.json
Attachment #8866288 - Flags: review?(aselagea)
Comment on attachment 8866288 [details] [diff] [review]
bug_1359595_tools.patch

There are some nits that need to be fixed when adding macosx64-devedition entries.
e.g. https://bugzilla.mozilla.org/attachment.cgi?id=8866288&action=diff#a/buildfarm/maintenance/production-masters.json_sec16

r+ with those changes.
Attachment #8866288 - Flags: review?(aselagea) → review+
Comment on attachment 8866288 [details] [diff] [review]
bug_1359595_tools.patch

Fixed the nits and pushed:
https://hg.mozilla.org/build/tools/rev/9de932394e04
Attachment #8866288 - Flags: checked-in+
Comment on attachment 8865832 [details] [diff] [review]
diff for bug_1359595_v2_adding-macox64-win32-win64-devedition-platforms-for-jamun.diff

Note: we stopped doing XP tests starting with Firefox 53 (see bug 1310836), so we don't need them here either.

That'll probably mean touching https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla-tests/config.py#3950
(Assignee)

Comment 47

2 years ago
Fixed the typo in win7_vm_devedition from production_config.py
Attachment #8866406 - Flags: review?(kmoir)
Attachment #8866406 - Flags: review?(kmoir) → review+
Posted patch bug1359595puppet.patch (obsolete) — Splinter Review
fix puppet
patch to remove xp_ix_devedition platform since it's not needed and causing reconfig to fail
Comment on attachment 8866525 [details] [diff] [review]
bug1359595rmix.patch

I landed this so the reconfig on the windows masters could proceed.
Attachment #8866525 - Flags: checked-in+
We don't have any devedition tests on Windows for the most recent push to jamun since we were missing several things at that point [1]. They still appear as pending in TH, but I can't see them in the list of pending jobs.

I manually triggered one of those tests today after the landed the latest patches and it seems it completed fine. [2]

Could we do another push to confirm Windows tests work fine as well?

[1] treeherder.mozilla.org/#/jobs?repo=jamun&revision=2bf70a420d7f15ede51d2e23af599b6a76e6f2cf&filter-searchStr=devedition
[2] https://treeherder.mozilla.org/#/jobs?repo=jamun&revision=2bf70a420d7f15ede51d2e23af599b6a76e6f2cf&filter-searchStr=devedition&selectedJob=98314360
(Reporter)

Comment 54

2 years ago
(In reply to Alin Selagea [:aselagea][:buildduty] from comment #52)
> We don't have any devedition tests on Windows for the most recent push to
> jamun since we were missing several things at that point [1]. They still
> appear as pending in TH, but I can't see them in the list of pending jobs.
> 
> I manually triggered one of those tests today after the landed the latest
> patches and it seems it completed fine. [2]
> 
> Could we do another push to confirm Windows tests work fine as well?
> 
> [1]
> treeherder.mozilla.org/#/
> jobs?repo=jamun&revision=2bf70a420d7f15ede51d2e23af599b6a76e6f2cf&filter-
> searchStr=devedition
> [2]
> https://treeherder.mozilla.org/#/
> jobs?repo=jamun&revision=2bf70a420d7f15ede51d2e23af599b6a76e6f2cf&filter-
> searchStr=devedition&selectedJob=98314360

I just pushed a merge from beta -> jamun, let's see how it goes! https://treeherder.mozilla.org/#/jobs?repo=jamun&revision=ab077434ec6f6ac7cad3a4a4fa05f136e5cd72d5
(Assignee)

Comment 55

2 years ago
Enable talos tests for macosx64, win32, win64 devedition platforms.
Attachment #8867244 - Flags: review?(aselagea)
Comment on attachment 8867244 [details] [diff] [review]
bug_1359595_enable-talos.patch

That was easy :-)
Attachment #8867244 - Flags: review?(aselagea) → review+
Comment on attachment 8866406 [details] [diff] [review]
bug_1359595_bb-configs.patch

That's in production.
Attachment #8866406 - Flags: checked-in+
Right now you are running every (or maybe it's nearly-every, looks like a few non-e10s suites are on t-w732-spot but not also duplicated on g-w732-spot) test suite on g-w732-spot and on t-w732-spot and on t-w732-ix instead of what you want, the suites that can run on t-w732-spot only running there, the ones that can't but can run on g-w732-spot only running there, and the ones that can't tolerate AWS at all running on t-w732-ix.
(Assignee)

Comment 61

2 years ago
I managed to make jamun a clone of mozilla-beta by copying the final list of tests for the regular edition to the dev-edition at the end of the file once they are defined.

I also would like to know if we should enable PGO for jamun as well, since it is enabled on mozilla-beta
Flags: needinfo?(kmoir)
Attachment #8867770 - Flags: review?(kmoir)
(Assignee)

Comment 62

2 years ago
The builder diff
Comment on attachment 8867770 [details] [diff] [review]
bug_1359595_v2_bb-configs.patch

Do you still need to define the tests for devedition now that you are just copying them from beta

i.e. can parts like this be deleted?

https://hg.mozilla.org/build/buildbot-configs/file/96d29d2e7abc/mozilla-tests/config.py#l2007

I wouldn't worry about pgo
Flags: needinfo?(kmoir)
(Assignee)

Comment 64

2 years ago
I managed to delete the suite_configs and leave an opt/debug_unittest_suites as empty lists in the json for the devedition slave platforms. 

The builder diff looks the same as the one from version 2 patch.
Attachment #8867770 - Attachment is obsolete: true
Attachment #8867770 - Flags: review?(kmoir)
Attachment #8868077 - Flags: review?(kmoir)
Attachment #8868077 - Flags: review?(kmoir) → review+
(Assignee)

Comment 66

2 years ago
Comment on attachment 8866452 [details] [diff] [review]
bug1359595puppet.patch

I noticed this patch landed in production so I marked it as obsolete.
Attachment #8866452 - Attachment is obsolete: true
(Assignee)

Comment 67

2 years ago
Comment on attachment 8864844 [details] [diff] [review]
bug_1359595_adding-win32-win64-devedition-puppet.patch

I noticed that kmoir made a patch like this and excluded xp_ix_devedition since we don't need it. That patch landed in production so I will mark this one as obsolete.
Attachment #8864844 - Attachment is obsolete: true
Comment on attachment 8868077 [details] [diff] [review]
bug_1359595_v3_bb-configs.patch

backed out because non devedition jobs were not working on jamun
Attachment #8868077 - Flags: checked-in+ → checked-in-
Posted patch bug1359595bb.patch (obsolete) — Splinter Review
I updated your patch since bhearsum asked how the test work was going and I know they would like them running on beta very soon

Some notes
- the regular tests are not currently configured to run on jamun right now.  If you look at buildbot-configs/mozilla-tests/project_branches.py they are disabled.  So that was intentional, not the problem of the previous patch.
-we don't need to run any devedition tests on try so I changed some defaults to false
-we don't need to run debug test for dev edition since they are opt builds not debug builds
-I re-initialized the list of tests to [] in the loop you wrote because I noticed the dev-edition tests were not empty when updating them, probably from some of the many loops above
-I didn't run into any builder limit issues with this patch

I'll attach my builder diffs for win and mac for you to confirm.  It would be a good idea to compare the list of builders on beta with those for devedition for each platform on jamun and make sure they match
Attachment #8868077 - Attachment is obsolete: true
Attachment #8869211 - Flags: review?(spacurar)
Posted file mac.diff
Posted file win.diff
Attachment #8869218 - Attachment is patch: false
(Assignee)

Comment 72

2 years ago
Posted patch diff_mac.diffSplinter Review
I notice my mac diff is different from yours after I patched my buildbot-configs with your last patch. Windows diff looks the same.

The win diff is the same with mine.
Flags: needinfo?(kmoir)
Okay, does the diff match the list of tests that you would expect on beta?
Flags: needinfo?(kmoir)
Depends on: 1362387
Comment on attachment 8869211 [details] [diff] [review]
bug1359595bb.patch

Review of attachment 8869211 [details] [diff] [review]:
-----------------------------------------------------------------

looks good. one question about deepcopy below.

::: mozilla-tests/config.py
@@ +140,4 @@
>  PLATFORMS['win32-devedition']['win7_ix_devedition'] = {'name': "Windows 7 32-bit DevEdition"}
>  PLATFORMS['win32-devedition']['win7_vm_devedition'] = {'name': "Windows 7 VM 32-bit DevEdition"}
>  PLATFORMS['win32-devedition']['win7_vm_gfx_devedition'] = {'name': "Windows 7 VM-GFX 32-bit DevEdition",
> +                                     'try_by_default': False}

only this slave_platform has try_by_default set?

@@ +3266,5 @@
> +                # re-initializing the values of the opt tests since they seem to be set for a list above 
> +                BRANCHES[branch]['platforms'][str(platform + '-devedition')][str(slave_platform + '_devedition')]['opt_unittest_suites'] = []
> +                # same since we don't need debug tests for devedition
> +                BRANCHES[branch]['platforms'][str(platform + '-devedition')][str(slave_platform + '_devedition')]['debug_unittest_suites'] = []
> +                BRANCHES[branch]['platforms'][str(platform + '-devedition')][str(slave_platform + '_devedition')]['opt_unittest_suites'] = BRANCHES['mozilla-beta']['platforms'][platform][slave_platform]['opt_unittest_suites']

I'm torn as to whether this should be a deepcopy or point the same list.

On the one hand, if you mutate the list of the non-devedition platform below this loop, you get the benefit of updating the devedition equivalent so they stay in sync.

On the other hand, we might not always want that. In the future, it's easy to imagine test jobs being specific to devedition or vice versa. Say you want to have the same base of test jobs but then later add specific devedition ones. And this could cause unexpected results to the untrained eye.

Given all of that, I think I am in favor of keeping this loop at the bottom of the file and using deepcopy instead.

sigh... oh, loops and mutable objects, how you cause so much fun.

Kim, what do you think is best?
Attachment #8869211 - Flags: review?(spacurar) → review+
added try on default to false to all and switched to deepcopy. Thanks for the suggestions Jordan.
Attachment #8869211 - Attachment is obsolete: true
Comment on attachment 8869516 [details] [diff] [review]
bug1359595-bb2.patch

Review of attachment 8869516 [details] [diff] [review]:
-----------------------------------------------------------------

look great to me :)
Attachment #8869516 - Flags: review+
Attachment #8869516 - Flags: checked-in+
Posted patch bug1359595config.patch (obsolete) — Splinter Review
buildbot jobs failed on jamun due to missing config file, this will fix that
Attachment #8869577 - Flags: review?(jlund)
Comment on attachment 8869577 [details] [diff] [review]
bug1359595config.patch

Review of attachment 8869577 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +3266,5 @@
>              for slave_platform in PLATFORMS[platform]['slave_platforms']:
>                  if slave_platform not in BRANCHES['mozilla-beta']['platforms'][platform].keys():
>                      continue
> +                # re-initializing the values of the opt tests since they seem
> +                # to be set for a list above

s/for/from/ ?

@@ +3271,1 @@
>                  BRANCHES[branch]['platforms'][str(platform + '-devedition')][str(slave_platform + '_devedition')]['opt_unittest_suites'] = []

do we need this line?

@@ +3271,3 @@
>                  BRANCHES[branch]['platforms'][str(platform + '-devedition')][str(slave_platform + '_devedition')]['opt_unittest_suites'] = []
> +                # copy from beta
> +                BRANCHES[branch]['platforms'][str(platform + '-devedition')][str(slave_platform + '_devedition')] = deepcopy(BRANCHES['mozilla-beta']['platforms'][platform][slave_platform])

I'm slow, how does this help?

the ..[str(slave_platform + '_devedition')] value was missing something from [slave_platform] equivalent?
I removed the line we didn't need.

The tests failed earlier because we didn't copy the 'suite_config' part of the tests and thus when they ran on jamun they were missing the config to use

 'win8_64': {
            'opt_unittest_suites': UNITTEST_SUITES['opt_unittest_suites'] + JITTEST + MARIONETTE + \
                                   REFTEST_NOACCEL + REFTEST_ONE_CHUNK + WEB_PLATFORM_REFTESTS + \
                                   WEB_PLATFORM_TESTS_CHUNKED,
            'debug_unittest_suites': UNITTEST_SUITES['debug_unittest_suites'] + JITTEST + \
                                     MOCHITEST_5 + MOCHITEST_DT_8 + REFTEST_NOACCEL_TWO_CHUNKS + \
                                     REFTEST_TWO_CHUNKS,
            'suite_config': {
                'mochitest-gpu': {
                    'config_files': ["unittests/win_unittest.py"],
                },
Attachment #8869577 - Attachment is obsolete: true
Attachment #8869577 - Flags: review?(jlund)
Attachment #8869589 - Flags: review?(jlund)
Comment on attachment 8869589 [details] [diff] [review]
bug1359595config2.patch

Review of attachment 8869589 [details] [diff] [review]:
-----------------------------------------------------------------

ah, makes sense!
Attachment #8869589 - Flags: review?(jlund) → review+
Attachment #8869589 - Flags: checked-in+
(Assignee)

Comment 81

2 years ago
I enabled DeveEition tests for mozilla-beta. There are the same tests which run on jamun. And I did not receive any limited builders errors.

There seemed to be an indentation problem: originally there were only 3 spaces instead of 4 between each lines and I added one space where it had to be added. This is why the same lines look like they were deleted and re-added (after the "else:" part).
Attachment #8870035 - Flags: review?(kmoir)
(Assignee)

Comment 82

2 years ago
The builder diff after DevEdition tests were enabled on mozilla-beta.
Comment on attachment 8870035 [details] [diff] [review]
bug_1359595_bb-configs_m-b.patch

looks good, would be a good idea to verify with bhearsum that tests are ready to enable on beta before landing
Attachment #8870035 - Flags: review?(kmoir) → review+
(Reporter)

Comment 84

2 years ago
(In reply to Kim Moir [:kmoir] from comment #83)
> Comment on attachment 8870035 [details] [diff] [review]
> bug_1359595_bb-configs_m-b.patch
> 
> looks good, would be a good idea to verify with bhearsum that tests are
> ready to enable on beta before landing

Woohoo! I don't see any reason why we can't enable them.
Depends on: 1362396
The tests don't realize that they are running on PGO builds, having names like "Windows 7 VM-GFX 32-bit DevEdition mozilla-beta opt test reftest" rather than "Windows 7 VM-GFX 32-bit DevEdition mozilla-beta pgo test reftest", which I'm pretty sure means we forgot to set pgo_strategy: per-checkin, and we are instead relying on the mozharness fallback of forcing PGO on any build on a release branch. We should set pgo_strategy, just in case there are other side effects of not setting it besides just the test naming.
Looks like tests are working on dev edition on beta. 

https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=f7c942b561d57638df5c1c2542dd3d946dc71907

The only remaining work is 
1) Write bb patches to enable talos jobs for linux64. Renable talos for linux64 in taskcluster.  Add this job to treeherder.
2) address comment 86 above
Whiteboard: leaveopen
Ugh, must be more complicated than just pgo_strategy, since it's BRANCHES[branch] that has a pgo_strategy.
Sebastian, 

I looked on your master that you were having problems with.  Have you tried configuring your local master's master/master_config.json with a similar config to what a real production master would look like with your changes? i.e. from production_masters.json

"limit_fx_platforms": [
      "linux",
      "linux64",
      "linux64-stylo",
      "linux64-stylosequential",
      "linux64-devedition"
    ],
    "limit_fx_slave_platforms": {
      "linux64": [
        "ubuntu64_hw"
      ],
      "linux64-stylo": [
        "ubuntu64_hw_stylo"
      ],
      "linux64-stylo-seq": [
        "ubuntu64_hw_styloseq"
      ],
      "linux64-devedition": [
        "ubuntu64_hw_devedition"
      ]
    },


Do the new builders appear on your master with these changes?

If you get stuck, please reach out to other people in channel if I'm not online :-)
(Assignee)

Comment 90

2 years ago
I tried it this way as well but still same "linux64-devedition" KeyError.
I simply wanted to unblock things here, so I did a bit of research to see what we can do. 

From what I could see, the key error on the build masters was due to [1], which in turn was looking at the code here:[2] - we had no 'linux64-devedition' entry there, so I added it similar to how we did with the other devedition platforms.
That resulted in a new jamun build, which I removed since it's not needed.

I then restricted the talos tests to m-b and jamun. Running '/buildbot-configs/test-masters.sh' finishes without any error now. 

[1] https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/config.py#2981
[2] https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/config.py#137
Attachment #8877222 - Flags: review?(bhearsum)
Attachment #8877222 - Flags: feedback?(kmoir)
Here's the builder diff:

--- old_sorted  2017-06-13 09:13:02.215327462 -0700
+++ new_sorted  2017-06-13 09:28:03.315589153 -0700
@@ -2379,0 +2380,30 @@ Ubuntu HW 12.04 x64 cedar talos tp5o-e10
+Ubuntu HW 12.04 x64 DevEdition jamun talos chromez-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos chromez ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos dromaeojs-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos dromaeojs ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos g1-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos g1 ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos g2-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos g2 ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos g3-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos g3 ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos g4-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos g4 ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos other-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos other ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos svgr-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos svgr ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos tp5o-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition jamun talos tp5o ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos chromez-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos dromaeojs-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos g1-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos g2-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos g3-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos g4-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos g5-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos other-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos perf-reftest-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos quantum-pageload-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos svgr-e10s ScriptFactory
+Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos tp5o-e10s ScriptFactory
Comment on attachment 8877222 [details] [diff] [review]
enable_devedition_talos_linux64.patch

So we don't need to enable the build for Linux64 since these builds run in taskcluster.  I just remembered what I did for Linux64 talos which was to modify the tests so they don't fail if there aren't upstream builders, see

https://hg.mozilla.org/build/buildbot-configs/file/tip/test/test_no_testers_without_builders.py#l27
Attachment #8877222 - Flags: feedback?(kmoir) → feedback-
Attachment #8877222 - Flags: review?(bhearsum)
Comment on attachment 8877222 [details] [diff] [review]
enable_devedition_talos_linux64.patch

I was thinking about this patch this morning.  Fixing the tests won't address the issue because the root cause is a missing key, not dependent builders.

You might be able to get away with just a definition for
linux64-devedition in mozilla/config.py as just an empty dict because the builds are just deleted anyways, not sure.

diff --git a/mozilla/config.py b/mozilla/config.py
--- a/mozilla/config.py
+++ b/mozilla/config.py
@@ -73,6 +73,7 @@ GLOBAL_VARS = {
         'linux64-asan-debug': {},
         'linux64-av': {},
         'linux64-add-on-devel': {},
+        'linux64-devedition': {},
         'macosx64-debug': {},
         'macosx64-add-on-devel': {},
         'win32-debug': {},
@@ -657,6 +658,107 @@ PLATFORM_VARS = {
             # algorithm for nightlies
             'consider_for_nightly': False,
         },
+        'linux64-devedition': {},
Attachment #8877222 - Flags: review+
(Assignee)

Comment 95

2 years ago
This is the version you mentioned about in https://bugzilla.mozilla.org/show_bug.cgi?id=1359595#c94

It seems to be working and the builder diff looks the same.
Attachment #8877671 - Flags: review?(kmoir)
Attachment #8877671 - Flags: review?(kmoir) → review+
(Assignee)

Comment 97

2 years ago
The builder diff after I added talos tests on jamun. As you can see linux64-nightly/opt-talos is also included since the devedition is a copy of the devedition ones. After I added jamun on run-on-projects in taskcluster/ci/test/tests.yml, both of them have been enabled.
(Assignee)

Comment 98

2 years ago
Comment on attachment 8878402 [details] [diff] [review]
enable_devedition_talos_linux64_taskcluster.patch

This applies for jamun branch.
Comment on attachment 8878402 [details] [diff] [review]
enable_devedition_talos_linux64_taskcluster.patch

When I ran this locally, I got 

 File "/Users/kmoir/hg/mozilla-inbound/taskcluster/taskgraph/transforms/job/__init__.py", line 209, in configure_taskdesc_for_run
    func(config, job, taskdesc)
  File "/Users/kmoir/hg/mozilla-inbound/taskcluster/taskgraph/transforms/job/mozharness_test.py", line 459, in mozharness_test_buildbot_bridge
    prefix=BUILDER_NAME_PREFIX[platform],
KeyError: u'linux64-devedition-nightly'

Did you test this patch against m-i using 

./mach taskgraph target -p ~/Downloads/parameters-m-i.yml 

where parameters-m-i.yml is a push from a recent run on mozilla-inbound?

Also, you can remove mozilla-aurora from the list of branches, it is removed in your other patch to deprecate mozilla-aurora
Attachment #8878402 - Flags: review?(kmoir)
(Assignee)

Comment 101

2 years ago
This works now. linux64-devedition-nightly was missing from BUILDER_NAME_PREFIX in mozharness_test.py

I was using a broken parameters.yml file before, I grabbed a new one and now it works.
Attachment #8879070 - Flags: review?(kmoir)
Attachment #8879070 - Flags: review?(kmoir) → review+
Sebastian, it looks like the last patch was not landed on inbound.  Can you please arrange to have it checked in and ensure that the test results are green when it lands.  After it is merged to mozilla-central by the sheriffs, you will need to arrange to land the patch on beta.  If you land something on beta you need to include a=release in addition to r=your_reviewer.  This is because beta is a special branch where you need special permission to land.  We have approval from relman to land changes for dawn and devedition.  Once the patch lands on beta, you need to ensure that the linux64 talos jobs run and are green.
Flags: needinfo?(spacurar)

Comment 103

2 years ago
Pushed by aselagea@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/e8dd56285144
enable linux64 devedition talos tests on m-b and jamun, r=kmoir (patch by spacurar)
Comment on attachment 8879070 [details] [diff] [review]
enable_devedition_talos_linux64_taskcluster_v2.patch

I've adjusted this patch a bit because the aurora entries were already removed (see bug 1363742). I also added a "DevEdition" suffix to the builder name in order to match the similar Mac tests.

Note: no linux64 devedition talos tests will run on m-i or m-c since they use the "default" filter, which has 'include_nightly': False. They should be available on jamun and mozilla-beta when the merges are done.
Flags: needinfo?(spacurar)
Attachment #8879070 - Flags: checked-in+
(In reply to Alin Selagea [:aselagea][:buildduty] from comment #104)

> Note: no linux64 devedition talos tests will run on m-i or m-c since they
> use the "default" filter, which has 'include_nightly': False. They should be
> available on jamun and mozilla-beta when the merges are done.

Well, ignore this part. I was simply referring to the fact that no linux64 devedition talos tests will show up when using a 'parameter.yml' file from m-i, due to above mentioned reason.

Comment 106

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/e8dd56285144
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
This patch still needs rebased and  to be uplifted to beta, so reopening to allow spacurar to do that.
Status: RESOLVED → REOPENED
Keywords: leave-open
Resolution: FIXED → ---
Whiteboard: leaveopen
(Assignee)

Comment 108

2 years ago
This is the rebase done for mozilla-beta.
Attachment #8880839 - Flags: review?(kmoir)
(Assignee)

Comment 109

2 years ago
The builder diff for mozilla-beta.
Attachment #8880839 - Flags: review?(kmoir) → review+
Sebastian, it looks like the talos tests for linux64 devedition are failing with malformed payload exceptions.  This can sometimes occur when the name of job on taskcluster and buildbot do not match.  Can you look at the names and correspond that they match?

https://tools.taskcluster.net/task-group-inspector/#/W9JbvkwDSY6wUkS5h7rhxw/B_DTgjjGRpWkqFOQQt8ZtQ?_k=o463a9
Flags: needinfo?(spacurar)
Yeah, that seems to be the case:
e.g.

TC: Ubuntu HW 12.04 x64 DevEdition mozilla-beta devedition talos chromez-e10s
BB: Ubuntu HW 12.04 x64 DevEdition mozilla-beta talos chromez-e10s ScriptFactory

We'll need to fix that.
(Assignee)

Comment 113

2 years ago
Flags: needinfo?(spacurar)
Attachment #8883981 - Flags: review?(aselagea)
(Assignee)

Comment 114

2 years ago
Attachment #8883983 - Flags: review?(aselagea)
Attachment #8883981 - Flags: review?(aselagea) → review+
Attachment #8883983 - Flags: review?(aselagea) → review+
I'll land these on Monday. We'll also need to rebase the TC patch and uplift it to beta.

Comment 116

2 years ago
Pushed by aselagea@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/fd4a832c33ec
fix linux64 devedition builder name, r=aselagea (patch by spacurar)
I rebased the patch in order to be able to uplift it to beta.
The previous push to m-i went fine: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=fd4a832c33eceb4d642e4c38b26eeefa3dea5a24

@Ryan: do you think you can help with this uplift?
Flags: needinfo?(ryanvm)
Comment on attachment 8884839 [details] [diff] [review]
beta_rebase-talos_name_tc.patch

https://hg.mozilla.org/releases/mozilla-beta/rev/c7b2725a5992

For future reference, you can use |hg commit -u| to change the patch author when committing.
Flags: needinfo?(ryanvm)
Attachment #8884839 - Flags: checked-in+
Thanks Ryan!

So we have the new buildernames:
TC: Ubuntu HW 12.04 x64 devedition mozilla-beta talos chromez-e10s
BB: Ubuntu HW 12.04 x64 devedition mozilla-beta talos chromez-e10s ScriptFactory

However, the corresponding tasks still end up with exceptions:
e.g. https://tools.taskcluster.net/groups/dHRL5y1nTMmAHvGq32KOlg/tasks/BQV4opjcRxKA2f-xtO3wAg/runs/0
Looking at the logs from a bbb master, I got the following error:
"python: buildbot_bridge_tclistener bbb.services - task BQV4opjcRxKA2f-xtO3wAg: run 0: buildername: Ubuntu HW 12.04 x64 devedition mozilla-beta talos chromez-e10s: Payload is invalid (invalid buildername), refusing to create BuildRequest"

Again, the builder name seems correct. The problem here is these devedition talos tests *do not* appear on the corresponding bb-masters, so we'll need to update [1]
in order for this to work.

@Sebastian: could you please create a patch for that?

[1] https://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/production-masters.json
Flags: needinfo?(spacurar)
(Assignee)

Comment 122

2 years ago
Flags: needinfo?(spacurar)
Attachment #8886199 - Flags: review?(aselagea)
Comment on attachment 8886199 [details] [diff] [review]
tools_talos-name.patch

There's some indentation that needs to be fixed.
r+ with those changes.
Attachment #8886199 - Flags: review?(aselagea) → review+
Most of the work for but 1362387 seems to be tracked here so I think it's safe to close that as well when work is done here.
Comment on attachment 8886199 [details] [diff] [review]
tools_talos-name.patch

Fixed indentation issues and pushed:
https://hg.mozilla.org/build/tools/rev/438415f41189

I can now see Linux64 Devedition talos tests on the corresponding buildot masters.
Attachment #8886199 - Flags: checked-in+
Unfortunately jobs are not begin run because there are no slaves assigned to the builders, eg:
http://buildbot-master52.build.mozilla.org:8201/builders/Ubuntu%20HW%2012.04%20x64%20devedition%20mozilla-beta%20talos%20g3-e10s

These were pending on m-b 0d6599e2269d96ec841687d1d482d8904b685f8b, until I cancelled them to shush the nagios alert for backlog:
Ubuntu HW 12.04 x64 devedition mozilla-beta talos dromaeojs-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos g3-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos svgr-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos g5-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos tp5o-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos g1-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos g4-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos other-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos g2-e10s
Ubuntu HW 12.04 x64 devedition mozilla-beta talos chromez-e10s
Not so fast! That breaks the travis tests - https://travis-ci.org/mozilla-releng/build-buildbot-configs/jobs/254288086

I suspect that's because Buildslaves.py needs to learn about ubuntu64_hw_devedition (via BuildSlaves-tests.py.erb in puppet). Going to leave you to that but please bear in mind bug 1381269.
And we might want to think about making travis go red when a job has no slaves attached to it.
Thanks for your investigation here, Nick!

Rather than touching that puppet manifest(which doesn't seem up to date anyway) [1], I think we'll need to update Buildslaves.py.template here: [2].
I also fixed the typo you mentioned above. Ran /buildbot-configs/test-masters.sh on my test master and went fine.

[1] https://hg.mozilla.org/build/puppet/file/9155129eb0d3/modules/buildmaster/templates/BuildSlaves-tests.py.erb
[2] https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla-tests/BuildSlaves.py.template
Attachment #8887006 - Flags: review?(aobreja)
Also, I've noticed that we use scl3 bb masters (bm103, bm104, bm105) for linux64 stylo talos jobs, so I think we should do the same for linux64 devedition. 

Here's the patch to update those.
Attachment #8887010 - Flags: review?(aobreja)
Comment on attachment 8887010 [details] [diff] [review]
fix_production_masters.patch

looks good
Attachment #8887010 - Flags: review?(aobreja) → review+
Attachment #8887006 - Flags: review?(aobreja) → review+
(In reply to Alin Selagea [:aselagea][:buildduty] from comment #130)

> Rather than touching that puppet manifest(which doesn't seem up to date
> anyway) [1], I think we'll need to update Buildslaves.py.template here: [2].

Disregard this. Puppet changes are also needed, otherwise Buildslaves.py won't get updated on the bb-masters and the reconfigs will fail. I'll upload a patch for this shortly.
Attachment #8887056 - Flags: review?(aobreja) → review+
Okay, so all reconfigs completed successfully and the Linux64 devedition talos jobs are now visible on the three scl3 bb-masters. Travis errors are also gone. On top of that, we have the first beta push for which these tests have actually started and..they're all green: \o/

https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=e26b1f5d635e2b62ba99f5ec90a0c10509a6eee9&filter-searchStr=devedition%20talos%20linux

I think we now have all the desired devedition tests in place, so we can go ahead and close this bug. If anything else is needed, we should file a new bug to avoid adding more load to this one.
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.