570816 - Release unittests should be done on the test minis

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Comment 1

[Chris AtLee [:catlee]]

Attachment: WIP

Comment 2

[Rail Aliiev [:rail]]

Attachment: Move release opt_unittests to Talos

Please correct me if I understand something wrong.

There are some masters listen for sendchanges being sent by the build masters.

After a successful build the master "broadcasts" to unittest_masters (from config.py) about available binaries. If the branch of the build matches one of the listened, then the master starts the job.

talos-r3 generates a list of the branches it listens using generateTalosBranchObjects. If the particular branch has BRANCHES[branch]['release_tests'] set to a number, then talos also generates release unittests.

To make talos-r3 accept release opt_unittests we need to generate sendchanges using the following naming:
%(branch)s-release-%(platform)s-opt-unittest
(so the build master will ignore these tests).


Q: Is this correct for talos-r3 config.py? Won't there any side effect?
BRANCHES['mozilla-central']['release_tests'] = 5

Comment 3

[Chris AtLee [:catlee]]

Comment on attachment 450085 [details] [diff] [review]
Move release opt_unittests to Talos

Looks good, except that http://talos-master02.mozilla.org:8012 should be http://talos-master02.build.mozilla.org:8012

Comment 5

[Rail Aliiev [:rail]]

Attachment: buildbot-configs changes

Alice, could you take a look at these changes, especially talos-r3/config.py release_tests and build_branch bits?

Comment 6

[Armen [:armenzg]]

Comment on attachment 451014 [details] [diff] [review]
buildbot-configs changes

> BRANCHES['mozilla-central']['platforms']['linux64']['enable_debug_unittests'] = True
>+BRANCHES['mozilla-central']['release_tests'] = 5
what is this for?

> BRANCHES['mozilla-1.9.2']['cold_tests'] = (1, True, {}, NO_WIN)
> BRANCHES['mozilla-1.9.2']['svg_tests'] = (1, True, {}, ALL_PLATFORMS)
> BRANCHES['mozilla-1.9.2']['v8_tests'] = (0, True, {}, ALL_PLATFORMS)
> BRANCHES['mozilla-1.9.2']['scroll_tests'] = (1, True, {}, ALL_PLATFORMS)
> BRANCHES['mozilla-1.9.2']['repo_path'] = "mozilla-1.9.2"
>-BRANCHES['mozilla-1.9.2']['platforms']['linux']['enable_opt_unittests'] = False 
>-BRANCHES['mozilla-1.9.2']['platforms']['linux']['enable_debug_unittests'] = False 
>+BRANCHES['mozilla-1.9.2']['platforms']['linux']['enable_opt_unittests'] = True
>+BRANCHES['mozilla-1.9.2']['platforms']['linux']['enable_debug_unittests'] = True
This is the exact change that I tried to get in a couple of weeks ago when the talos master started growing on pending jobs.
Are you enabling this to enable the release unit tests?
What about the other platforms?
If it is not needed let's keep it disabled until talos master moves to the schedulerdb. If needed let's watch for the pending jobs and backout if needed.

Otherwise it looks good.

Comment 7

[Rail Aliiev [:rail]]

(In reply to comment #6)
> >+BRANCHES['mozilla-central']['release_tests'] = 5
> what is this for?

http://hg.mozilla.org/build/buildbotcustom/file/0a67b218559e/misc.py#l1889

For calling generateTalosReleaseBranchObjects() from generateTalosBranchObjects() and generating release Talos tests for mozilla-central releases.

> This is the exact change that I tried to get in a couple of weeks ago when the
> talos master started growing on pending jobs.

Hehe. :)

> Are you enabling this to enable the release unit tests?

Yes. Will this change enable unit tests for all builds?

> What about the other platforms?

Definitely should be added. 

> If it is not needed let's keep it disabled until talos master moves to the
> schedulerdb. If needed let's watch for the pending jobs and backout if needed.
> 
> Otherwise it looks good.

John, do we have any deadline for this? I remember that you marked this bug as "will be done by the end of this week". :)

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #7)
> (In reply to comment #6)
> > If it is not needed let's keep it disabled until talos master moves to the
> > schedulerdb. If needed let's watch for the pending jobs and backout if needed.
> > 
> > Otherwise it looks good.
> 
> John, do we have any deadline for this? I remember that you marked this bug as
> "will be done by the end of this week". :)

The time pressure here is that we need this running before we can safely feel ok turning off unittest-on-builders...which is blocking our Q2 goal about wait times. So, deadline... as soon as safely possible? 

I see question above about moving talos-master to scheduler db. Is this because of expected increase in load on talos-master? Does splitting the load out onto test-master01 address the load concerns?

Comment 9

[Rail Aliiev [:rail]]

This "tests" area is some kind of fogy for me, so I'm trying to make sure that I understand everything I do and make sure that it work in staging. :)

I'll test the patch in staging next Monday or Tuesday (depending on machine booking) and will try to land the patch during the next downtime.

Thanks for the information.

Comment 10

[Rail Aliiev [:rail]]

Attachment: buildbot-configs changes

Interdiff: 
* I removed all unittest builders from mozilla2 and mozilla2-staging release_master.py. 
* removed trailing coma (generates tuple instead of str)
* talos-r3/config.py: explicitly enable unittests for 1.9.1 we may need for releases

sm01 generates 2 types of sendchanges:
1. to talos_masters:
python /builds/slave/linux_build/tools/buildfarm/utils/retry.py -s 5 -t 1800 -r 5 buildbot sendchange --master talos-staging-master02.build.mozilla.org:9010 --username sendchange --branch mozilla-1.9.2-release-linux --revision e8f2f4a1c466 --comments  http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/3.6.5-candidates/build1/linux-i686/en-US/firefox-3.6.5.tar.bz2

2. to unittest_masters:
python /builds/slave/linux_build/tools/buildfarm/utils/retry.py -s 5 -t 1800 -r 0 buildbot sendchange --master staging-master.build.mozilla.org:9010 --username sendchange-unittest --branch mozilla-1.9.2-release-linux-opt-unittest --revision e8f2f4a1c466 --comments  http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/3.6.5-candidates/build1/linux-i686/en-US/firefox-3.6.5.tar.bz2 http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/3.6.5-candidates/build1/linux-i686/en-US/firefox-3.6.5.tests.zip

Comment 11

[Armen [:armenzg]]

Comment on attachment 452708 [details] [diff] [review]
buildbot-configs changes

Your patch is the way to go but not yet.

We are going to have a transition period in which we are going to have some release tests being run on the build side (we don't yet have win32 unit tests on talos machines) and some others on the talos side (10.5, 10.6, fed32/64) until we get unit tests running for Windows (not only running but green runs like in w2k3 build machines) on the talos side.

WRT to mozilla-1.9.1 we never backported what was required to have opt_unittests (IIUC it is just the matter of breaking mochitests into chunks. see screenshot [1]). This means that you should not be turning optimized unitests but the regular unit tests style. Check below:
> BRANCHES['mozilla-1.9.1']['platforms']['linux']['enable_unittests'] = True
> BRANCHES['mozilla-1.9.1']['platforms']['linux']['enable_opt_unittests'] = False
> BRANCHES['mozilla-1.9.1']['platforms']['linux']['enable_checktests'] = False

Your patch as it is enables unit tests (normal and debug) for 1.9.1 and 1.9.2 for all types of builds which could make things difficult for talos (as mentioned on previous comment). We should enable this separately from your patch and see if the master could handle it (filed bug 573480). Let's bring this up in today's meeting.

So for now change the list of "unittestPlatforms" to be just Windows on the build side and do not remove the blocks you were removing.

BTW are you adding a release_master.py file on the talos side? Are the sendchanges being handled by the talos side?

Good job; this is getting very close.

[1] Comparison of unit test builder on a 1.9.2 based release and a 1.9.1 release:
http://grab.by/534K

Comment 12

[Armen [:armenzg]]

(In reply to comment #7)
> (In reply to comment #6)
> > >+BRANCHES['mozilla-central']['release_tests'] = 5
> > what is this for?
> 
> http://hg.mozilla.org/build/buildbotcustom/file/0a67b218559e/misc.py#l1889
> 
> For calling generateTalosReleaseBranchObjects() from
> generateTalosBranchObjects() and generating release Talos tests for
> mozilla-central releases.
> 
Ah! I think I got this now. This is to enable talos jobs for whenever we do a "mozilla-central" release. It seems that we only had this for "mozilla-1.9.2" releases. Good job! (IIUIC)

Comment 13

[Rail Aliiev [:rail]]

Attachment: buildbot-configs changes

Looks like I finally understood the method catlee initially wanted to use. :) Broadcast the tests everywhere (build+talos) for now, remove the build masters from the broadcast later.

Attaching almost unchanged (except removed some trailing comas, causing build failures) first patch, refreshed.

Comment 14

[Rail Aliiev [:rail]]

Attachment: buildbot-configs changes

Forgot to add macosx to talos-r3.

Comment 15

[Armen [:armenzg]]

Comment on attachment 452997 [details] [diff] [review]
buildbot-configs changes

This patch looks good.

If you want to land this patch right away get rid of the changes on talos-r3/config.py and leave it to bug 573480 to enable those (I will start enabling them a couple of branches/platforms a day - patches to come soon).

If not, this will have to deployed early in a morning and watch talos-master02 and test-master01 for increase of pending jobs (this problem would not be there if buildbot-0.8.0 was already deployed; meanwhile we have to be careful).

Comment 16

[Rail Aliiev [:rail]]

(In reply to comment #15)

> If you want to land this patch right away get rid of the changes on
> talos-r3/config.py and leave it to bug 573480 to enable those (I will start
> enabling them a couple of branches/platforms a day - patches to come soon).

I'd go with this patch landed before 4.0b1 if we want to run unittests on Talos machines using Fedora instead of CentOS. What do you think?

Comment 17

[Armen [:armenzg]]

(In reply to comment #16)
> (In reply to comment #15)
> 
> > If you want to land this patch right away get rid of the changes on
> > talos-r3/config.py and leave it to bug 573480 to enable those (I will start
> > enabling them a couple of branches/platforms a day - patches to come soon).
> 
> I'd go with this patch landed before 4.0b1 if we want to run unittests on Talos
> machines using Fedora instead of CentOS. What do you think?

In that regards you should be safe as of now since beta1 will be shipped out of 2.0 or central which is currently enabled on talos-r3/confgig.py.

I will send an email today wrt to the plan on bug 573480.

Comment 18

[Chris AtLee [:catlee]]

Comment on attachment 452997 [details] [diff] [review]
buildbot-configs changes

2641:443bc1b4d4c2

Comment 19

[Armen [:armenzg]]

We now have 1.9.1 and 1.9.2 coverage on the minis for Fedora and OSX. This is only for 32-bit platforms since we are not doing 64-bit coverage on those branches.

Comment 20

[Rail Aliiev [:rail]]

Armen,

Could we resolve this bug? mozilla-central release tests already run on minis (see http://hg.mozilla.org/build/buildbot-configs/rev/443bc1b4d4c2) and IIRC we don't want to run tests on minis for 1.9.1/1.9.2.

Comment 21

[Armen [:armenzg]]

You recall correctly. No 1.9.1 and 1.9.2 unit tests on the minis (I will disable this shortly).

There is nothing left to be done in here AFAIK. Good job :)

Comment 22

[Armen [:armenzg]]

Attachment: fix branch names used for unit tests

I don't know when the regression came about but I need this for releases.

Comment 23

[Armen [:armenzg]]

Comment on attachment 463233 [details] [diff] [review]
fix branch names used for unit tests

From IRC.

Comment 24

[Armen [:armenzg]]

Comment on attachment 463233 [details] [diff] [review]
fix branch names used for unit tests

http://hg.mozilla.org/build/buildbot-configs/rev/73e9569bcc7c

I want to take this fix for beta3 build3 which is coming up today.

Comment 25

[Armen [:armenzg]]

Nothing left to be done in here.

This is how we now run unit tests for releases:
* m-c and 2.0 on the minis (except win32)
* 1.9.1 and 1.9.2 on the builders