659254 - Not doing sendchange for release unittests on mozilla-2.0 and mozilla-beta

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

I didn't get any unittest runs for 5.0b2, and it turns out we only did a talos sendchange. Same problem when I look back at 4.0.1. I think the issue is that we have overloaded releaseConfig['unittestPlatforms'] to handle both the old-style tests-on-builders (1.9.1 and 1.9.2), and the new tests-on-testers regime. 

Going through release.py, we have 
* creating schedulers for tests-on-builder, http://hg.mozilla.org/build/buildbotcustom/file/3a5d663958f5/process/release.py#l421
* create builders for tests-on-builders, http://hg.mozilla.org/build/buildbotcustom/file/3a5d663958f5/process/release.py#l778

* control test packaging and master list for sendchanges, http://hg.mozilla.org/build/buildbotcustom/file/3a5d663958f5/process/release.py#l662

For 1.9.1 and 1.9.2 we want the first two, for 2.0 and later we only want the last one.

Comment 1

[Armen [:armenzg]]

Attachment: [configs] add unittest platforms so we generate sendchanges

You were right at that time and I completely had my mind under mozilla-tests/ rather than mozilla/.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 534862 [details] [diff] [review]
[configs] add unittest platforms so we generate sendchanges

This will create unit tests on build slaves and have all sorts of spurious orage. I think we need a new config variable.

Comment 3

[Armen [:armenzg]]

Attachment: [configs] add sendchanges flag

Comment 4

[Armen [:armenzg]]

Attachment: [custom] handle new sendchange flag

Comment 5

[Armen [:armenzg]]

Comment on attachment 536133 [details] [diff] [review]
[configs] add sendchanges flag

Ooops! No review as I have not tested it yet.

Comment 6

[Armen [:armenzg]]

Attachment: [custom] handle new sendchange flag

Comment 7

[Armen [:armenzg]]

Attachment: differences on pm01-builder

We can see that the package tests step gets added plus the sendchanges.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 536133 [details] [diff] [review]
[configs] add sendchanges flag

The approach looks good, but here's some suggestions for improvements.

'enable_sendchanges_to_test_masters' fits the naming style of non-release config vars, but we're using camel case for release vars. Something like 'unittestSendchange' would be more appropriate.

Please update the staging configs too.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 536164 [details] [diff] [review]
[custom] handle new sendchange flag

See the review of the configs patch for my thoughts about the naming of the config variable. Would also be good to add some comments before these blocks, identifying which are only for unittests-on-builders. 

Why are you making the vars optional now ?

Comment 10

[Armen [:armenzg]]

Attachment: [configs] add enableUnittests flag

For both unit tests on the builders and on the test masters we use sendchanges so I decided to use "enableUnittests" as the flag rather than "unittestSendchange" since the sendchanges are implied. I can go either way.

Adding the unit tests on the builders is still done through unittestPlatforms which is only used for 1.9.2 and I added a note.

Comment 11

[Armen [:armenzg]]

Attachment: [custom] enable unit tests with enableUnittests flag

I wasn't even thinking that I was making it optional. I just thought "hey let me be consistent" since I wanted to make the new flag optional (I wanted to support both ways).

I believe we should enable with one flag and let the people either use unittestPlatforms for unit tests on builders or let the sendchanges trigger the jobs on the test masters.

What do you think? Looks good?

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 536288 [details] [diff] [review]
[custom] enable unit tests with enableUnittests flag

Please give Thunderbird and SeaMonkey a warning about this change, they'll need to update their configs too.

Comment 13

[Armen [:armenzg]]

Callek, jhopkins we are adding a new variable in our release configs:
+releaseConfig['enableUnittests'] = True

This variable enables the "make package" and doing sendchanges.

"unittestPlatforms" will only be used to generate the unit test builders on the same machine where the sendchange is executed.

All you need to do is add that line to your release-$product-$branch.py files if you follow a similar structure as we do.

Sounds good?

Comment 14

[John Hopkins (:jhopkins)]

Armen: does this complement the existing enable_opt_unittests option?

Comment 15

[Armen [:armenzg]]

(In reply to comment #14)
> Armen: does this complement the existing enable_opt_unittests option?

There are two pieces in here.
On the test masters we have enable_opt_unittests set to True so the test builders and schedulers get added (this controls both normal test builders plus release test builders). On the build masters this value is set to False since we don't unit tests on them anymore (we used to do unit tests on w2k3, Centos 5 & darwin9/10).

The other part is that on the release-firefox-*.py files we have the unittestPlatform value to add the test builders and schedulers plus enabling packaging and sendchanges.

With this new *release* specific variable we can enforce the release builds to do "make package" and do the sendchanges without adding builders/schedulers. These two functions (adding builders/schedulers and adding packaging-of-tests/sendchanges) were coupled together with unittestPlatforms. The new variable decouples these functions making the *build* masters to package-tests and do sendchanges without adding unit test builders/schedulers.

Not sure how you guys do your configuration but if it is similar to us (builders on some masters and test builders on another(s)) then all you have to do is add is:
+releaseConfig['enableUnittests'] = True

I hope it helps but let me know :)

Comment 16

[Armen [:armenzg]]

Both landed. We want to take this with beta4
http://hg.mozilla.org/build/buildbotcustom/rev/329b837bd034
http://hg.mozilla.org/build/buildbot-configs/rev/4141c2959d37