951911 - Schedule gaia-integration tests on all trunk trees

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

The gaia-integration tests are now running on cedar, with just a couple of test failures.

To facilitate greening them up, let's schedule them across all trunk branches (on linux32/64 b2g desktop builds), but hide them initially until they're running green.

Comment 1

[Jonathan Griffin (:jgriffin)]

Attachment: Schedule gaia-integration tests on all trunk trees,

I think this should cause them to run only on trunk trees.  I also fixed an unrelated bug in which we weren't correctly removing marionette-webapi from older debug branches (which had no effect since we're not running them on said older branches atm).

Comment 2

[Aki Sasaki (not active)]

Comment on attachment 8349743 [details] [diff] [review]
Schedule gaia-integration tests on all trunk trees,

Thanks!

Comment 3

[Jonathan Griffin (:jgriffin)]

https://hg.mozilla.org/build/buildbot-configs/rev/48b38ad35d40

jhopkins, can we get a reconfig Thursday to pick this up?  Gaia devs are eager to have these tests running across the board.

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Is there a reason this is running (and failing) visibly on Aurora?
https://tbpl.mozilla.org/php/getParsedLog.php?id=32195224&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=32195140&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=32196392&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=32196351&tree=Mozilla-Aurora

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

(In reply to Wes Kocher (:KWierso) from comment #4)
> Is there a reason this is running (and failing) visibly on Aurora?
> https://tbpl.mozilla.org/php/getParsedLog.php?id=32195224&tree=Mozilla-Aurora
> https://tbpl.mozilla.org/php/getParsedLog.php?id=32195140&tree=Mozilla-Aurora
> https://tbpl.mozilla.org/php/getParsedLog.php?id=32196392&tree=Mozilla-Aurora
> https://tbpl.mozilla.org/php/getParsedLog.php?id=32196351&tree=Mozilla-Aurora

I just hid them on Aurora, but they're still running there.

Comment 6

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I'm not seeing these jobs on the trunk branch "Adjust hidden builders" page, so I can't hide them there. :\

Comment 7

[Ryan VanderMeulen [:RyanVM]]

They have to finish running once first

Comment 8

[Phil Ringnalda (:philor)]

What percentage of https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy would you say these meet? I'm guessing around one third, where the two thirds remaining are all things that should have been done before running them on every tree.

Comment 9

[John Hopkins (:jhopkins)]

(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> jhopkins, can we get a reconfig Thursday to pick this up?  Gaia devs are
> eager to have these tests running across the board.

coop did a reconfig last night

Comment 10

[Jonathan Griffin (:jgriffin)]

We're running them on every tree to provide better feedback to the devs; they shouldn't be running on Aurora...I'll make a patch to fix that.

They certainly aren't ready to be visible.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

They're running on b2g26 as well.

Comment 12

[Jonathan Griffin (:jgriffin)]

Attachment: Fix logic that excludes Gi tests from older branches,

My logic for excluding Gi tets from older branches was a little off; this is better, I think.

Comment 13

[Aki Sasaki (not active)]

Comment on attachment 8350198 [details] [diff] [review]
Fix logic that excludes Gi tests from older branches,

I think the |for i in slave_p['{opt,debug}_unittest_suites']:| loops will end up skipping the next suite after the remove(), but I also think we never have duplicate suites, so this is probably fine.  We could even throw a |break| in there after the remove().

If the below bothers you, I bet we can find a solution on StackOverflow.

#!/usr/bin/env python
a = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
for b in a:
    if b == 2:
        a.remove(b)
    else:
        print b

Output:
1
4
5
6
7
8
9
10

Comment 14

[Jonathan Griffin (:jgriffin)]

Attachment: Fix logic that excludes Gi tests from older branches,

You're right, a list comprehension is much better for this sort of thing.

Comment 15

[Aki Sasaki (not active)]

Comment on attachment 8350233 [details] [diff] [review]
Fix logic that excludes Gi tests from older branches,

>+        for slave_platform in (('linux64_gecko', 'ubuntu64_vm-b2gdt'),
>+                               ('linux32_gecko', 'ubuntu32_vm-b2gdt')):
>+            if nested_haskey(branch['platforms'], slave_platform[0], slave_platform[1]):
>+                slave_p = branch['platforms'][slave_platform[0]][slave_platform[1]]
>+                    slave_p['opt_unittest_suites'] = [x for x in slave_p['opt_unittest_suites']

I think you need to unindent this line, and the below 2 lines, by 4 spaces.

>+                                                      if x[0] != "gaia-integration"]
>+                    slave_p['debug_unittest_suites'] = [x for x in slave_p['debug_unittest_suites']
>+                                                      if x[0] != "gaia-integration"]

I think this line needs to be unindented by 2 spaces to line up appropriately.


The net builder diff:

(bb08)deathduck:/src/jgriffin/buildbot-configs/bm51-tests1-linux [14:34:06] (default)
810$ diff ../../{clean,dirty}.dump| more
3478d3477
< b2g_ubuntu64_vm mozilla-b2g26_v1_2 opt test gaia-integration ScriptFactory

Comment 16

[Jonathan Griffin (:jgriffin)]

Thanks...any idea why this patch doesn't unschedule them from aurora (gecko version 27) too?

Comment 17

[Aki Sasaki (not active)]

I think because we define gecko_versions separately in mozilla-tests/b2g_config.py for some reason:
http://hg.mozilla.org/build/buildbot-configs/file/3f8bf08ee201/mozilla-tests/b2g_config.py#l30

Looks like we call setMainFirefoxVersions() to set those in mozilla/config.py
http://hg.mozilla.org/build/buildbot-configs/file/3f8bf08ee201/mozilla/config.py#l1506

Looks like mozilla-tests/config.py as well:
http://hg.mozilla.org/build/buildbot-configs/file/3f8bf08ee201/mozilla-tests/config.py#l80

I bet if we add that import and setMainFirefoxVersions() call in b2g_config.py,

a) it would fix the problem, but
b) it would die since there are no mozilla-beta or mozilla-release in mozilla-tests/b2g_config.py

Maybe we could fix (b) with |if 'mozilla-beta' in BRANCHES:| etc.
http://hg.mozilla.org/build/buildbot-configs/file/3f8bf08ee201/mozilla/master_common.py#l234

Sorry, we know this is ugly.  I want to move all this per-branch scheduling stuff in-tree for LWR in our graph definitions, if that's any consolation, but that won't help us in the short term.

Comment 18

[Aki Sasaki (not active)]

Attachment: gaia-integration

Ok, this looks better:

diff ../../{clean,dirty}.dump | more
3417d3416
< b2g_ubuntu64_vm mozilla-aurora opt test gaia-integration ScriptFactory
3419d3417
< b2g_ubuntu32_vm mozilla-aurora opt test gaia-integration ScriptFactory
3478d3475
< b2g_ubuntu64_vm mozilla-b2g26_v1_2 opt test gaia-integration ScriptFactory

More pylint and pyflakes friendly too, and we gain the capability to base stuff off gecko_version in mozilla-tests/b2g_config.py in the future too.

Comment 19

[Rail Aliiev [:rail]]

Comment on attachment 8350360 [details] [diff] [review]
gaia-integration

looops

Comment 20

[Aki Sasaki (not active)]

Comment on attachment 8350360 [details] [diff] [review]
gaia-integration

https://hg.mozilla.org/build/buildbot-configs/rev/e3d8bc678e79

Comment 21

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Hidden on try

Comment 22

[Armen [:armenzg]]

In production (not CCed).