Closed Bug 951911 Opened 6 years ago Closed 6 years ago

Schedule gaia-integration tests on all trunk trees

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jgriffin, Assigned: jgriffin)

References

Details

Attachments

(1 file, 3 obsolete files)

The gaia-integration tests are now running on cedar, with just a couple of test failures.

To facilitate greening them up, let's schedule them across all trunk branches (on linux32/64 b2g desktop builds), but hide them initially until they're running green.
I think this should cause them to run only on trunk trees.  I also fixed an unrelated bug in which we weren't correctly removing marionette-webapi from older debug branches (which had no effect since we're not running them on said older branches atm).
Attachment #8349743 - Flags: review?(aki)
Comment on attachment 8349743 [details] [diff] [review]
Schedule gaia-integration tests on all trunk trees,

Thanks!
Attachment #8349743 - Flags: review?(aki) → review+
https://hg.mozilla.org/build/buildbot-configs/rev/48b38ad35d40

jhopkins, can we get a reconfig Thursday to pick this up?  Gaia devs are eager to have these tests running across the board.
Flags: needinfo?(jhopkins)
I'm not seeing these jobs on the trunk branch "Adjust hidden builders" page, so I can't hide them there. :\
They have to finish running once first
What percentage of https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy would you say these meet? I'm guessing around one third, where the two thirds remaining are all things that should have been done before running them on every tree.
(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> jhopkins, can we get a reconfig Thursday to pick this up?  Gaia devs are
> eager to have these tests running across the board.

coop did a reconfig last night
Flags: needinfo?(jhopkins)
We're running them on every tree to provide better feedback to the devs; they shouldn't be running on Aurora...I'll make a patch to fix that.

They certainly aren't ready to be visible.
They're running on b2g26 as well.
My logic for excluding Gi tets from older branches was a little off; this is better, I think.
Attachment #8350198 - Flags: review?(aki)
Comment on attachment 8350198 [details] [diff] [review]
Fix logic that excludes Gi tests from older branches,

I think the |for i in slave_p['{opt,debug}_unittest_suites']:| loops will end up skipping the next suite after the remove(), but I also think we never have duplicate suites, so this is probably fine.  We could even throw a |break| in there after the remove().

If the below bothers you, I bet we can find a solution on StackOverflow.

#!/usr/bin/env python
a = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
for b in a:
    if b == 2:
        a.remove(b)
    else:
        print b

Output:
1
4
5
6
7
8
9
10
Attachment #8350198 - Flags: review?(aki) → review+
You're right, a list comprehension is much better for this sort of thing.
Attachment #8350233 - Flags: review?(aki)
Attachment #8350198 - Attachment is obsolete: true
Comment on attachment 8350233 [details] [diff] [review]
Fix logic that excludes Gi tests from older branches,

>+        for slave_platform in (('linux64_gecko', 'ubuntu64_vm-b2gdt'),
>+                               ('linux32_gecko', 'ubuntu32_vm-b2gdt')):
>+            if nested_haskey(branch['platforms'], slave_platform[0], slave_platform[1]):
>+                slave_p = branch['platforms'][slave_platform[0]][slave_platform[1]]
>+                    slave_p['opt_unittest_suites'] = [x for x in slave_p['opt_unittest_suites']

I think you need to unindent this line, and the below 2 lines, by 4 spaces.

>+                                                      if x[0] != "gaia-integration"]
>+                    slave_p['debug_unittest_suites'] = [x for x in slave_p['debug_unittest_suites']
>+                                                      if x[0] != "gaia-integration"]

I think this line needs to be unindented by 2 spaces to line up appropriately.


The net builder diff:

(bb08)deathduck:/src/jgriffin/buildbot-configs/bm51-tests1-linux [14:34:06] (default)
810$ diff ../../{clean,dirty}.dump| more
3478d3477
< b2g_ubuntu64_vm mozilla-b2g26_v1_2 opt test gaia-integration ScriptFactory
Attachment #8350233 - Flags: review?(aki) → review+
Thanks...any idea why this patch doesn't unschedule them from aurora (gecko version 27) too?
I think because we define gecko_versions separately in mozilla-tests/b2g_config.py for some reason:
http://hg.mozilla.org/build/buildbot-configs/file/3f8bf08ee201/mozilla-tests/b2g_config.py#l30

Looks like we call setMainFirefoxVersions() to set those in mozilla/config.py
http://hg.mozilla.org/build/buildbot-configs/file/3f8bf08ee201/mozilla/config.py#l1506

Looks like mozilla-tests/config.py as well:
http://hg.mozilla.org/build/buildbot-configs/file/3f8bf08ee201/mozilla-tests/config.py#l80

I bet if we add that import and setMainFirefoxVersions() call in b2g_config.py,

a) it would fix the problem, but
b) it would die since there are no mozilla-beta or mozilla-release in mozilla-tests/b2g_config.py

Maybe we could fix (b) with |if 'mozilla-beta' in BRANCHES:| etc.
http://hg.mozilla.org/build/buildbot-configs/file/3f8bf08ee201/mozilla/master_common.py#l234

Sorry, we know this is ugly.  I want to move all this per-branch scheduling stuff in-tree for LWR in our graph definitions, if that's any consolation, but that won't help us in the short term.
Attached patch gaia-integrationSplinter Review
Ok, this looks better:

diff ../../{clean,dirty}.dump | more
3417d3416
< b2g_ubuntu64_vm mozilla-aurora opt test gaia-integration ScriptFactory
3419d3417
< b2g_ubuntu32_vm mozilla-aurora opt test gaia-integration ScriptFactory
3478d3475
< b2g_ubuntu64_vm mozilla-b2g26_v1_2 opt test gaia-integration ScriptFactory

More pylint and pyflakes friendly too, and we gain the capability to base stuff off gecko_version in mozilla-tests/b2g_config.py in the future too.
Attachment #8349743 - Attachment is obsolete: true
Attachment #8350233 - Attachment is obsolete: true
Attachment #8350360 - Flags: review?(catlee)
Attachment #8350360 - Flags: review?(catlee) → review?(rail)
Comment on attachment 8350360 [details] [diff] [review]
gaia-integration

looops
Attachment #8350360 - Flags: review?(rail) → review+
Hidden on try
In production (not CCed).
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.