Schedule gaia-integration tests on all trunk trees

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: jgriffin, Assigned: jgriffin)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 3 obsolete attachments)

(Assignee)

Description

4 years ago
The gaia-integration tests are now running on cedar, with just a couple of test failures.

To facilitate greening them up, let's schedule them across all trunk branches (on linux32/64 b2g desktop builds), but hide them initially until they're running green.
(Assignee)

Comment 1

4 years ago
Created attachment 8349743 [details] [diff] [review]
Schedule gaia-integration tests on all trunk trees,

I think this should cause them to run only on trunk trees.  I also fixed an unrelated bug in which we weren't correctly removing marionette-webapi from older debug branches (which had no effect since we're not running them on said older branches atm).
Attachment #8349743 - Flags: review?(aki)

Comment 2

4 years ago
Comment on attachment 8349743 [details] [diff] [review]
Schedule gaia-integration tests on all trunk trees,

Thanks!
Attachment #8349743 - Flags: review?(aki) → review+
(Assignee)

Comment 3

4 years ago
https://hg.mozilla.org/build/buildbot-configs/rev/48b38ad35d40

jhopkins, can we get a reconfig Thursday to pick this up?  Gaia devs are eager to have these tests running across the board.
Flags: needinfo?(jhopkins)
Is there a reason this is running (and failing) visibly on Aurora?
https://tbpl.mozilla.org/php/getParsedLog.php?id=32195224&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=32195140&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=32196392&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=32196351&tree=Mozilla-Aurora
(In reply to Wes Kocher (:KWierso) from comment #4)
> Is there a reason this is running (and failing) visibly on Aurora?
> https://tbpl.mozilla.org/php/getParsedLog.php?id=32195224&tree=Mozilla-Aurora
> https://tbpl.mozilla.org/php/getParsedLog.php?id=32195140&tree=Mozilla-Aurora
> https://tbpl.mozilla.org/php/getParsedLog.php?id=32196392&tree=Mozilla-Aurora
> https://tbpl.mozilla.org/php/getParsedLog.php?id=32196351&tree=Mozilla-Aurora

I just hid them on Aurora, but they're still running there.
I'm not seeing these jobs on the trunk branch "Adjust hidden builders" page, so I can't hide them there. :\
They have to finish running once first
What percentage of https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy would you say these meet? I'm guessing around one third, where the two thirds remaining are all things that should have been done before running them on every tree.
(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> jhopkins, can we get a reconfig Thursday to pick this up?  Gaia devs are
> eager to have these tests running across the board.

coop did a reconfig last night
Flags: needinfo?(jhopkins)
(Assignee)

Comment 10

4 years ago
We're running them on every tree to provide better feedback to the devs; they shouldn't be running on Aurora...I'll make a patch to fix that.

They certainly aren't ready to be visible.
They're running on b2g26 as well.
(Assignee)

Comment 12

4 years ago
Created attachment 8350198 [details] [diff] [review]
Fix logic that excludes Gi tests from older branches,

My logic for excluding Gi tets from older branches was a little off; this is better, I think.
Attachment #8350198 - Flags: review?(aki)

Comment 13

4 years ago
Comment on attachment 8350198 [details] [diff] [review]
Fix logic that excludes Gi tests from older branches,

I think the |for i in slave_p['{opt,debug}_unittest_suites']:| loops will end up skipping the next suite after the remove(), but I also think we never have duplicate suites, so this is probably fine.  We could even throw a |break| in there after the remove().

If the below bothers you, I bet we can find a solution on StackOverflow.

#!/usr/bin/env python
a = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
for b in a:
    if b == 2:
        a.remove(b)
    else:
        print b

Output:
1
4
5
6
7
8
9
10
Attachment #8350198 - Flags: review?(aki) → review+
(Assignee)

Comment 14

4 years ago
Created attachment 8350233 [details] [diff] [review]
Fix logic that excludes Gi tests from older branches,

You're right, a list comprehension is much better for this sort of thing.
Attachment #8350233 - Flags: review?(aki)
(Assignee)

Updated

4 years ago
Attachment #8350198 - Attachment is obsolete: true

Comment 15

4 years ago
Comment on attachment 8350233 [details] [diff] [review]
Fix logic that excludes Gi tests from older branches,

>+        for slave_platform in (('linux64_gecko', 'ubuntu64_vm-b2gdt'),
>+                               ('linux32_gecko', 'ubuntu32_vm-b2gdt')):
>+            if nested_haskey(branch['platforms'], slave_platform[0], slave_platform[1]):
>+                slave_p = branch['platforms'][slave_platform[0]][slave_platform[1]]
>+                    slave_p['opt_unittest_suites'] = [x for x in slave_p['opt_unittest_suites']

I think you need to unindent this line, and the below 2 lines, by 4 spaces.

>+                                                      if x[0] != "gaia-integration"]
>+                    slave_p['debug_unittest_suites'] = [x for x in slave_p['debug_unittest_suites']
>+                                                      if x[0] != "gaia-integration"]

I think this line needs to be unindented by 2 spaces to line up appropriately.


The net builder diff:

(bb08)deathduck:/src/jgriffin/buildbot-configs/bm51-tests1-linux [14:34:06] (default)
810$ diff ../../{clean,dirty}.dump| more
3478d3477
< b2g_ubuntu64_vm mozilla-b2g26_v1_2 opt test gaia-integration ScriptFactory
Attachment #8350233 - Flags: review?(aki) → review+
(Assignee)

Comment 16

4 years ago
Thanks...any idea why this patch doesn't unschedule them from aurora (gecko version 27) too?

Comment 17

4 years ago
I think because we define gecko_versions separately in mozilla-tests/b2g_config.py for some reason:
http://hg.mozilla.org/build/buildbot-configs/file/3f8bf08ee201/mozilla-tests/b2g_config.py#l30

Looks like we call setMainFirefoxVersions() to set those in mozilla/config.py
http://hg.mozilla.org/build/buildbot-configs/file/3f8bf08ee201/mozilla/config.py#l1506

Looks like mozilla-tests/config.py as well:
http://hg.mozilla.org/build/buildbot-configs/file/3f8bf08ee201/mozilla-tests/config.py#l80

I bet if we add that import and setMainFirefoxVersions() call in b2g_config.py,

a) it would fix the problem, but
b) it would die since there are no mozilla-beta or mozilla-release in mozilla-tests/b2g_config.py

Maybe we could fix (b) with |if 'mozilla-beta' in BRANCHES:| etc.
http://hg.mozilla.org/build/buildbot-configs/file/3f8bf08ee201/mozilla/master_common.py#l234

Sorry, we know this is ugly.  I want to move all this per-branch scheduling stuff in-tree for LWR in our graph definitions, if that's any consolation, but that won't help us in the short term.

Comment 18

4 years ago
Created attachment 8350360 [details] [diff] [review]
gaia-integration

Ok, this looks better:

diff ../../{clean,dirty}.dump | more
3417d3416
< b2g_ubuntu64_vm mozilla-aurora opt test gaia-integration ScriptFactory
3419d3417
< b2g_ubuntu32_vm mozilla-aurora opt test gaia-integration ScriptFactory
3478d3475
< b2g_ubuntu64_vm mozilla-b2g26_v1_2 opt test gaia-integration ScriptFactory

More pylint and pyflakes friendly too, and we gain the capability to base stuff off gecko_version in mozilla-tests/b2g_config.py in the future too.
Attachment #8349743 - Attachment is obsolete: true
Attachment #8350233 - Attachment is obsolete: true
Attachment #8350360 - Flags: review?(catlee)

Updated

4 years ago
Attachment #8350360 - Flags: review?(catlee) → review?(rail)
Comment on attachment 8350360 [details] [diff] [review]
gaia-integration

looops
Attachment #8350360 - Flags: review?(rail) → review+

Comment 20

4 years ago
Comment on attachment 8350360 [details] [diff] [review]
gaia-integration

https://hg.mozilla.org/build/buildbot-configs/rev/e3d8bc678e79
Attachment #8350360 - Flags: checked-in+
Hidden on try

Comment 22

4 years ago
In production (not CCed).
(Assignee)

Updated

4 years ago
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.