897569 - Please schedule gaia-ui-tests on desktop-b2g-builds across all m-c based branches

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

The gaia-ui-tests that are running on cedar are stably green; let's schedule them to run on all trunk branches, including try (but not mozilla-b2g18..).

Sheriffs, we'd like to disable them initially when they show up, to give us a chance to experiment with some processes that we'll recommend to developers when a commit they submit causes bustage.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

(In reply to Jonathan Griffin (:jgriffin) from comment #0)
> Sheriffs, we'd like to disable them initially when they show up

Hide you mean?

Comment 2

[Jonathan Griffin (:jgriffin)]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #1)
> (In reply to Jonathan Griffin (:jgriffin) from comment #0)
> > Sheriffs, we'd like to disable them initially when they show up
> 
> Hide you mean?

Er, yep!

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Tweaking summary to:

1) avoid confusion with gaia-ui tests on pandas being disabled. These are tests being run on desktop-b2g builds running on (iirc) ubuntu32 + ubuntu64.

2) per yesterday's b2g meeting, jgriffin said these tests should be enabled on m-c and branches that land to/from there (*-inbound, try, aurora/beta/release). I dont remember if these should be enabled on mozilla-b2g18* or not.

(jgriffin, please adjust if I missed anything).

Comment 4

[Jonathan Griffin (:jgriffin)]

(In reply to John O'Duinn [:joduinn] from comment #3)
> Tweaking summary to:
> 
> 1) avoid confusion with gaia-ui tests on pandas being disabled. These are
> tests being run on desktop-b2g builds running on (iirc) ubuntu32 + ubuntu64.

That's correct.

> 
> 2) per yesterday's b2g meeting, jgriffin said these tests should be enabled
> on m-c and branches that land to/from there (*-inbound, try,
> aurora/beta/release). I dont remember if these should be enabled on
> mozilla-b2g18* or not.

No to mozilla-b2g18.  Yes to other trees that merge in and out of m-c, including services-central, fx-team, and birch.

> 
> (jgriffin, please adjust if I missed anything).

Comment 5

[cmtalbert]

Can we get some traction here?  These have been green for a week, and we still haven't scheduled them out across the other trees. In the meantime, we're getting justifiable flak from the developers about turning these things on.  We are so close to the finish line on this one; it would be a real travesty to run out of steam this close to done.

Comment 6

[Ed Morley [:emorley]]

Attachment: Patch v1

Schedule gaia-ui-test on desktop-b2g-builds across all branches other than esr17 and mozilla-b2g18*.

Comment 7

[Ed Morley [:emorley]]

(Thought I'd have a stab at this, hope you don't mind :-))

Comment 8

[Ed Morley [:emorley]]

Note gaia-ui-test jobs are no longer green on cedar (eg https://tbpl.mozilla.org/php/getParsedLog.php?id=25985915&tree=Cedar#error1), but at least if we get them scheduled everywhere, we can hide temporarily until green and still have the advantage of finer-grained regression ranges in the meantime :-)

Comment 9

[Jonathan Griffin (:jgriffin)]

(In reply to Ed Morley [:edmorley UTC+1] from comment #8)
> Note gaia-ui-test jobs are no longer green on cedar (eg
> https://tbpl.mozilla.org/php/getParsedLog.php?id=25985915&tree=Cedar#error1),
> but at least if we get them scheduled everywhere, we can hide temporarily
> until green and still have the advantage of finer-grained regression ranges
> in the meantime :-)

Just filed this as bug 900630.

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 784514 [details] [diff] [review]
Patch v1

Review of attachment 784514 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/b2g_config.py
@@ +1500,5 @@
> +    if branch in ('mozilla-esr17', 'mozilla-b2g18_v1_0_0',
> +                  'mozilla-b2g18_v1_0_1', 'mozilla-b2g18_v1_1_0_hd'):
> +        if 'linux64_gecko' in BRANCHES[branch]['platforms']:
> +            if 'ubuntu64_vm-b2gdt' in BRANCHES[branch]['platforms']['linux64_gecko']:
> +                del BRANCHES[branch]['platforms']['linux64_gecko']['ubuntu64_vm-b2gdt']

Don't we want to run GAIA_UNITTESTS here still, just not GAIA_UI? The rest of the patch looks fine.

Comment 11

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #10)
> Comment on attachment 784514 [details] [diff] [review]
> Patch v1
> 
> Review of attachment 784514 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: mozilla-tests/b2g_config.py
> @@ +1500,5 @@
> > +    if branch in ('mozilla-esr17', 'mozilla-b2g18_v1_0_0',
> > +                  'mozilla-b2g18_v1_0_1', 'mozilla-b2g18_v1_1_0_hd'):
> > +        if 'linux64_gecko' in BRANCHES[branch]['platforms']:
> > +            if 'ubuntu64_vm-b2gdt' in BRANCHES[branch]['platforms']['linux64_gecko']:
> > +                del BRANCHES[branch]['platforms']['linux64_gecko']['ubuntu64_vm-b2gdt']
> 
> Don't we want to run GAIA_UNITTESTS here still, just not GAIA_UI? The rest
> of the patch looks fine.

Comment 12

[cmtalbert]

(In reply to Ben Hearsum [:bhearsum] from comment #10)
> Comment on attachment 784514 [details] [diff] [review]
> Patch v1
> 
> Review of attachment 784514 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: mozilla-tests/b2g_config.py
> @@ +1500,5 @@
> > +    if branch in ('mozilla-esr17', 'mozilla-b2g18_v1_0_0',
> > +                  'mozilla-b2g18_v1_0_1', 'mozilla-b2g18_v1_1_0_hd'):
> > +        if 'linux64_gecko' in BRANCHES[branch]['platforms']:
> > +            if 'ubuntu64_vm-b2gdt' in BRANCHES[branch]['platforms']['linux64_gecko']:
> > +                del BRANCHES[branch]['platforms']['linux64_gecko']['ubuntu64_vm-b2gdt']
> 
> Don't we want to run GAIA_UNITTESTS here still, just not GAIA_UI? The rest
> of the patch looks fine.

Yes, we want both.

Comment 13

[Jonathan Griffin (:jgriffin)]

Tests are green again on cedar, so it would be a good time to put this in production.  :bhearsum, can you r+ or r-?

Comment 14

[Ed Morley [:emorley]]

(In reply to Ben Hearsum [:bhearsum] from comment #10)
> Don't we want to run GAIA_UNITTESTS here still, just not GAIA_UI? The rest
> of the patch looks fine.

I think there was a reason for this - unfortunately it's been almost two weeks since I wrote the patch so will need to re-assimilate those configs again. Was hoping to get to it today, but got back late from the ER (plaster cast mishaps, yey). Will take a look tomorrow :-)

Comment 15

[[e-mail] (:bhearsum)]

(In reply to Ed Morley [:edmorley UTC+1] from comment #14)
> (In reply to Ben Hearsum [:bhearsum] from comment #10)
> > Don't we want to run GAIA_UNITTESTS here still, just not GAIA_UI? The rest
> > of the patch looks fine.
> 
> I think there was a reason for this - unfortunately it's been almost two
> weeks since I wrote the patch so will need to re-assimilate those configs
> again. Was hoping to get to it today, but got back late from the ER (plaster
> cast mishaps, yey). Will take a look tomorrow :-)

Do you know the answer to this, Jonathan?

Comment 16

[Jonathan Griffin (:jgriffin)]

I think the gaia-unit tests are runnable on older branches, but not very sheriffable.  The reason is that they don't benefit from the gaia pushbot we have on trunk; so failures in gaia-unit tests on non-trunks branches will be impossible to blame on specific gaia commits.

Given this, I'm fine with not running them on older branches.  The primary push from engineering is to get them running on trunk branches.

Comment 17

[cmtalbert]

Bhearsum, is this safe to land on one of the build repos? OR do we need to wait for a reconfigure?  Let's close this out, happy to help, just not sure how to do so without stepping on your toes.

Comment 18

[Ed Morley [:emorley]]

Rebased and pushed :-)
https://hg.mozilla.org/build/buildbot-configs/rev/88a3f11a4a02

(Clint, will need reconfig by releng)

Comment 19

[Armen [:armenzg]]

Change merged to production branch and live on the buildbot masters.

Comment 20

[[e-mail] (:bhearsum)]

Sounds like we're done here?

Comment 21

[Ryan VanderMeulen [:RyanVM]]

I guess we need a new bug for un-hiding them when they aren't perma-fail.

Comment 22

[Ed Morley [:emorley]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #21)
> I guess we need a new bug for un-hiding them when they aren't perma-fail.

Filed bug 907611 :-)