1067480 - Add TBPL support for Android instrumentation tests

Status: RESOLVED FIXED

(Tree Management Graveyard :: TBPL, defect)

Description

[Nick Alexander :nalexander [he/him]]

My jobs appear to be running (\o/) and are currently listed as "U".  If it's easy, I'd like to make the instrumentation tests a suite like "I (Ba Br)" for Instrumentation -- Background, Browser but I'm not picky.  (We might want to keep single letters for the future.  Perhaps "Inst", or "Ai" for Android instrumentation).

This is the Android instrumentation analog to Bug 974358.

Comment 1

[Nick Alexander :nalexander [he/him]]

These jobs are not yet shown, so this is not urgent; but it's nicer to look at while developing the scripts.

Comment 2

[Nick Alexander :nalexander [he/him]]

kwierso: the jobs are running on Ash and Cedar.  They were added to the buildbot configurations by Bug 1064010.

Comment 3

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: 1067480

I think this is what we want.

Comment 4

[Nick Alexander :nalexander [he/him]]

Comment on attachment 8489524 [details] [diff] [review]
1067480

That seems reasonable.  Thanks for moving on this so quickly.

Comment 5

[Ed Morley [:emorley]]

Comment on attachment 8489524 [details] [diff] [review]
1067480

Thank you for taking this on Wes :-)

The patch as written does label the jobs correctly. 

My only thoughts are:
1) The "B[x]" symbols tend to be for builds, not tests.
2) These jobs appear in a group, and with the attached patch that group appears in the middle of non-grouped jobs. I think we should try and move the group to after mochitest or something.

Asking for review from Ryan, since he's spent some time making the job names/groups/symbols consistent across the board, so I think he might have an opinion :-)

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8489524 [details] [diff] [review]
1067480

Review of attachment 8489524 [details] [diff] [review]:
-----------------------------------------------------------------

Wes, if you and Nick are OK with the comments below, I've already made the changes below locally and rebased it on top of bug 1068764.

::: js/Config.js
@@ +287,5 @@
>                               "Unknown B2G Device Image Nightly",
>                               "Unknown B2G Device Image Nightly (Engineering)"],
>      "L10n Repack": ["L10n Nightly"],
>      "Android x86 Test Combos": ["Android x86 Test Set"],
> +    "Android Instrumentation Tests": ["Android Instrumentation Background", 

Trailing whitespace.

@@ +513,5 @@
>      "Marionette Framework Unit Tests" : "Mn",
>      "Marionette WebAPI Tests" : "Mnw",
>      "Android x86 Test Set" : "S",
>      "Android x86 Test Combos" : "Sets",
> +    "Android Instrumentation Tests" : "Inst",

I think we can get away with "I" here since it's just being used as a group symbol.

@@ +515,5 @@
>      "Android x86 Test Set" : "S",
>      "Android x86 Test Combos" : "Sets",
> +    "Android Instrumentation Tests" : "Inst",
> +    "Android Instrumentation Background" : "Ba",
> +    "Android Instrumentation Browser" : "Br",

What Ed said - these should come after the reftests. Otherwise, I think we're OK using Ba and Br here.

Comment 7

[Nick Alexander :nalexander [he/him]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #6)
> Comment on attachment 8489524 [details] [diff] [review]
> 1067480
> 
> Review of attachment 8489524 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Wes, if you and Nick are OK with the comments below, I've already made the
> changes below locally and rebased it on top of bug 1068764.

Fine by me.
 
> ::: js/Config.js
> @@ +287,5 @@
> >                               "Unknown B2G Device Image Nightly",
> >                               "Unknown B2G Device Image Nightly (Engineering)"],
> >      "L10n Repack": ["L10n Nightly"],
> >      "Android x86 Test Combos": ["Android x86 Test Set"],
> > +    "Android Instrumentation Tests": ["Android Instrumentation Background", 
> 
> Trailing whitespace.
> 
> @@ +513,5 @@
> >      "Marionette Framework Unit Tests" : "Mn",
> >      "Marionette WebAPI Tests" : "Mnw",
> >      "Android x86 Test Set" : "S",
> >      "Android x86 Test Combos" : "Sets",
> > +    "Android Instrumentation Tests" : "Inst",
> 
> I think we can get away with "I" here since it's just being used as a group
> symbol.

OK; wasn't sure if we wanted to camp the high-level namespace.

Comment 8

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Fine with me.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Thanks for the patch, Wes :)

https://hg.mozilla.org/webtools/tbpl/rev/75481643eaf0

Comment 10

[Ed Morley [:emorley]]

Ryan/Wes, what were your thoughts on:

(In reply to Ed Morley [:edmorley] from comment #5)
> 1) The "B[x]" symbols tend to be for builds, not tests.

(ie maybe using something that doesn't start with "B")

Comment 11

[Ryan VanderMeulen [:RyanVM]]

(In reply to Ed Morley [:edmorley] from comment #10)
> Ryan/Wes, what were your thoughts on:
> 
> (In reply to Ed Morley [:edmorley] from comment #5)
> > 1) The "B[x]" symbols tend to be for builds, not tests.
> 
> (ie maybe using something that doesn't start with "B")

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #6)
> Otherwise, I think we're OK using Ba and Br here.

Given that they're inside the I() group rather than standalone jobs, I don't see it being an issue here.

Comment 12

[Ed Morley [:emorley]]

Ah missed that :-)

Comment 13

[Nick Alexander :nalexander [he/him]]

(In reply to Ed Morley [:edmorley] from comment #12)
> Ah missed that :-)

I'm not strongly disposed to any such choices, BTW.  I anticipated having multiple suites, but further exploration with the foopy/Panda set up suggests that we might want to fold them all together:

1) the time to configure a job on a Panda job is ~10 minutes.

2) the time to run a suite, even a large-ish one, is ~5 minutes.  A small-ish suite is ~1 minute.

Let's run them separately and reconsider after we have some more experience.  Thanks, folks!

Comment 14

[Ryan VanderMeulen [:RyanVM]]

In production :)