971176 - [tracker] run Android 2.3 test jobs on EC2

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

This bug is to help us track it from Release Engineering's side.

Comment 1

[Armen [:armenzg]]

== Old status from February ==
* Waiting to green things up
** https://bugzilla.mozilla.org/show_bug.cgi?id=967704#c16
** https://tbpl.mozilla.org/?tree=Ash&jobname=Android%202.3

Comment 2

[Armen [:armenzg]]

See status in here: https://bugzilla.mozilla.org/show_bug.cgi?id=967704#c30

Comment 3

[Kim Moir [:kmoir] ET]

:gbrown now that there are a lot of 2.3 tests running on emulators are there any tests on Tegras that we can start to disable?

Comment 4

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Yes, I think so!

Let's keep these caveats in mind:

 1. Android 2.3 armv6 tests are not running yet
 2. Android 2.3 Opt tests do not run yet on Aurora, Beta, etc (we are riding the trains)
 3. We do not and will not attempt to run Talos tests on Android 2.3
 4. Some tests are disabled on Android 2.3, as tracked in bug 979921, notably:
     - about one third of the robocop tests are skipped
     - many media mochitests are skipped: content/media/test/*, content/media/webaudio/test/*, dom/media/tests/mochitest/*

I think we can stop running all "Android 2.2 Opt" tests (including Talos) on trunk now.

:blassey -- What do you think?

Comment 5

[Brad Lassey [:blassey] (use needinfo?)]

#4 concerns me, since the two areas you call out (UI and media) are things that can be different between android versions. I've emailed mobile developers to see if we can make a push on fixing up those robocop tests. Need-info'ing Maire to see her comfort for running without these media tests on 2.x.

What we can do now is shut down the rest of them, but keep robocop and media mochitests running on 2.2.

Comment 6

[Kim Moir [:kmoir] ET]

Okay so I'll write a patch to do the following

-change should ride the trains
-leave robocop and media mochitests running on 2.2 (media mochitests correspond to M2-M4)
-all other 2.2 opt tests should be disabled including talos on trunk

Sound good?

Comment 7

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Yes, that sounds good.

Comment 8

[Kim Moir [:kmoir] ET]

Okay thanks, I've opened bug 1017599 to track disabling the tests

Comment 9

[Maire Reavy [:mreavy]]

There are significant differences between 2.2 and 2.3, so I'm very uncomfortable not testing 2.3 if it is still a product/platform we care about.

Comment 10

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Maire Reavy [:mreavy] (Please needinfo me) from comment #9)
> There are significant differences between 2.2 and 2.3, so I'm very
> uncomfortable not testing 2.3 if it is still a product/platform we care
> about.

To be clear, we're bringing up 2.3 testing and shutting down 2.2 testing (in conjunction with EOL'ing 2.2). right now, many media related tests don't pass on 2.3, so they're disabled. The question is if we feel comfortable with only testing media stuff on 4.0. If you're not comfortable with that, you'll need to the tests fixed on 2.3 ASAP.

Comment 11

[Geoff Brown [:gbrown] (pto Apr 11-18)]

:mreavy -- To get more context on the media problems, see bug 981898, bug 981889, and bug 981881.

Comment 12

[Maire Reavy [:mreavy]]

Sorry for my late reply.  I saw the needinfo (thanks!), but I didn't realize this was blocked on me.

Thanks for the pointers to the bugs, Geoff.  

I think for now we should disable them, but use the existing bugs to investigate the failures -- at least to a certain point.

It looks like JW is actively looking at resolving the failures on bug 981898 for Android 2.3.  I also can ask him to take a look at bug 981889.  JW has been an active contributor to web audio over the past year.

For Bug 981881, I love to ask gcp to take a day to investigate (ideally within the next 2-3 weeks)and assess how much work it would be to resolve the webrtc failures on that bug.   They all appear to have the same root cause.

If the work required on each bug isn't more than a few days each, we should fix them because there are enough differences between 2.3 and 4.x that it's worth that effort for peace of mind.

Comment 13

[Kim Moir [:kmoir] ET]

I think this can be closed now.  We do run 2.3 tests on Emulators on EC2. We may need to shift around the instance types that we use for them but the substantial work is done here. See bug 1024091 for details.