Closed Bug 1155362 Opened 5 years ago Closed 5 years ago

enable SETA for Android in buildbot scheduling, configs

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

x86
macOS
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: kmoir, Assigned: kmoir)

Details

Attachments

(5 files)

We have SETA for desktop enabled in production in bug 1131269, we should also enable this for Android builds. (As discussed in April 15 mobile meeting by jmaher)
:jmaher is there a bug for implementing this and providing data on the SETA side of things?
Flags: needinfo?(jmaher)
Assignee: nobody → kmoir
Summary: enable SETA for Android → enable SETA for Android in buildbot scheduling, configs
hey, this is a great bug for it.  Vaibhav and I plan to get the data for Android integrated into the solution.  We wanted to get a better understanding of it before figuring out how to make it really work.  Initial analysis shows huge wins though.
Flags: needinfo?(jmaher)
Oh, I was thinking that it might be good to have a bug for the buildbot configs etc. and another bug for changing the SETA app to provide Android data.
action to add new config_seta_mobile.py link on masters
new config_seta_mobile.file + modifications to add it to mobile_config.py. It has to be separate file from config_seta.py for desktop because otherwise there are key errors due to the different platforms supported.

As before, the steps to deploy are 
1) land seta_config_mobile.py
2) reconfig
3) run custom action to add link
4) land changes to mobile_config.py
5) reconfig
6) -> fewer tests run on m-i and fx-team
Attachment #8596599 - Flags: review?(bugspam.Callek)
Attachment #8596605 - Flags: review?(bugspam.Callek)
Attachment #8596599 - Flags: review?(bugspam.Callek) → review+
Attachment #8596605 - Flags: review?(bugspam.Callek) → review+
Attachment #8596599 - Flags: checked-in+
Comment on attachment 8596605 [details] [diff] [review]
bug1155362bbconfigs.patch

new links created on masters so SETA on Android should be enabled on the next reconfig
Attachment #8596605 - Flags: checked-in+
I've verified on treeherder that this is enabled for Android 2.3 armv7 api9

Once we have SETA data for Android 4.3 (which was only enabled this week), we can add this to the seta configs.  I'll leave this bug open to account for that.
RyanVM and jmaher noticed a scheduling issue for Android armv7 API 9 where SETA is enabled

For example, looking at today's builds on m-i.  M-3 is a test that should run less frequently according to SETA.  However, it doesn't appear that it is running often enough. i.e. with a timeout of 3400 and a skipcount of 10, it only ran on the 6:02 and the 9:51 build on inbound
9:51 All Android 2.3 tests scheduled
Android 2.3 API9 opt Mochitest Mochitest M(3)
Requested: Tue Apr 28, 11:05:05
Started: Tue Apr 28, 11:05:33
Ended: Tue Apr 28, 11:38:18 
9:45 SETA subset
9:40 a SETA subset
8:57 a SETA subset
8:30 a SETA subset
8:21 a SETA subset
8:12 a SETA subset
7:57 a SETA subset
6:02 All Android 2.3 tests scheduled (started 6:32)
example: Android 2.3 API9 opt Mochitest Mochitest M(3)
Requested: Tue Apr 28, 7:26:40
Started: Tue Apr 28, 7:30:03
Ended: Tue Apr 28, 7:59:12
(In reply to Kim Moir [:kmoir] from comment #10)
> For example, looking at today's builds on m-i.  M-3 is a test that should
> run less frequently according to SETA.  However, it doesn't appear that it
> is running often enough. i.e. with a timeout of 3400 and a skipcount of 10,
> it only ran on the 6:02 and the 9:51 build on inbound

I agree that we should find out why the scheduling isn't happening as expected, but did we actually miss any failures in the larger gap? 

I want us to be *more* aggressive in skipping useless tests, and this kind of happy accident can help us set a better, longer threshold.
No, I don't think we missed any regressions. I was just trying to figure out why scheduling isn't working as I expected.
I love the ambition here.  One concern with skipping too many builds is that we have a much larger window to bisect the regression.  Just today, the sheriffs had a failure that was merged around as a side effect of SETA (bug 1159150, the second instance to date).  This is a worse case scenario as the work to fix it is more than fixing prior to a merge.

could we maybe keep our current levels for a bit, and we can work on tracking these merged regressions and finding a way to adjust processes/tools on the ateam side to avoid them completely.  That might take a few weeks, but it will prove where we are in the overall process.

So far the reception of SETA is great and our jobs have shows a significant decline.  Actually the feedback has been a large reduction in oranges to star :)
:jmaher, how long do you think it will take to generate SETA data for Android 4.3? (I realize it was only enabled last week, but thought I'd ask :-)
Flags: needinfo?(jmaher)
ideally a month- we can roll with it and just add jobs as they come in- so in a technical sort of way, we can do it with no data.  Giving it a month ensures that we hit failures and see how it related to that of the other platforms.
Flags: needinfo?(jmaher)
I don't see the Android 4.3 data in SETA yet, do you expect to appear soon?
Flags: needinfo?(jmaher)
I need to update all the jobs again- let me do that and get back to you- it takes a bit of sanity checking- so stay tuned.
Flags: needinfo?(jmaher)
I have updated SETA for 4.3 opt and debug.  Do let me know if there are issues with it.
remove config_seta_mobile.py due to refactoring in bug 1131269
Attachment #8616736 - Flags: review?(coop)
more cleanup
Attachment #8616778 - Flags: review?(coop)
Attachment #8616736 - Flags: review?(coop) → review+
Attachment #8616778 - Flags: review?(coop) → review+
Attachment #8616736 - Flags: checked-in+
Attachment #8616778 - Flags: checked-in+
cleaned up masters, now don't need this action anymore
Attachment #8616865 - Flags: review?(coop)
Attachment #8616865 - Flags: review?(coop) → review+
Attachment #8616865 - Flags: checked-in+
This seems to be working well and Android 4.3 was enabled on Friday.  We have seta enabled for Android 2.3 and Android 4.3. I didn't bother disabling it for 4.0 since these are on there way out anyways.
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.