The default bug view has changed. See this FAQ.

enable SETA for Android in buildbot scheduling, configs

RESOLVED FIXED

Status

Release Engineering
Platform Support
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: kmoir, Assigned: kmoir)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(5 attachments)

(Assignee)

Description

2 years ago
We have SETA for desktop enabled in production in bug 1131269, we should also enable this for Android builds. (As discussed in April 15 mobile meeting by jmaher)
(Assignee)

Comment 1

2 years ago
:jmaher is there a bug for implementing this and providing data on the SETA side of things?
Flags: needinfo?(jmaher)
(Assignee)

Updated

2 years ago
Assignee: nobody → kmoir
Summary: enable SETA for Android → enable SETA for Android in buildbot scheduling, configs
hey, this is a great bug for it.  Vaibhav and I plan to get the data for Android integrated into the solution.  We wanted to get a better understanding of it before figuring out how to make it really work.  Initial analysis shows huge wins though.

Updated

2 years ago
Flags: needinfo?(jmaher)
(Assignee)

Comment 3

2 years ago
Oh, I was thinking that it might be good to have a bug for the buildbot configs etc. and another bug for changing the SETA app to provide Android data.
(Assignee)

Comment 4

2 years ago
Created attachment 8596599 [details] [diff] [review]
bug1155362tools.patch

action to add new config_seta_mobile.py link on masters
(Assignee)

Comment 5

2 years ago
Created attachment 8596605 [details] [diff] [review]
bug1155362bbconfigs.patch

new config_seta_mobile.file + modifications to add it to mobile_config.py. It has to be separate file from config_seta.py for desktop because otherwise there are key errors due to the different platforms supported.

As before, the steps to deploy are 
1) land seta_config_mobile.py
2) reconfig
3) run custom action to add link
4) land changes to mobile_config.py
5) reconfig
6) -> fewer tests run on m-i and fx-team
(Assignee)

Updated

2 years ago
Attachment #8596599 - Flags: review?(bugspam.Callek)
(Assignee)

Updated

2 years ago
Attachment #8596605 - Flags: review?(bugspam.Callek)

Updated

2 years ago
Attachment #8596599 - Flags: review?(bugspam.Callek) → review+

Updated

2 years ago
Attachment #8596605 - Flags: review?(bugspam.Callek) → review+
In production: https://hg.mozilla.org/build/buildbot-configs/rev/2bd0fd3b12e3
(Assignee)

Updated

2 years ago
Attachment #8596599 - Flags: checked-in+
(Assignee)

Comment 7

2 years ago
Comment on attachment 8596605 [details] [diff] [review]
bug1155362bbconfigs.patch

new links created on masters so SETA on Android should be enabled on the next reconfig
Attachment #8596605 - Flags: checked-in+
In production: https://hg.mozilla.org/build/buildbot-configs/rev/6551b6f52a01
(Assignee)

Comment 9

2 years ago
I've verified on treeherder that this is enabled for Android 2.3 armv7 api9

Once we have SETA data for Android 4.3 (which was only enabled this week), we can add this to the seta configs.  I'll leave this bug open to account for that.
(Assignee)

Comment 10

2 years ago
RyanVM and jmaher noticed a scheduling issue for Android armv7 API 9 where SETA is enabled

For example, looking at today's builds on m-i.  M-3 is a test that should run less frequently according to SETA.  However, it doesn't appear that it is running often enough. i.e. with a timeout of 3400 and a skipcount of 10, it only ran on the 6:02 and the 9:51 build on inbound
9:51 All Android 2.3 tests scheduled
Android 2.3 API9 opt Mochitest Mochitest M(3)
Requested: Tue Apr 28, 11:05:05
Started: Tue Apr 28, 11:05:33
Ended: Tue Apr 28, 11:38:18 
9:45 SETA subset
9:40 a SETA subset
8:57 a SETA subset
8:30 a SETA subset
8:21 a SETA subset
8:12 a SETA subset
7:57 a SETA subset
6:02 All Android 2.3 tests scheduled (started 6:32)
example: Android 2.3 API9 opt Mochitest Mochitest M(3)
Requested: Tue Apr 28, 7:26:40
Started: Tue Apr 28, 7:30:03
Ended: Tue Apr 28, 7:59:12
(In reply to Kim Moir [:kmoir] from comment #10)
> For example, looking at today's builds on m-i.  M-3 is a test that should
> run less frequently according to SETA.  However, it doesn't appear that it
> is running often enough. i.e. with a timeout of 3400 and a skipcount of 10,
> it only ran on the 6:02 and the 9:51 build on inbound

I agree that we should find out why the scheduling isn't happening as expected, but did we actually miss any failures in the larger gap? 

I want us to be *more* aggressive in skipping useless tests, and this kind of happy accident can help us set a better, longer threshold.
(Assignee)

Comment 12

2 years ago
No, I don't think we missed any regressions. I was just trying to figure out why scheduling isn't working as I expected.
I love the ambition here.  One concern with skipping too many builds is that we have a much larger window to bisect the regression.  Just today, the sheriffs had a failure that was merged around as a side effect of SETA (bug 1159150, the second instance to date).  This is a worse case scenario as the work to fix it is more than fixing prior to a merge.

could we maybe keep our current levels for a bit, and we can work on tracking these merged regressions and finding a way to adjust processes/tools on the ateam side to avoid them completely.  That might take a few weeks, but it will prove where we are in the overall process.

So far the reception of SETA is great and our jobs have shows a significant decline.  Actually the feedback has been a large reduction in oranges to star :)
(Assignee)

Comment 14

2 years ago
:jmaher, how long do you think it will take to generate SETA data for Android 4.3? (I realize it was only enabled last week, but thought I'd ask :-)
Flags: needinfo?(jmaher)
ideally a month- we can roll with it and just add jobs as they come in- so in a technical sort of way, we can do it with no data.  Giving it a month ensures that we hit failures and see how it related to that of the other platforms.
Flags: needinfo?(jmaher)
(Assignee)

Comment 16

2 years ago
I don't see the Android 4.3 data in SETA yet, do you expect to appear soon?
Flags: needinfo?(jmaher)
I need to update all the jobs again- let me do that and get back to you- it takes a bit of sanity checking- so stay tuned.
Flags: needinfo?(jmaher)
I have updated SETA for 4.3 opt and debug.  Do let me know if there are issues with it.
(Assignee)

Comment 19

2 years ago
Created attachment 8616736 [details] [diff] [review]
bug1155362cleanup.patch

remove config_seta_mobile.py due to refactoring in bug 1131269
Attachment #8616736 - Flags: review?(coop)
(Assignee)

Comment 20

2 years ago
Created attachment 8616778 [details] [diff] [review]
bug1155362bbcleanup.patch

more cleanup
Attachment #8616778 - Flags: review?(coop)

Updated

2 years ago
Attachment #8616736 - Flags: review?(coop) → review+

Updated

2 years ago
Attachment #8616778 - Flags: review?(coop) → review+
(Assignee)

Updated

2 years ago
Attachment #8616736 - Flags: checked-in+
(Assignee)

Updated

2 years ago
Attachment #8616778 - Flags: checked-in+
(Assignee)

Comment 21

2 years ago
Created attachment 8616865 [details] [diff] [review]
bug1155362toolscleanup.patch

cleaned up masters, now don't need this action anymore
Attachment #8616865 - Flags: review?(coop)

Updated

2 years ago
Attachment #8616865 - Flags: review?(coop) → review+
(Assignee)

Updated

2 years ago
Attachment #8616865 - Flags: checked-in+
(Assignee)

Comment 22

2 years ago
This seems to be working well and Android 4.3 was enabled on Friday.  We have seta enabled for Android 2.3 and Android 4.3. I didn't bother disabling it for 4.0 since these are on there way out anyways.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.