1219094 - releng work for dropping api-11 through api-14

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Richard Newman [:rnewman]]

Part of Bug 1155801.

We'll only do this if we land the changes in that bug!

jlund, could you move this to the right component?

Comment 1

[Richard Newman [:rnewman]]

Note that this might also affect the updater -- Honeycomb Nightly and Aurora users will need to be orphaned on their current build, and 14+ Nightly and Aurora users will need to jump to the api-14 build.

Comment 2

[Jordan Lund (:jlund)]

I can take care of the renaming parts and also the logic for how we are going to phase out api-11 and integrate api-14 assuming we ride the trains on this.

It sounds like there is a Friday meeting to discuss how we are proceeding with branches like esr, etc.

As I am not involved with the discussions or Bug 1155801, I will wait until I get pinged in this bug before doing anything.

Comment 3

[Ed Morley [:emorley]]

I wonder if it's worth using a more generic name that won't get out of date again in the future?

The problem is that with the rename now, things like perf stats will appear as a new platform, so make it hard to follow trends - when in fact other than dropping old Honeycomb code, it's the same platform.

Really we have two builds, a "old android api" and "newer android api" build. The actual version numbers used for both are somewhat academic (unless someone is debugging failures that only occur on one, where they can look up what target API is used at the time).

Comment 4

[Jordan Lund (:jlund)]

(In reply to Ed Morley [:emorley] from comment #3)
> I wonder if it's worth using a more generic name that won't get out of date
> again in the future?

make sense. Though, I believe in the future we would like to do more than one split. So if we could keep that in mind with the naming, wfm.

Comment 5

[Geoff Brown [:gbrown]]

:rnewman -- Is it time to proceed with this bug?

Comment 6

[Jordan Lund (:jlund)]

this bug will still consist of:

1) Prepare to rename Android API-11 builds to API-14

but will also include

2) stop serving api-11 -> api-13 based nightlies to users.

rnewman: I'm assuming we will be riding the trains with this? e.g. we still want to serve updates to honeycomb aurora users until 46 reaches m-a, correct?

Comment 7

[Richard Newman [:rnewman]]

Yes. Nightly is now 14+, and that'll ride the trains to release in 46.

Comment 8

[Jordan Lund (:jlund)]

rail: can you sanity check: https://aus4-admin.mozilla.org/rules#nightlytest-honeycomb

we want to stop serving honeycomb users m-c nightly updates.

if r+, I'll change the channel name to 'nightly'.

I should note, as https://bugzilla.mozilla.org/show_bug.cgi?id=1155801#c63 finished on Jan 1st, I would imagine a large subset of honeycomb users have already received updates for a nightly that is newer than that.

Comment 9

[Jordan Lund (:jlund)]

list of versions derived from the following convo:

14:20:25 <jlund> https://en.wikipedia.org/wiki/Android_version_history suggests honeycomb is represented by 3.0–3.2.6
14:20:43 <jlund> anyone know where I can get an official explicit list of versions within that range?
14:20:57 <jlund> looking to disable updates to honeycomb users
14:21:36 <jlund> this requires a special balrog rule. e.g. for api-9 split, I use the following list: 2.3, 2.3.1, 2.3.2
14:24:33 <•mfinkle> jlund: your list sounds reasable
14:24:37 <•mfinkle> jlund: http://developer.android.com/reference/android/os/Build.VERSION_CODES.html
14:24:56 <•mfinkle> doesn't show the minor/minor release
14:25:24 <•mfinkle> jlund: but we do know that anything in the 3.x range is honeycomb only
14:25:40 <•mfinkle> the next higher release is 4.0
14:27:36 <jlund> mfinkle: hm, I think I need to specify the full version including minor dot as we don't match substrings in os version lookup
14:28:02 <•mfinkle> jlund: start with your list then


also, here is the discussion that determined we likely have had users try and fail to update on honeycomb..

14:30:21 <jlund> mfinkle: k. one more question while I have you, https://bugzilla.mozilla.org/show_bug.cgi?id=1155801#c63 suggests we dropped honeycomb on the 1st
14:30:25 <firebot> Bug 1155801 — FIXED, [e-mail] — Drop Honeycomb support from Firefox builds
14:30:37 <jlund> our current nightly points to http://download.cdn.mozilla.net/pub/mobile/nightly/2016/01/2016-01-04-03-02-17-mozilla-central-android-api-11/
14:30:45 <jlund> with buildid 20160104030217
14:31:43 <jlund> so I am assuming there are honeycomb users who have gotten a build that is newer than bug 1155801 
14:32:04 <•mfinkle> if they did, they are not happy
14:32:57 <•mfinkle> jlund: any honeycomb user that tries to use a current nightly should get an install failure

Comment 10

[Jordan Lund (:jlund)]

hmm, my assumption of OS VERSION may be wrong; it might allow substring values. e.g. 'Windows' would catch 'Windows_NT'. In which case, numbered versions would be bad. e.g. '3.2' would catch '2.3.2' and '3.2.2'

Comment 11

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #10)
> hmm, my assumption of OS VERSION may be wrong; it might allow substring
> values. e.g. 'Windows' would catch 'Windows_NT'. In which case, numbered
> versions would be bad. e.g. '3.2' would catch '2.3.2' and '3.2.2'

yes, it looks like we accept substrings: https://github.com/mozilla/balrog/blob/078e23b660d621ae841af172b9f706231f1d5ecf/auslib/db.py#L710

this might cause spurious results as our list of possible versions grow ..

Comment 12

[Jordan Lund (:jlund)]

I've set update rate to 0% for all non api-9-10 fennec m-c updates while we figure this out today so we don't keep trying to serve updates to honeycomb users.

Comment 13

[Rail Aliiev [:rail]]

1) yeah, substring matching may bite us in the future, but probably we can deal with that later
2) please bump the priority to something >100 so it's easier to spot these watersheds
3) maybe also add the build target to narrow down the criteria? I don't know exactly what the value is, it can be found in the update URLs.

otherwise lgtm

Comment 14

[Jordan Lund (:jlund)]

ftr - yesterday evening, I reverted the update rate back to 100% for api-11 (now api-14) and put the No-Update for honeycomb rule into action on the nightly channel

I didn't include build target because I wanted to keep it in sync with the other apk splits.

Tomorrow morning I'll address the s/11/14 part.

Comment 15

[Richard Newman [:rnewman]]

jlund: I think we're going to change 14 to 15 right now; 14 has ~0 users, so we might as well make that change to avoid churn. I'll let you know when you can s/11/15 :D

Comment 16

[Jordan Lund (:jlund)]

(In reply to Richard Newman [:rnewman] from comment #15)
> jlund: I think we're going to change 14 to 15 right now; 14 has ~0 users, so
> we might as well make that change to avoid churn. I'll let you know when you
> can s/11/15 :D

okay sounds good. I'll hold off changing things until 15 is the new minimum and sticks

Comment 17

[Jordan Lund (:jlund)]

api 15+ has landed on central. I've added: 4.0, 4.0.1, and 4.0.2 to the No-Update list for m-c nightly based on versions with api-14: http://developer.android.com/guide/topics/manifest/uses-sdk-element.html

rail: looks like we are going to need to address this sooner rather than later bc '4.0' is in '4.0.3' and '4.0.3' == api-15 (we only want to block api 11-14) ..

I have a series of patches lined up that I'm hoping to land tomorrow morning.

Comment 18

[Rail Aliiev [:rail]]

You can add another rule for 4.0.3 with higher priority and point it to Fennec-mozilla-central-nightly-latest. This way balrog would catch 4.0.3 in that rule and serve updates as expected. Does it make sense?

Comment 19

[Jordan Lund (:jlund)]

(In reply to Rail Aliiev [:rail] from comment #18)
> You can add another rule for 4.0.3 with higher priority and point it to
> Fennec-mozilla-central-nightly-latest. This way balrog would catch 4.0.3 in
> that rule and serve updates as expected. Does it make sense?

done. 4.0.3 should be getting updates again

Comment 20

[Jordan Lund (:jlund)]

Attachment: 160108_bug_1219094_drop_api_11_through_14-treeherder.patch

Hey cam,

we are dropping api-11 on trunk. We need to support api-11 on other gecko based branches.

How does this look?

Comment 21

[Jordan Lund (:jlund)]

Attachment: 160108_bug_1219094_drop_api_11_through_14-bbot-cfgs.patch

1st up. because this is riding the trains, we need to phase out api-11 on each gecko bump.

Comment 22

[Jordan Lund (:jlund)]

Attachment: 160108_bug_1219094_drop_api_11_through_14-bbot-tools.patch

iirc, the mozconfig whitelist file is used by releaserunner/sanity? in which case, I'll have to uplift this change to 'old-release-runner' branch too?

I think it's safe to remove the old non-split 'android' platform across our buildbot masters. We actually need to do this anyway because bbot test slaves complain about having too many builders attached to them

Comment 23

[Jordan Lund (:jlund)]

Attachment: 160108_bug_1219094_drop_api_11_through_14-gecko-tc-mh.patch

this one is a bit funnier. Because some android builds (afaik) are still scheduled through buildbot, we have to do this in two stages.

1) keep api-11 around until buildbot is reconfiged with trunk based api-15 patch
2) remove api-11 files

however we can:

1) change things wholesale at once for anything taskcluster scheduling/task based. \o/

I know for a fact that b2gdroid and dummy apk are only scheduled via TC so that's why I didn't bother even keeping the api-11 bits for those around at all.

Comment 24

[Jordan Lund (:jlund)]

Attachment: 160108_bug_1219094_drop_api_11_through_14-puppet.patch

simple one to keep up with tc routes

Comment 25

[Jordan Lund (:jlund)]

Attachment: 160108_bug_1219094_drop_api_11_through_14-cloud-tools.patch

again, need to keep api-11 around here.

Comment 26

[Jordan Lund (:jlund)]

Attachment: 160108_bug_1219094_drop_api_11_through_14-slave-health.patch

last one ... I think

Let's see what I forgot :)

sorry for the r? storm rail. Who would have thought there would be this much logic let alone places to change for a silly name. feel free to 302 any of the requests on; I figured it be easier to just iterate thought them all at once for most them.

Comment 27

[Jordan Lund (:jlund)]

Comment on attachment 8705921 [details] [diff] [review]
160108_bug_1219094_drop_api_11_through_14-bbot-cfgs.patch

if you are going to run dump_masters/checkconfig against this patch, you will need to add this to your env: https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla-tests/config_seta.py#39

this is because the following doesn't exist yet in http://alertmanager.allizom.org/data/setadetails/:

+                  "android-4-3-armv7-api15": ("android-api-15", ["ubuntu64_vm_armv7_mobile", "ubuntu64_vm_armv7_large"])

speaking of which, jmaher: would you be able to help me add the api-15 based apk split to the above url?

Comment 28

[Jordan Lund (:jlund)]

Comment on attachment 8705925 [details] [diff] [review]
160108_bug_1219094_drop_api_11_through_14-gecko-tc-mh.patch

hey nalexander :)

fyi - I didn't touch any of this mozbuild stuff: https://dxr.mozilla.org/mozilla-central/search?q=api-11+path%3Amozbuild&redirect=true&case=false

Not too familiar with it. not sure if we have to keep any of the api-11 bits around or if we can just swap it to 15 immediately. seems like you have commited to these two files a fair bit.

Comment 29

[Rail Aliiev [:rail]]

Comment on attachment 8705921 [details] [diff] [review]
160108_bug_1219094_drop_api_11_through_14-bbot-cfgs.patch

Review of attachment 8705921 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/mobile_config.py
@@ +3293,4 @@
>      for platform in branch['platforms']:
>          if not platform in PLATFORMS:
>              continue
> +        if platform not in ('android-api-9'):

A nit. Missing trailing coma, ('android-api-9') is converted to 'android-api-9', in which case:

 'android' in ('android-api-9')
 True

@@ +3455,5 @@
>  remove_suite_from_slave_platform(BRANCHES, PLATFORMS, 'mochitest-chrome', 'ubuntu64_vm_large', branches_to_keep=trunk_branches)
>  
>  # bug 1214362 - Enable DOM push Mochitests on Android Fennec
> +BRANCHES['cedar']['platforms']['android-api-15']['ubuntu64_vm_armv7_mobile']['opt_unittest_suites'] += ANDROID_4_3_MOCHITEST_PUSH
> +BRANCHES['cedar']['platforms']['android-api-15']['ubuntu64_vm_armv7_large']['debug_unittest_suites'] += ANDROID_4_3_MOCHITEST_PUSH

Not related to this patch, just wonder why these configs are not in the projects_branches.py file... I guess, because it's too strict?

::: mozilla/config.py
@@ +1758,5 @@
> +            'nightly_signing_servers': 'dep-signing',
> +            'dep_signing_servers': 'dep-signing',
> +            'use_mock': True,
> +            'mock_target': 'mozilla-centos6-x86_64-android',
> +            'mock_packages': ['autoconf213', 'mozilla-python27-mercurial',

I wonder if we need these mock_packages set here... I thought that we use the ones defined in mozharness.

Comment 30

[Rail Aliiev [:rail]]

Comment on attachment 8705925 [details] [diff] [review]
160108_bug_1219094_drop_api_11_through_14-gecko-tc-mh.patch

Review of attachment 8705925 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 31

[Rail Aliiev [:rail]]

Thank you for taking care of this!

Comment 32

[Joel Maher ( :jmaher ) (UTC -8)]

:jlund, I am not sure exactly what we need to add to SETA, I can help get it added- SETA takes existing jobs that it finds and generates data based on that, is it possible that we don't have these jobs on inbound?  I have done some preliminary work to add api15 to the list for the SETA backend.  looking on mozilla-inbound, I don't see api15 jobs.  We can preseed stuff, I just need to know what we are going to be running and I can do that.

Comment 33

[Jordan Lund (:jlund)]

(In reply to Rail Aliiev [:rail] from comment #31)
> Thank you for taking care of this!

thank you for the reviews :)

I think as for the order in landing these, the treeherder one should go first so I'll wait for the review of that one.

Comment 34

[Jordan Lund (:jlund)]

(In reply to Joel Maher (:jmaher) from comment #32)
> :jlund, I am not sure exactly what we need to add to SETA, I can help get it
> added- SETA takes existing jobs that it finds and generates data based on
> that, is it possible that we don't have these jobs on inbound?  I have done
> some preliminary work to add api15 to the list for the SETA backend. 
> looking on mozilla-inbound, I don't see api15 jobs.  We can preseed stuff, I
> just need to know what we are going to be running and I can do that.


I have never touched seta but here is what I discovered:

we will need to pre-seed this because buildbot won't reconfig unless api-15 exists in setadetails

the end goal:

* replace api-11 based builds and tests with api-15

which meant needing to patch the following:

diff --git a/mozilla-tests/config_seta.py b/mozilla-tests/config_seta.py
index aff4c71..c123d14 100644
--- a/mozilla-tests/config_seta.py
+++ b/mozilla-tests/config_seta.py
@@ -22,7 +22,7 @@ seta_platforms = {"Rev4 MacOSX Snow Leopard 10.6": ("macosx64", ["snowleopard"])
                   "Rev7 MacOSX Yosemite 10.10.5": ("macosx64", ["yosemite_r7"]),
                   "Ubuntu Code Coverage VM 12.04 x64": ("linux64-cc", ["ubuntu64_vm", "ubuntu64_vm_lnx_large"]),                  
                   "android-2-3-armv7-api9": ("android-api-9", ["ubuntu64_vm_mobile", "ubuntu64_vm_large"]),
-                  "android-4-3-armv7-api11": ("android-api-11", ["ubuntu64_vm_armv7_mobile", "ubuntu64_vm_armv7_large"])
+                  "android-4-3-armv7-api15": ("android-api-15", ["ubuntu64_vm_armv7_mobile", "ubuntu64_vm_armv7_large"])
                   }

which broke https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla-tests/config_seta.py?case=true&from=config_seta.py#115

because that code gets it's data from something like: http://alertmanager.allizom.org/data/setadetails/?date=2016-01-11&buildbot=1&branch=mozilla-inbound&inactive=1

but uses platforms from:https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla-tests/config_seta.py?case=true&from=config_seta.py#14

so, iiuc, we will need to add the following jobs to setadetails from today forward:
copy all android-4-3-armv7-api11 jobtypes from: https://pastebin.mozilla.org/8856403 and s/android-4-3-armv7-api11/android-4-3-armv7-api15

this all assumes that data on alertmanager.allizom.org/data/setadetails is manually entered. if it is updated via automation, there might be a way to hack this together by bumping the '11's to '15's in two stages at this line: https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla-tests/config_seta.py?case=true&from=config_seta.py#25

jmaher: does this make more sense now?

Comment 35

[Cameron Dawson [:camd]]

Comment on attachment 8705919 [details] [diff] [review]
160108_bug_1219094_drop_api_11_through_14-treeherder.patch

This looks good to me.  Please need-info me if you need me to land this for you.  Thanks!  :)

Comment 36

[Nick Alexander :nalexander [he/him]]

(In reply to Jordan Lund (:jlund) from comment #28)
> Comment on attachment 8705925 [details] [diff] [review]
> 160108_bug_1219094_drop_api_11_through_14-gecko-tc-mh.patch
> 
> hey nalexander :)
> 
> fyi - I didn't touch any of this mozbuild stuff:
> https://dxr.mozilla.org/mozilla-central/search?q=api-
> 11+path%3Amozbuild&redirect=true&case=false
> 
> Not too familiar with it. not sure if we have to keep any of the api-11 bits
> around or if we can just swap it to 15 immediately. seems like you have
> commited to these two files a fair bit.

I thought I responded to this.  In any case, their should be no build system changes needed, 'cuz right now we're lieing: we say we're building API 11 but in reality we're building API 15+.

Comment 37

[GitHub Autolander Bot]

Attachment: [treeherder] lundjordan:api-15 > mozilla:master

Comment 38

[Joel Maher ( :jmaher ) (UTC -8)]

 Ibelieve my work is done here for now- clearing the needinfo.

Comment 39

[Jordan Lund (:jlund)]

(In reply to Cameron Dawson [:camd] from comment #35)
> Comment on attachment 8705919 [details] [diff] [review]
> 160108_bug_1219094_drop_api_11_through_14-treeherder.patch
> 
> This looks good to me.  Please need-info me if you need me to land this for
> you.  Thanks!  :)

thanks for the review. looks like I do not have write access. I created a PR. Cam, Could you land for me please: https://github.com/mozilla/treeherder/pull/1243

also, next time should I use reviewable for r? requests?

Comment 40

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/91e11a06ec9741669a974593b7050f1cedd3779e
Bug 1219094 - Add support for Android api-15

Comment 41

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/137131f8a45c150bd06c15469cac0db660fd7e80
Bug 1219094 - Add support for Android api-15

Comment 42

[Ed Morley [:emorley]]

(In reply to Jordan Lund (:jlund) from comment #39)
> thanks for the review. looks like I do not have write access. I created a
> PR. Cam, Could you land for me please:
> https://github.com/mozilla/treeherder/pull/1243
> 
> also, next time should I use reviewable for r? requests?

The process for contributing to Treeherder is now (we've only recently started using autolander):
1) File a bug in a Treeherder component (unless it's a one-line doc fix or something)
2) Create a pull request with that bug number in it
3) Wait for autolander to create the attachment in bugzilla that links to the PR (this will only work if the bug is in a whitelisted component; which are currently Firefox OS and Tree Management)
4) Flag r? on that attachment as normal

Even if autolander is whitelisted to allow the Release Engineering component, I still think there is an advantage of filing bugs for the Treeherder parts of these bugs -- people component watching can receive notifications of them. Had it not been by change I wouldn't have seen this bug or comment 39 etc :-)

Comment 43

[Ed Morley [:emorley]]

s/change/chance/

Comment 44

[Cameron Dawson [:camd]]

Thanks for merging this, Ed.  You're a rock-star, man.  :)
Dang, sorry I let this sit too long.

Comment 45

[Jordan Lund (:jlund)]

waiting on treeherder patch to land in production: http://whatsdeployed.io/?owner=mozilla&repo=treeherder&name[]=Stage&url[]=https://treeherder.allizom.org/revision.txt&name[]=Prod&url[]=https://treeherder.mozilla.org/revision.txt

will land the remaining patches after that.

Comment 46

[Jordan Lund (:jlund)]

Comment on attachment 8705928 [details] [diff] [review]
160108_bug_1219094_drop_api_11_through_14-slave-health.patch

looks like we are good to go with treeherder. patches are landing now

Comment 47

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/3752abafdf19

Comment 48

[Jordan Lund (:jlund)]

Attachment: 160122_bug_1219094_drop_api11_fix_base_jobs.patch

apparently I forgot at least one spot ;)

exception in task graph generation:

TemplatesException: "/Users/jlund/devel/mozilla/cleanRepos/mozilla-inbound/testing/taskcluster/tasks/builds/android_api_11_b2gdroid.yml" is not a file

Comment 49

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6a250cbf8c2c

Comment 50

[Jordan Lund (:jlund)]

Comment on attachment 8711163 [details] [diff] [review]
160122_bug_1219094_drop_api11_fix_base_jobs.patch

r+ by garndt via irc. thanks!

Comment 51

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/3752abafdf19
https://hg.mozilla.org/mozilla-central/rev/6a250cbf8c2c

Comment 52

[Nick Thomas [:nthomas] (UTC+12)]

Are there any plans to remove directories like mobile/android/config/mozconfigs/android-api-11/ to avoid future confusion ?

Comment 54

[Jordan Lund (:jlund)]

(In reply to Nick Thomas [:nthomas] from comment #52)
> Are there any plans to remove directories like
> mobile/android/config/mozconfigs/android-api-11/ to avoid future confusion ?

yep. I'll remove all of these for central and aurora now. needed an overlap last week where they both existed.

Comment 55

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/5f47728bef61

Comment 56

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #54)
> (In reply to Nick Thomas [:nthomas] from comment #52)
> > Are there any plans to remove directories like
> > mobile/android/config/mozconfigs/android-api-11/ to avoid future confusion ?
> 
> yep. I'll remove all of these for central and aurora now. needed an overlap
> last week where they both existed.

this is done now. removed api-11 mozconfigs on inbound and aurora

Comment 57

[Nick Thomas [:nthomas] (UTC+12)]

Thanks.

Comment 58

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/5f47728bef61