Closed Bug 1384482 Opened 7 years ago Closed 6 years ago

Fennec: Make Treeherder and archive.mozilla.org broadcast the right Android API level (API-16+)

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task, P1)

Tracking

(firefox56+ fixed, firefox57+ fixed)

RESOLVED FIXED
Tracking Status
firefox56 + fixed
firefox57 + fixed

People

(Reporter: ioana.chiorean, Assigned: jlorenzo)

References

Details

Attachments

(13 files, 4 obsolete files)

59 bytes, text/x-review-board-request
mtabara
: review+
jlorenzo
: checked-in+
Details
51 bytes, text/x-github-pull-request
jlorenzo
: review+
jlorenzo
: checked-in+
Details | Review
47 bytes, text/x-github-pull-request
emorley
: review+
jlorenzo
: checked-in+
Details | Review
57 bytes, text/x-github-pull-request
mtabara
: review+
jlorenzo
: checked-in+
Details | Review
59 bytes, text/x-review-board-request
rail
: review+
jlorenzo
: checked-in+
Details
59 bytes, text/x-review-board-request
mtabara
: review+
jlorenzo
: checked-in+
Details
59 bytes, text/x-review-board-request
mtabara
: review+
jlorenzo
: checked-in-
Details
55 bytes, text/x-github-pull-request
jlorenzo
: review+
jlorenzo
: checked-in+
Details | Review
47 bytes, text/x-github-pull-request
emorley
: review+
jlorenzo
: checked-in+
Details | Review
3.91 KB, patch
mtabara
: review+
jlorenzo
: checked-in+
Details | Diff | Splinter Review
48 bytes, text/x-github-pull-request
Details | Review
6.50 KB, patch
mtabara
: review+
jlorenzo
: checked-in+
Details | Diff | Splinter Review
1.71 KB, patch
mtabara
: review+
jlorenzo
: checked-in+
Details | Diff | Splinter Review
Part of Bug 1316462 (submitted as per old version process, similar to Bug 1219094) 

First in my mind:
- updater 
- releases page
- ftp
What is the first version that will drop support for API 15?

The regular Fennec doesn't have a built-in updater, so nothing to do there or with Balrog rules.

The MozillaOnline Fennec does still use the built-in updater. I'm not sure if it's sending the API version in the update ping. Ben, do you know?

We will need to update the configuration in the Google Play Store to stop offering the new version to old versions of Android. Johan, do you know where that needs to happen?
Flags: needinfo?(jlorenzo)
Flags: needinfo?(ioana.chiorean)
Flags: needinfo?(bhearsum)
Summary: RelRng work for dropping API 15 for android fennec builds → RelEng work for dropping API 15 for android fennec builds
(In reply to Chris AtLee [:catlee] from comment #1)
> What is the first version that will drop support for API 15?
> 
> The regular Fennec doesn't have a built-in updater, so nothing to do there
> or with Balrog rules.
> 
> The MozillaOnline Fennec does still use the built-in updater. I'm not sure
> if it's sending the API version in the update ping. Ben, do you know?

Maybe...I see pings like this for GMP: https://aus5.mozilla.org:443/update/3/GMP/54.0/20170608225136/Android_arm-eabi-gcc3/en-US/release/Linux%2019/default/default/update.xml

The "19" in the OS_VERSION field could be the API version? We have a bunch of different ones in the logs:
Linux%2010
Linux%2012
Linux%2013
Linux%2015
Linux%2016
Linux%2017
Linux%2018
Linux%2019
Linux%2021
Linux%2022
Linux%2023
Linux%2024
Linux%2025
Linux%2026

...and 26 is the current API version. Reading https://dxr.mozilla.org/mozilla-central/source/xpcom/base/nsSystemInfo.cpp#763 (which is used in builting OS_VERSION) seems to confirm that, but I'd want to double check with someone more knowledgable before relying on that.
Flags: needinfo?(bhearsum)
(In reply to Chris AtLee [:catlee] from comment #1)
> What is the first version that will drop support for API 15?

will be 56 or 57 - still in debate atm

> The regular Fennec doesn't have a built-in updater, so nothing to do there
> or with Balrog rules.
Oh. I followed the old bug (Bug 1219094) and though it is still the case.
Flags: needinfo?(ioana.chiorean)
(In reply to Chris AtLee [:catlee] from comment #1)

> The regular Fennec doesn't have a built-in updater, so nothing to do there
> or with Balrog rules.

https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/updater/UpdateService.java


> We will need to update the configuration in the Google Play Store to stop
> offering the new version to old versions of Android. Johan, do you know
> where that needs to happen?

The new APK will include minSdkVersion=16, which should mean it's not offered in Play to earlier devices.
Per the conclusion just made earlier today we are dropping API 15 in 56, already check-in needed: https://bugzilla.mozilla.org/show_bug.cgi?id=1316462#c63. My understanding after discussing with the Fennec frontend team is the Releng work here should be independent of making  "dropping API 15" work? Otherwise this bug will become super urgent to handle before the 56 merge day (next Wednesday).(and we'll be sorry if that's the case)
Flags: needinfo?(catlee)
See Also: → 1250999
Priority: -- → P1
(In reply to Chris AtLee [:catlee] from comment #1)
> We will need to update the configuration in the Google Play Store to stop
> offering the new version to old versions of Android. Johan, do you know
> where that needs to happen?

If I understand correctly, we're going to replace an API by a more recent one, just like we did in bug 1155801. Google Play seems to handle this kind of deprecation quite well. I looked back at the Artifact library[1]. I compared Firefox 45.0.2 and 46.0. The new API level is there.

However, we'll need some changes in our toolchain, just like Sylvestre did last time. More importantly, we'll need to support the new API level in MozApkPublisher:push_apk.py[2]. get_apk.py should be updated[3], but that's not urgent.

Sylvestre, do you confirm that's the only things you had to do back then?


Now, for the new parts, we'll also have to change pushapkscript. We may have to update the APK detection mechanism[4] and tell pushapkscript that central[5] will expect a new architecture. This part may need to be done after bug 1316462 comment 63 reaches central. This may break the daily nightly job even more[6].


[1] https://play.google.com/apps/publish/?#ArtifactLibraryPlace:p=org.mozilla.firefox
[2] Changes should be similar to bug 1258099. We'll likely have to update the tests that weren't present back then.
[3] Like bug 1258096
[4] https://github.com/mozilla-releng/pushapkscript/blob/f6073dcd5f5218e513408a830c753825d09b1d7c/pushapkscript/apk.py
[5] Still called "aurora" because of the dawn project, https://github.com/mozilla-releng/pushapkscript/blob/f6073dcd5f5218e513408a830c753825d09b1d7c/pushapkscript/apk.py#L12
[6] Which has been red for 6 hours, because of bug 1369668
Flags: needinfo?(jlorenzo) → needinfo?(sledru)
See Also: → 1219094, 1258099, 1258096
yeah, this isn't really hard to do. Maybe we should use this as an opportunity to document this!
Flags: needinfo?(sledru)
(In reply to Sylvestre Ledru [:sylvestre] from comment #7)
> yeah, this isn't really hard to do. Maybe we should use this as an
> opportunity to document this!

Cool! That works for me!
Flags: needinfo?(catlee)
Warning! API-16+ has made central, but it's still branded as API-15+. We'll have to rename the jobs in-tree. This can be done as part of this job. I said in bug 1316462 comment 68, that this can be renamed in a follow up patch, in the future.

The good thing is, we don't need comment 6 to be done ASAP. Like jobs in-tree, this is just a matter of renaming.
Got confused using TryChooser to build on Android, with the following error, so a vote to clean up the TryChooser bits too!

> [task 2017-08-08T20:36:44.051139Z] Exception: Unknown platform(s) [android-api-15-frontend,android-api-15-gradle-dependencies] specified for try
Assignee: nobody → jlorenzo
Attached file Trychooser PR
Attachment #8895823 - Flags: review?(rgarbas)
(In reply to Johan Lorenzo [:jlorenzo] from comment #17)
> Created attachment 8895823 [details] [review]
> Trychooser PR

I removed android-api-15-gradle-dependencies and android-api-15-frontend because these platforms are not supported anymore the try syntax supported on Taskcluster (see comment 10). Nonetheless, android-api-XX-gradle-dependencies is still run even if you use the platform "android-api-XX"[1]. android-api-15-frontend isn't defined anywhere in Taskcluster. If we want to support it, let's do it in a separate bug. 

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=a68d9a175a03c60aa33ab9acb3fc7021111eb1eb&selectedJob=122004585
Comment on attachment 8895376 [details]
Bug 1384482 - Rename android-api-15 into android-api-16

https://reviewboard.mozilla.org/r/166566/#review172266

::: testing/mozharness/configs/merge_day/beta_to_release.py:18
(Diff revision 6)
>      "replacements": [
>          # File, from, to
>          ("{}{}".format(d, f),
>          "ac_add_options --with-branding=mobile/android/branding/beta",
>          "ac_add_options --with-branding=mobile/android/branding/official")
> -        for d in ["mobile/android/config/mozconfigs/android-api-15/",
> +        for d in ["mobile/android/config/mozconfigs/android-api-16/",

one caution is that we use m-c's merge day scripts for beta->release and I don't know if that would cause issues around here.
Blocks: 1389420
Thanks for the warning, Callek! I reverted the changes on leave that file and left a pointer to the follow up bug, bug 1389420.

Apart from that, this try push[1] is green with the exception of the l10n job[2]. That job failed because, http://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-api-16 isn't created yet:

> [task 2017-08-10T13:53:28.093283Z] 13:53:28     INFO -  http://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-api-16/en-US/fennec-57.0a1.en-US.android-arm.apk:
> [task 2017-08-10T13:53:28.093496Z] 13:53:28     INFO -  2017-08-10 13:53:28 ERROR 404: Not Found.

I don't what we can do for it, unless landing the patch on central and let central creates this folder.

But before, I landed this patch on date[3] and triggered a nightly graph there[4]. 

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=975790b48d5c0a0352014639f77c5d59a6d1bdda
[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=975790b48d5c0a0352014639f77c5d59a6d1bdda&selectedJob=122283247
[3] https://treeherder.mozilla.org/#/jobs?repo=date&revision=099d970a6517d9226626528e15d3669467e52c53
[4] https://tools.taskcluster.net/groups/dpBcGjPpTWy00KXcSfWWLQ
Attached file Treeherder PR
Thanks for the help on IRC, Ed! Here are the new platforms added.

I determined them by doing the following:

1. Download the JSON of the task graph for try[1], a regular push[2], and a nightly graph[3]
2. On these 3 files, run:
> jq '.[] | .task.extra.treeherder.machine.platform' task-graph.json | sort | uniq
3. Get the Android platforms from it

[1] https://queue.taskcluster.net/v1/task/CLHwq5mDT0a1dapbbtuPbg/runs/0/artifacts/public/task-graph.json
[2] https://queue.taskcluster.net/v1/task/GXBnDjDHSl65ol4KYss0OA/runs/0/artifacts/public/task-graph.json
[3] https://queue.taskcluster.net/v1/task/dpBcGjPpTWy00KXcSfWWLQ/runs/0/artifacts/public/task-graph.json
Attachment #8896236 - Flags: review?(emorley)
Attachment #8896234 - Attachment is obsolete: true
Comment on attachment 8896236 [details] [review]
Treeherder PR

Looks great - thank you!
Attachment #8896236 - Flags: review?(emorley) → review+
Attached file beetmoverscript PR
Failure detected by staging a nightly on date[1]. Beetmover wrongly picked firefox_nightly instead of fennec_nightly[2].

[1] See comment 22, link 4
[2] https://tools.taskcluster.net/groups/dpBcGjPpTWy00KXcSfWWLQ/tasks/A8y96FHnSQGKplPT_bwGoQ/runs/0/logs/public%2Flogs%2Flive_backing.log#L5
Attachment #8896891 - Flags: review?(mtabara)
Attachment #8896891 - Flags: review?(mtabara) → review+
Summary: RelEng work for dropping API 15 for android fennec builds → Fennec: Make Treeherder and archive.mozilla.org broadcast the right Android API level
Summary: Fennec: Make Treeherder and archive.mozilla.org broadcast the right Android API level → Fennec: Make Treeherder and archive.mozilla.org broadcast the right Android API level (API-16+)
I tested the presence of leftovers with these 2 commands:
> find . -iregex '^.*api[ -_]*15.+$' # Make sure no file is named after API 15
> grep -ri -E 'api[ _-]*15' * # Make sure no file references another file
Green nightly graph at: https://tools.taskcluster.net/groups/IstGHfhOTruc65AN8WpGDg

The only failure is on aarch64, which was already broken on date before the patch. Moreover, the patch doesn't change aarch64.
Depends on: 1391622
I think I broke android-api-16 with the latest date merge for wv testing. Sorry about that. If we still need that on date, I can help fix.
Comment on attachment 8895376 [details]
Bug 1384482 - Rename android-api-15 into android-api-16

https://reviewboard.mozilla.org/r/166566/#review176200

Just to make sure I understandn correctly, these changes are supposed to land on central and ride the trains, right? We're not uplifting anything yet to beta, nor release.

::: mobile/android/config/mozconfigs/android-api-16-frontend/nightly:32
(Diff revision 10)
> -# From here on, like ../android-api-15/nightly.
> +# From here on, like ../android-api-16/nightly.
>  
>  ac_add_options --enable-profiling
>  
>  # Android
> +# Warning: Before increasing the with-android-min-sdk value, please note several places in and out

+1 for the comment!

::: mobile/android/config/mozconfigs/android-api-16-frontend/nightly:32
(Diff revision 10)
> -# From here on, like ../android-api-15/nightly.
> +# From here on, like ../android-api-16/nightly.
>  
>  ac_add_options --enable-profiling
>  
>  # Android
> +# Warning: Before increasing the with-android-min-sdk value, please note several places in and out

Potential nit: "Warning for developers" maybe to draw attention even more?

::: mobile/android/config/mozconfigs/android-api-16-gradle-dependencies/nightly:34
(Diff revision 10)
>  ac_add_options --disable-tests
>  
>  ac_add_options --enable-profiling
>  
>  # Android
> +# Warning: Before increasing the with-android-min-sdk value, please note several places in and out

Same super small nit as above, but definitely optional.

::: mobile/android/config/mozconfigs/android-api-16/debug:8
(Diff revision 10)
>  # Global options
>  ac_add_options --enable-debug
>  ac_add_options --enable-marionette
>  
>  # Android
> +# Warning: Before increasing the with-android-min-sdk value, please note several places in and out

Same super small nit as above, but definitely optional.

::: mobile/android/config/mozconfigs/android-api-16/debug-artifact:13
(Diff revision 10)
>  
>  # Global options
>  ac_add_options --enable-debug
>  ac_add_options --enable-marionette
>  
> -. "$topsrcdir/mobile/android/config/mozconfigs/android-api-15/nightly"
> +. "$topsrcdir/mobile/android/config/mozconfigs/android-api-16/nightly"

Anyh reason we're not adding the same comment here as for the others?

::: testing/mozharness/configs/merge_day/beta_to_release.py:18
(Diff revision 10)
>      "replacements": [
>          # File, from, to
>          ("{}{}".format(d, f),
>          "ac_add_options --with-branding=mobile/android/branding/beta",
>          "ac_add_options --with-branding=mobile/android/branding/official")
> +        # TODO Bug 1389420: Rename api-15 into api-16 after Firefox 57 made Beta

++ bug filing against merge day bug. Yes, as you said, I think this needs to happen after 56.0 hits release, 57 hits beta so that the changes are still to api15 for 56.0 in release (they will be riding the trains and will eventually hit 57 in November).

::: testing/mozharness/configs/releases/bouncer_fennec.py:11
(Diff revision 10)
>              "check_uptake": True,
>              "alias": "fennec-latest",
>              "ssl-only": True,
>              "add-locales": False,  # Do not add locales to let "multi" work
>              "paths": {
> -                "android-api-15": {
> +                "android-api-16": {

These are riding the trains, ride? You're not uplifting this to beta, nor release, correct?
Attachment #8895376 - Flags: review?(mtabara) → review+
(In reply to Johan Lorenzo [:jlorenzo] from comment #34)
> Comment on attachment 8895376 [details]
> Bug 1384482 - Rename android-api-15 into android-api-16
> 
> Review request updated; see interdiff:
> https://reviewboard.mozilla.org/r/166566/diff/10-11/

Hm. Lots of other things changed in this interdiff. Not sure I'm following some of those changes but if you want me to review that again[1], can we quickly touch base on IRC/vidyo in the next days to discuss about this? I think I'm missing some of the context. Thanks!

[1]: https://reviewboard.mozilla.org/r/166566/diff/10-11/
Flags: needinfo?(jlorenzo)
The interdiff is because of a rebase on top central, which contained Callek's bug 1390983. The only conflict was in taskcluster/ci/nightly-l10n/kind.yml, which was pretty simple to deal with. The rest is just garbage, generated by HG. You can see it's not part of the final patch.

I'm going to address review comments in the next diff.
Flags: needinfo?(jlorenzo)
For nightly updates, I think you'll need to update 
  https://hg.mozilla.org/build/tools/file/default/lib/python/release/platforms.py#l22 
too, so that balrogworker via 
  https://hg.mozilla.org/build/tools/file/default/lib/python/balrog/submitter/cli.py#l278
uses the right build target when submitting into the Balrog blob.
Comment on attachment 8901166 [details]
Bug 1384482 - Add Android API level 16 for balrog submission

https://reviewboard.mozilla.org/r/172642/#review177976
Attachment #8901166 - Flags: review?(rail) → review+
Comment on attachment 8901207 [details]
Bug 1384482 - Make Fennec relpro use api-16 on beta

https://reviewboard.mozilla.org/r/172680/#review177996

Looks good to me. As discussed on IRC, you also need to tweak bb-configs - https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/config.py#l1478
We'd need to land those first since bbcustom is using those, then this patch + a reconfig.
Attachment #8901207 - Flags: review?(mtabara) → review+
Comment on attachment 8901230 [details]
Bug 1384482 - Add android-api-16 config

https://reviewboard.mozilla.org/r/172696/#review178004

::: mozilla/config.py:1580
(Diff revision 1)
> +                'script_maxtime': int(5.5 * 3600),
> +            },
> +            'product_name': 'firefox',
> +            'unittest_platform': 'android-api-16-opt',
> +            'app_name': 'browser',
> +            'brand_name': 'Minefield',

Nit: this sounds like a badass branding name :) Not sue if this changed for api-16; again, sounds like it can be ignored.

::: mozilla/config.py:1583
(Diff revision 1)
> +            'unittest_platform': 'android-api-16-opt',
> +            'app_name': 'browser',
> +            'brand_name': 'Minefield',
> +            'base_name': 'Android armv7 API 16+ %(branch)s',
> +            'mozconfig': 'in_tree',
> +            'src_mozconfig': 'mobile/android/config/mozconfigs/android-api-16/nightly',

https://hg.mozilla.org/releases/mozilla-beta/file/tip/mobile/android/config/mozconfigs is missing this file but I'm pretty sure this is harmless. It's unused in builbotcustom at this point.
Comment on attachment 8901230 [details]
Bug 1384482 - Add android-api-16 config

https://reviewboard.mozilla.org/r/172696/#review178012

Oups, forgot to r+!
Attachment #8901230 - Flags: review?(mtabara) → review+
Comment on attachment 8895376 [details]
Bug 1384482 - Rename android-api-15 into android-api-16

https://reviewboard.mozilla.org/r/166566/#review176200

> Potential nit: "Warning for developers" maybe to draw attention even more?

I think this warning is intented to everybody who's willing to bump the value. Specifying developers may reduce the scope of the comment :/

I'd prefer to leave it as is.

> Anyh reason we're not adding the same comment here as for the others?

`ac_add_options --with-android-min-sdk=16` is missing there. This file just depends on `../android-api-16/nightly`. If somebody changes something here, they'd have to change another file, which contain the min-sdk argument.

> These are riding the trains, ride? You're not uplifting this to beta, nor release, correct?

Like said on IRC last week, this has to land on beta too. The reason is: API 16 is part of 56 because of Bug 1316462.
Attachment #8896236 - Flags: checked-in+
Attachment #8896891 - Flags: checked-in+
Keywords: leave-open
Depends on: 1394385
See Also: → 1394747
Attached file releasetasks PR
Was r+'d by mtabara on the PR.
Attachment #8902215 - Flags: review+
Alright, I think we're good to land the main patch (attachment 8895376 [details]) on central, then beta. To sum up:
* Comment 54 introduced [1] in the patch. Otherwise jamun remains broken
* Comment 55 is just a rebase on top of central. No big issue found.
* Nightlies on central were tested against the date branch[2]. I backed out the old patch and relanded the latest version before comment 54 to see if it was still valid. We were green there[3] (modulo push-apk which is expected to fail on this branch)
* Betas were tested against jamun[4]. The latest try on it was good: graph 2 is fully green[5] and graph 1[6] has a 3 failures which are unrelated to this bug. Failures are tracked in bug 1394747. We might see an issue related to uptake monitoring. I'll look after this.

As a consequence, I believe all the pieces have been identified, patched, or on the edge of being landed (releasetasks - attachment 8902215 [details] [review] - requires the main patch to reach beta, first). I'm going to land the main patch on central. See how next nightly goes, and uplift it to beta.

[1] https://hg.mozilla.org/projects/jamun/rev/aea9b8d20e6979cc94b8332a373a79cbbcd7852a
[2] https://treeherder.mozilla.org/#/jobs?repo=date&revision=d7b67f2388ef88c9fee95a05c74d06db8b5cf8a6
[3] https://tools.taskcluster.net/groups/c-Ehoji7Tn6mJHm2m8Q3BQ
[4] https://treeherder.mozilla.org/#/jobs?repo=jamun&revision=aea9b8d20e6979cc94b8332a373a79cbbcd7852a
[5] https://tools.taskcluster.net/groups/Sr03C9t0TKehE6jn6gT53A
[6] https://tools.taskcluster.net/groups/T2TDo_JNTP2LMfneMcHONA
Pushed by jlorenzo@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e3c7045f22a7
Rename android-api-15 into android-api-16 r=mtabara
Attached file Treeherder PR part 2
New platforms appeared on central: 
* android-4-0-armv7-api-16
* android-4-0-armv7-api-16-old-id
* android-4-2-armv7-api16
* android-4-3-armv7-api-16
* android-4-4-armv7-api16
* android-5-0-armv8-api16
* android-6-0-armv8-api16
* android-7-1-armv8-api16
Attachment #8902673 - Flags: review?(emorley)
Comment on attachment 8902673 [details] [review]
Treeherder PR part 2

If these are taskcluster jobs, I would prefer to fix the job names instead. If they are from buildbot, then it probably isn't worth it.
Attachment #8902673 - Flags: review?(emorley) → review-
I think I scrambled the nomenclature after a rebase. I initially wanted to be consistent in having api-16 everywhere, but I realized there are some out of tree workers which are sticking to "api16". Then, let's just comply to that.

I found all the culprits with `grep -r 'android-.-.-armv7-api-16' *`. That matches with the extra lines I added in the first revision of attachment 8902673 [details] [review]
Attachment #8903083 - Flags: review?(mtabara)
Comment on attachment 8903083 [details] [diff] [review]
Gecko: Bug 1384482 - Comply to Treeherder's nomenclature r=mtabara

Sgtm!
Attachment #8903083 - Flags: review?(mtabara) → review+
Comment on attachment 8902673 [details] [review]
Treeherder PR part 2

Culprits found above. I deleted the duplicated entries. What do you think?
Attachment #8902673 - Flags: review- → review?(emorley)
Comment on attachment 8895823 [details] [review]
Trychooser PR

Was merged by garbas at https://github.com/mozilla-releng/services/commit/2ff8f2fa508606fdd786546baf5b2929a13f3111
Attachment #8895823 - Flags: review?(rgarbas)
Attachment #8895823 - Flags: review+
Attachment #8895823 - Flags: checked-in+
Pushed by jlorenzo@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7599d3db781e
Comply to Treeherder's nomenclature r=mtabara
Comment on attachment 8902673 [details] [review]
Treeherder PR part 2

Many thanks!

If it's any help, at some point in the future I'd like to stop using the `android-4-2-armv7-api16` -> `Android 4.2 API16+` type mappings in Treeherder and ideally specify everything in-tree. This will be more viable once we stop using buildbot.
Attachment #8902673 - Flags: review?(emorley) → review+
Attachment #8902673 - Flags: checked-in+
Attachment #8895376 - Flags: checked-in+
Only jamun was activated.
Attachment #8903130 - Flags: review?(mtabara)
Attachment #8903130 - Flags: review?(mtabara) → review+
Status update: All the identified pieces are landed on/for central/beta. I'm keeping this bug open until next Fennec beta is out (56.0b9 which goes to build next Monday). If no there is no missing part left, I'll close this bug.
Comment on attachment 8903130 [details] [diff] [review]
Buildbotcustom: Bug 1384482 - Activate api-16 on beta r=mtabara

I backed this out because an unrelated change in buildbot-configs had a travis test failure, see https://travis-ci.org/mozilla-releng/build-buildbot-configs/jobs/270640812. Looks like we only run test-masters.sh when buildbot-configs changes.
Attachment #8903130 - Flags: checked-in+ → checked-in-
I don't think we need to change files like mozilla/release-fennec-mozilla-beta.py because we don't build Fennec on BB anymore. What do you think catlee?
Attachment #8903653 - Flags: review?(catlee)
Attachment #8903653 - Flags: review?(catlee) → review+
Depends on: 1396092
Comment on attachment 8903653 [details] [diff] [review]
BB configs: Add android-api-16 to global platforms r=catlee

https://hg.mozilla.org/build/buildbot-configs/rev/fe2caf39964ffd47dcf250ff30d836b2d8a130a8
Attachment #8903653 - Flags: checked-in+
Comment on attachment 8903130 [details] [diff] [review]
Buildbotcustom: Bug 1384482 - Activate api-16 on beta r=mtabara

Let's see if attachment 8903653 [details] [diff] [review] fixes the breakage.

Relanded at https://hg.mozilla.org/build/buildbotcustom/rev/78a330f14120fea9edbb74c4c75f4fafb9a0d9d5

I'm going to run BB configs to see if the breakage still occurs
Attachment #8903130 - Flags: checked-in- → checked-in+
Depends on: 1396356
Even after backing out my patches, the same error remains:
* https://travis-ci.org/mozilla-releng/build-buildbotcustom/builds/271324677
* https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/271324987

I filed 1396356 to track these failures.
Bug 1396356 is fixed. I relanded the BB configs patch, which resulted to:

> INFO  - starting to print log file '/home/travis/build/mozilla-releng/build-buildbot-configs/test-output/bm75-try1-universal-DAx0OS-checkconfig.log'
> INFO  - Traceback (most recent call last):
> INFO  -   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/py27/lib/python2.7/site-packages/buildbot-0.8.2_hg_7f288e5082f5_production_0.8-py2.7.egg/buildbot/scripts/runner.py", line 1042, in doCheckConfig
> INFO  -     ConfigLoader(configFileName=configFileName)
> INFO  -   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/py27/lib/python2.7/site-packages/buildbot-0.8.2_hg_7f288e5082f5_production_0.8-py2.7.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
> INFO  -     self.loadConfig(configFile, check_synchronously_only=True)
> INFO  -   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/py27/lib/python2.7/site-packages/buildbot-0.8.2_hg_7f288e5082f5_production_0.8-py2.7.egg/buildbot/master.py", line 809, in loadConfig
> INFO  -     % (b['name'], n))
> INFO  - ValueError: builder Android armv7 API 16+ try build uses undefined slave bld-linux64-ec2-001

I backed out the patch again.

On Friday and on IRC, :catlee suggested to add android-api-16 to mozilla-beta only. I'm going to try this approach, carrying over the r+.
Attachment #8903653 - Attachment is obsolete: true
Attachment #8904004 - Flags: review+
Depends on: 1396447
Depends on: 1396550
I was incredibly surprised 56.0b9[1] got all green, even though BuildBot didn't get all the expected changes (attachment 8903130 [details] [diff] [review] and attachment 8904004 [details] [diff] [review] were both backed out).

I dug into each of these build bot jobs. None of them is a false positive. Here's the detail of why:

* [beetmover] fennec mozilla-beta push to releases => The job doesn't reference any api-level. The script doesn't take that as a parameter. It just finds every file in the candidate directory[2] and copy them under another namespace. In other words, it worked because the in-tree beetmover jobs are correctly configured.
* fennec mozilla-beta bncr sub => The config file is in tree[3]. Thanks to this config file, bouncer is agnostic about the API level. For instance, [4] downloads the android-api-16 APK.
* fennec mozilla-beta bouncer aliases => Same as the previous job. It just changes the updates fennec-beta-latest. Today, [5] returns the same APK as [4]
* fennec mozilla-beta mark release as shipped => Ship-it doesn't know anything about what architecture is shipped or not.
* fennec mozilla-beta version bump => Independent from APKs architectures.
* fennec mozilla-beta uptake monitoring => Last but not least. I was worried about this job, but it managed to get the right payload. It turns out everything is deffined in releasetasks[6]


As a consequence, I'll forget about the remaining BB changes, and I'll backout the ones that turned out to be useless. 


[1] https://tools.taskcluster.net/groups/RuNCJvZRSDazUTfLOgneVQ
[2] https://dxr.mozilla.org/mozilla-central/rev/3ecda4678c49ca255c38b1697142b9118cdd27e7/testing/mozharness/scripts/release/push-candidate-to-releases.py#186
[3] https://dxr.mozilla.org/mozilla-central/rev/3ecda4678c49ca255c38b1697142b9118cdd27e7/testing/mozharness/configs/releases/bouncer_fennec_beta.py#11
[4] http://download.mozilla.org/?product=Fennec-56.0b9&os=android&lang=multi
[5] http://download.mozilla.org/?product=fennec-beta-latest&os=android&lang=multi
[6] https://github.com/mozilla-releng/releasetasks/blob/01375954f27f44b3f349330cb02da73015fdd02a/releasetasks/templates/mobile/uptake_monitoring.yml.tmpl#L35 and https://github.com/mozilla-releng/releasetasks/blob/01375954f27f44b3f349330cb02da73015fdd02a/releasetasks/release_configs/prod_mozilla-beta_fennec_full_graph.yml#L6
This backs out attachment 8901230 [details] and adds a comment for our future selves. I understood that the API level is just used to define slaves names. In reality, all the config is hold in-tree (per previous comment).
Attachment #8904004 - Attachment is obsolete: true
Attachment #8905099 - Flags: review?(mtabara)
Comment on attachment 8904004 [details] [diff] [review]
BB configs: Add android-api-16 to mozilla-beta r=catlee

Was backed out, and it's not needed at all (see previous comments).
Attachment #8904004 - Flags: checked-in-
Comment on attachment 8901230 [details]
Bug 1384482 - Add android-api-16 config

Will be removed by attachment 8905099 [details] [diff] [review].
Attachment #8901230 - Flags: checked-in+ → checked-in-
Comment on attachment 8905099 [details] [diff] [review]
Bug 1384482 - Strip useless android-api-16 config r=mtabara

Sorry for slipping this during initial review. Nice investigation, good to know for future. Am a bit surprised though, especially because of this[1].

Could this be related to the fact that we didn't add mozilla-beta here[2], hence 56.0b9 behaved like a api15 and didn't fail? 

[1]: https://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py#l2201
[2]: https://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py#l2199
Attachment #8905099 - Flags: review?(mtabara) → review+
I didn't fully backed out attachment 8901207 [details], because it has a handy refactor. I just deleted the condition about using sometimes api-16 and sometimes api-15.
Attachment #8903130 - Attachment is obsolete: true
Attachment #8905109 - Flags: review?(mtabara)
Attachment #8905109 - Flags: review?(mtabara) → review+
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #89)
> Could this be related to the fact that we didn't add mozilla-beta here[2],
> hence 56.0b9 behaved like a api15 and didn't fail? 

That's correct! b9 ran against builders dedicated to api-15. This is not a big deal, because the 6 jobs in comment 85 don't need specific machines. They can run on any linux machine. That's why we can keep android-api-15 there, and the jobs will keep passing.
(In reply to Johan Lorenzo [:jlorenzo] from comment #91)
> (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #89)
> > Could this be related to the fact that we didn't add mozilla-beta here[2],
> > hence 56.0b9 behaved like a api15 and didn't fail? 
> 
> That's correct! b9 ran against builders dedicated to api-15. This is not a
> big deal, because the 6 jobs in comment 85 don't need specific machines.
> They can run on any linux machine. That's why we can keep android-api-15
> there, and the jobs will keep passing.

Yep, makes sense.
Comment on attachment 8905099 [details] [diff] [review]
Bug 1384482 - Strip useless android-api-16 config r=mtabara

https://hg.mozilla.org/build/buildbot-configs/rev/8300468d9ccece27bc9f6547c1a0066814e17cf6

Reconfigs deployed to production.
Attachment #8905099 - Flags: checked-in+
Comment on attachment 8903083 [details] [diff] [review]
Gecko: Bug 1384482 - Comply to Treeherder's nomenclature r=mtabara

Was checked in, in comment 75.
Attachment #8903083 - Flags: checked-in+
Blocks: 1398848
I should have closed this bug 3 months ago.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.