1405412 - Migrate Android SDK to toolchain dependencies

Status: RESOLVED FIXED

(Taskcluster :: General, enhancement)

Description

[Nick Alexander :nalexander [he/him]]

This ticket tracks turning the Android SDK bits of the Deps task into a proper toolchain task.

Comment 1

[Nick Alexander :nalexander [he/him]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=943f3878f550bccaa42a098f4f59b88197dc7f64

Comment 2

[Nick Alexander :nalexander [he/him]]

It'll be helpful to have Bug 1406196 out of the way while doing this work.

Comment 3

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1405412 - Pre: Allow toolchain task images to not cache tc-vcs.

Not all Docker images are configured for tc-vcs caches: in particular,
android-build is not configured.  Until we fully remove tc-vcs, this
will let toolchain tasks use non-tc-vcs caching images.

Review commit: https://reviewboard.mozilla.org/r/187378/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/187378/

Comment 4

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1405412 - Migrate Android SDK to android-sdk-linux toolchain task.

The only tricky piece here is that the resulting toolchain archive is
private, and uses a newly allocated Task Cluster scope
(queue:get-artifact:project/gecko/android-sdk/*) to restrict access to
the archive.  All SCM levels (1, 2, 3) have been given the new scope:
see https://tools.taskcluster.net/auth/roles/moz-tree:level:1 and
friends.

Review commit: https://reviewboard.mozilla.org/r/187380/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/187380/

Comment 5

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1405412 - Post: Remove JDK repackaging script.

I don't think (the output of) this script is used anywhere.

Review commit: https://reviewboard.mozilla.org/r/187382/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/187382/

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8916133 [details]
Bug 1405412 - Pre: Allow toolchain task images to not cache tc-vcs.

https://reviewboard.mozilla.org/r/187378/#review192814

::: taskcluster/taskgraph/transforms/job/toolchain.py:51
(Diff revision 1)
>          'public',
>          'internal',
>      ),
>  
> +    # If true, tc-vcs will be enabled.  Not supported on Windows.
> +    Required('tc-vcs', default=True): bool,

Any chance we could default this to False?

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8916134 [details]
Bug 1405412 - Migrate Android SDK to android-sdk-linux toolchain task.

https://reviewboard.mozilla.org/r/187380/#review192816

this is so cool :)

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8916137 [details]
Bug 1405412 - Post: Remove JDK repackaging script.

https://reviewboard.mozilla.org/r/187382/#review192818

Comment 9

[Nick Alexander :nalexander [he/him]]

Comment on attachment 8916133 [details]
Bug 1405412 - Pre: Allow toolchain task images to not cache tc-vcs.

https://reviewboard.mozilla.org/r/187378/#review192814

> Any chance we could default this to False?

I think we could, but I don't have a good way of discovering the (few? zero?) cases where it's actually required.  I think I'd rather leave it as it is now (to reduce risk), and then whoever finally excises "tc.?vcs" from the tree can kill it here too.

Comment 10

[Nick Alexander :nalexander [he/him]]

Comment on attachment 8916133 [details]
Bug 1405412 - Pre: Allow toolchain task images to not cache tc-vcs.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/187378/diff/1-2/

Comment 11

[Nick Alexander :nalexander [he/him]]

Comment on attachment 8916134 [details]
Bug 1405412 - Migrate Android SDK to android-sdk-linux toolchain task.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/187380/diff/1-2/

Comment 12

[Nick Alexander :nalexander [he/him]]

Comment on attachment 8916137 [details]
Bug 1405412 - Post: Remove JDK repackaging script.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/187382/diff/1-2/

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5bd941606a64
Pre: Allow toolchain task images to not cache tc-vcs. r=dustin
https://hg.mozilla.org/integration/autoland/rev/5f4ff0de7242
Migrate Android SDK to android-sdk-linux toolchain task. r=dustin
https://hg.mozilla.org/integration/autoland/rev/44f885bc7c22
Post: Remove JDK repackaging script. r=dustin

Comment 14

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backed out bug 1352599 and bug 1405412 for failing build at python/mozbuild/mozbuild/test/configure/lint.py::Lint::test_mobile_android:

bug 1352599
https://hg.mozilla.org/integration/autoland/rev/6c9706f9f534eb684abbb11c72cdb6f31d94c272
https://hg.mozilla.org/integration/autoland/rev/fbc942d188228d69b3f6d33eeaa45cce2a9e808a
https://hg.mozilla.org/integration/autoland/rev/64a408bde7d2f24606f0806ce36f27480251a442
https://hg.mozilla.org/integration/autoland/rev/ca38508b320378f568c295da00c90ae6edcd9ffd

bug 1405412
https://hg.mozilla.org/integration/autoland/rev/11376fd5366ebe76c926e2e919245e5039c0b9e4
https://hg.mozilla.org/integration/autoland/rev/a6fab9d472152a3582869bac6088eb31b2957403
https://hg.mozilla.org/integration/autoland/rev/0e81b421af3564b89e2a9a646af24d83c36ba9fc

Latest push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=cb5f4236d080bb250f7f485a584216f014a18fba&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable
Build log: https://treeherder.mozilla.org/logviewer.html#?job_id=136603646&repo=autoland

[task 2017-10-12T21:17:52.266Z] 21:17:52  WARNING - ../python/mozbuild/mozbuild/test/configure/lint.py::Lint::test_mobile_android TEST-UNEXPECTED-FAIL
[task 2017-10-12T21:17:52.266Z] 21:17:52     INFO - =================================== FAILURES ===================================
[task 2017-10-12T21:17:52.266Z] 21:17:52     INFO - ___________________________ Lint.test_mobile_android ___________________________
[task 2017-10-12T21:17:52.266Z] 21:17:52     INFO - self = <lint.Lint testMethod=test_mobile_android>
[task 2017-10-12T21:17:52.266Z] 21:17:52     INFO -     def test(self):
[task 2017-10-12T21:17:52.267Z] 21:17:52     INFO - >       return func(self, project)
[task 2017-10-12T21:17:52.267Z] 21:17:52     INFO - ../python/mozbuild/mozbuild/test/configure/lint.py:26:
[task 2017-10-12T21:17:52.267Z] 21:17:52     INFO - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[task 2017-10-12T21:17:52.267Z] 21:17:52     INFO - ../python/mozbuild/mozbuild/test/configure/lint.py:58: in lint
[task 2017-10-12T21:17:52.267Z] 21:17:52     INFO -     sandbox.run(os.path.join(topsrcdir, 'moz.configure'))
[task 2017-10-12T21:17:52.268Z] 21:17:52     INFO - ../python/mozbuild/mozbuild/configure/lint.py:35: in run
[task 2017-10-12T21:17:52.268Z] 21:17:52     INFO -     self.include_file(path)
[task 2017-10-12T21:17:52.268Z] 21:17:52     INFO - ../python/mozbuild/mozbuild/configure/__init__.py:389: in include_file
[task 2017-10-12T21:17:52.268Z] 21:17:52     INFO -     exec_(code, self)
[task 2017-10-12T21:17:52.268Z] 21:17:52     INFO - ../python/mozbuild/mozbuild/util.py:59: in exec_
[task 2017-10-12T21:17:52.269Z] 21:17:52     INFO -     exec(object, globals, locals)
[task 2017-10-12T21:17:52.269Z] 21:17:52     INFO - ../moz.configure:225: in <module>
[task 2017-10-12T21:17:52.269Z] 21:17:52     INFO -     include(include_project_configure)
[task 2017-10-12T21:17:52.269Z] 21:17:52     INFO - ../python/mozbuild/mozbuild/configure/__init__.py:693: in include_impl
[task 2017-10-12T21:17:52.269Z] 21:17:52     INFO -     self.include_file(what)
[task 2017-10-12T21:17:52.270Z] 21:17:52     INFO - ../python/mozbuild/mozbuild/configure/__init__.py:389: in include_file
[task 2017-10-12T21:17:52.270Z] 21:17:52     INFO -     exec_(code, self)
[task 2017-10-12T21:17:52.270Z] 21:17:52     INFO - ../python/mozbuild/mozbuild/util.py:59: in exec_
[task 2017-10-12T21:17:52.270Z] 21:17:52     INFO -     exec(object, globals, locals)
[task 2017-10-12T21:17:52.270Z] 21:17:52     INFO - ../mobile/android/moz.configure:130: in <module>
[task 2017-10-12T21:17:52.271Z] 21:17:52     INFO -     include('../../build/moz.configure/java.configure')
[task 2017-10-12T21:17:52.271Z] 21:17:52     INFO - ../python/mozbuild/mozbuild/configure/__init__.py:693: in include_impl
[task 2017-10-12T21:17:52.271Z] 21:17:52     INFO -     self.include_file(what)
[task 2017-10-12T21:17:52.271Z] 21:17:52     INFO - ../python/mozbuild/mozbuild/configure/__init__.py:389: in include_file
[task 2017-10-12T21:17:52.271Z] 21:17:52     INFO -     exec_(code, self)
[task 2017-10-12T21:17:52.272Z] 21:17:52     INFO - ../python/mozbuild/mozbuild/util.py:59: in exec_
[task 2017-10-12T21:17:52.272Z] 21:17:52     INFO -     exec(object, globals, locals)
[task 2017-10-12T21:17:52.272Z] 21:17:52     INFO - ../build/moz.configure/java.configure:86: in <module>
[task 2017-10-12T21:17:52.272Z] 21:17:52     INFO -     help='Path to proguard.jar')
[task 2017-10-12T21:17:52.272Z] 21:17:52     INFO - ../python/mozbuild/mozbuild/configure/lint.py:121: in option_impl
[task 2017-10-12T21:17:52.272Z] 21:17:52     INFO -     result = super(LintSandbox, self).option_impl(*args, **kwargs)
[task 2017-10-12T21:17:52.273Z] 21:17:52     INFO - ../python/mozbuild/mozbuild/configure/__init__.py:610: in option_impl
[task 2017-10-12T21:17:52.273Z] 21:17:52     INFO -     kwargs = {k: self._resolve(v) for k, v in kwargs.iteritems()
[task 2017-10-12T21:17:52.273Z] 21:17:52     INFO - ../python/mozbuild/mozbuild/configure/__init__.py:611: in <dictcomp>
[task 2017-10-12T21:17:52.273Z] 21:17:52     INFO -     if k != 'when'}
[task 2017-10-12T21:17:52.273Z] 21:17:52     INFO - ../python/mozbuild/mozbuild/configure/__init__.py:463: in _resolve
[task 2017-10-12T21:17:52.274Z] 21:17:52     INFO -     need_help_dependency)
[task 2017-10-12T21:17:52.274Z] 21:17:52     INFO - ../python/mozbuild/mozbuild/util.py:944: in method_call
[task 2017-10-12T21:17:52.274Z] 21:17:52     INFO -     cache[args] = self.func(instance, *args)
[task 2017-10-12T21:17:52.274Z] 21:17:52     INFO - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[task 2017-10-12T21:17:52.275Z] 21:17:52     INFO - self = {'msvs_version': <mozbuild.configure.SandboxDependsFunction object at 0x7f55f4...android': <mozbuild.configure.SandboxDependsFunction object at 0x7f55f40e29d0>}
[task 2017-10-12T21:17:52.275Z] 21:17:52     INFO - obj = <mozbuild.configure.DependsFunction proguard_jar_default()>
[task 2017-10-12T21:17:52.275Z] 21:17:52     INFO - need_help_dependency = True
[task 2017-10-12T21:17:52.275Z] 21:17:52     INFO -     @memoize
[task 2017-10-12T21:17:52.276Z] 21:17:52     INFO -     def _value_for_depends(self, obj, need_help_dependency=False):
[task 2017-10-12T21:17:52.276Z] 21:17:52     INFO -         with_help = self._help_option in obj.dependencies
[task 2017-10-12T21:17:52.276Z] 21:17:52     INFO -         if with_help:
[task 2017-10-12T21:17:52.276Z] 21:17:52     INFO -             for arg in obj.dependencies:
[task 2017-10-12T21:17:52.276Z] 21:17:52     INFO -                 if self._missing_help_dependency(arg):
[task 2017-10-12T21:17:52.277Z] 21:17:52     INFO -                     raise ConfigureError(
[task 2017-10-12T21:17:52.277Z] 21:17:52     INFO -                         "`%s` depends on '--help' and `%s`. "
[task 2017-10-12T21:17:52.277Z] 21:17:52     INFO -                         "`%s` must depend on '--help'"
[task 2017-10-12T21:17:52.277Z] 21:17:52     INFO -                         % (obj.name, arg.name, arg.name))
[task 2017-10-12T21:17:52.277Z] 21:17:52     INFO -         elif ((self._help or need_help_dependency) and
[task 2017-10-12T21:17:52.278Z] 21:17:52     INFO -               self._missing_help_dependency(obj)):
[task 2017-10-12T21:17:52.278Z] 21:17:52     INFO -             raise ConfigureError("Missing @depends for `%s`: '--help'" %
[task 2017-10-12T21:17:52.278Z] 21:17:52     INFO - >                                obj.name)
[task 2017-10-12T21:17:52.278Z] 21:17:52     INFO - E           ConfigureError: Missing @depends for `proguard_jar_default`: '--help'
[task 2017-10-12T21:17:52.279Z] 21:17:52     INFO - ../python/mozbuild/mozbuild/configure/lint.py:116: ConfigureError

Comment 15

[Nick Alexander :nalexander [he/him]]

Comment on attachment 8916133 [details]
Bug 1405412 - Pre: Allow toolchain task images to not cache tc-vcs.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/187378/diff/2-3/

Comment 16

[Nick Alexander :nalexander [he/him]]

Comment on attachment 8916134 [details]
Bug 1405412 - Migrate Android SDK to android-sdk-linux toolchain task.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/187380/diff/2-3/

Comment 17

[Nick Alexander :nalexander [he/him]]

Comment on attachment 8916137 [details]
Bug 1405412 - Post: Remove JDK repackaging script.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/187382/diff/2-3/

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bbc2354236d3
Pre: Allow toolchain task images to not cache tc-vcs. r=dustin
https://hg.mozilla.org/integration/autoland/rev/d4d6cad20606
Migrate Android SDK to android-sdk-linux toolchain task. r=dustin
https://hg.mozilla.org/integration/autoland/rev/27213145f391
Post: Remove JDK repackaging script. r=dustin

Comment 19

[Mike Hommey [:glandium]]

Comment on attachment 8916133 [details]
Bug 1405412 - Pre: Allow toolchain task images to not cache tc-vcs.

https://reviewboard.mozilla.org/r/187378/#review194606

::: taskcluster/taskgraph/transforms/job/toolchain.py:51
(Diff revision 3)
>          'public',
>          'internal',
>      ),
>  
> +    # If true, tc-vcs will be enabled.  Not supported on Windows.
> +    Required('tc-vcs', default=True): bool,

bwarf, just when I was testing a patch that entirely removes tc-vcs support (nothing uses it anymore afaict).

Comment 20

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/bbc2354236d3
https://hg.mozilla.org/mozilla-central/rev/d4d6cad20606
https://hg.mozilla.org/mozilla-central/rev/27213145f391

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

Yeah, that's why I wanted default=false.

Comment 22

[Nick Alexander :nalexander [he/him]]

Clearing NI since this has landed \o/  Sorry about the tc-vcs, folks -- somebody else can figure out if it's used anywhere.