Closed Bug 1383973 Opened 7 years ago Closed 7 years ago

Update android-stuff to not use tc-vcs

Categories

(Firefox Build System :: Task Configuration, task)

Product:
Firefox Build System
Component:
Task Configuration
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
mozilla57

People

(Reporter: dustin, Assigned: nalexander)

References

Details

Attachments

(4 files)

Bug 1383973 - Allow `docker-image:` that is not 'desktop-build'.
59 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1383973 - "Rebase" android-gradle-build back onto desktop-build.
59 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1383973 - Introduce single build script for `run: using: mozharness`.
59 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1383973 - Use `run: using` and build transforms to avoid tc-vcs in android-* tasks.
59 bytes, text/x-review-board-request
dustin
: review+
Details 
Tasks of this kind still use tc-vcs, and thus can't use the mozilla-unified repo (and also, break occasionally and in annoying ways)
Flags: needinfo?(garndt)
tc-vcs is being retired - https://github.com/taskcluster/taskcluster-rfcs/issues/43
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → INCOMPLETE
I think you got a little overzealous with your bug closing :)
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Greg, is there someone familiar with how to port to run-task/robustcheckout that could look at these tasks?  I'm happy to help too if pointed in the right direction.
Flags: needinfo?(garndt) → needinfo?(gps)
Actually I should have flagged Nick as I believe he knows most about these tasks.
Flags: needinfo?(gps) → needinfo?(nalexander)
(In reply to Greg Arndt [:garndt] from comment #4)
> Actually I should have flagged Nick as I believe he knows most about these
> tasks.

I did wonder about this.  I have a bunch of in-progress work against the existing code, so I must ask that this not happen immediately -- but anytime after the middle of next week should be fine :)

There are two types of task.  All of the tasks save android-gradle-dependencies are build-like: they just run a few more mach commands after the initial disable-compile-environment build is done.  They should be all straight-forward to port to robustcheckout.

The odd task out is android-gradle-dependencies, which is a build-like task with a bunch of special steps before, during, and after the checkout-and-build phases.  I don't think this will be hard to port to robustcheckout, but it probably isn't trivial to move this to run-task.  This task really is a (composite) toolchain task, so I expect that it will want to be rewritten to use the toolchain-task infrastructure that glandium has built rather than ported directly to run-task.

Sadly I cannot help (other than answering sporadic email and NI) until the middle of September!
Flags: needinfo?(nalexander)
Will the dependencies task need to run during that time?  It seems it's mostly to produce artifacts which are manually uploaded to tooltool -- and I assume that's usually you nick?

We could potentially migrate the rest and drop the dependencies task to tier 3 (so that any tc-vcs issues won't close trees), then plan to actually turn tc-vcs off in October or November after the dependencies become toolchains..
(In reply to Dustin J. Mitchell [:dustin] from comment #6)
> Will the dependencies task need to run during that time?  It seems it's
> mostly to produce artifacts which are manually uploaded to tooltool -- and I
> assume that's usually you nick?
> 
> We could potentially migrate the rest and drop the dependencies task to tier
> 3 (so that any tc-vcs issues won't close trees), then plan to actually turn
> tc-vcs off in October or November after the dependencies become toolchains..
(In reply to Nick Alexander :nalexander from comment #7)
> (In reply to Dustin J. Mitchell [:dustin] from comment #6)

Gah!  Sorry, two things at once in BZ and hit Save Changes too soon.

> > Will the dependencies task need to run during that time?  It seems it's
> > mostly to produce artifacts which are manually uploaded to tooltool -- and I
> > assume that's usually you nick?

My expectation is that we will want to bump dependencies in this timeframe.  There is active development on Fennec's Activity Stream implementation, and a general desire to move forward to newer Android SDKs.  It would be tough to freeze the toolchains while I'm out.

The task is manually initiated.
 
> > We could potentially migrate the rest and drop the dependencies task to tier
> > 3 (so that any tc-vcs issues won't close trees), then plan to actually turn
> > tc-vcs off in October or November after the dependencies become toolchains..

I think this is a reasonable plan.
I forgot it's manually initiated -- there's probably no need to change the tier then.  Let's just migrate all of the other tasks and plan to migrate gradle-dependencies when you're back.
dustin: I started looking at removing tc-vcs, since I don't want to install Node.js in new Debian-based Android build images.  It turned out that using robustcheckout is easy if I use the existing `run: using: mozharness` build-kind, making this ticket roughly equivalent to Bug 1302209; if you're happy with this patch series, that ticket will be resolved as well.
Thank you for doing this, Nick.
Comment on attachment 8893186 [details]
Bug 1383973 - Allow `docker-image:` that is not 'desktop-build'.

https://reviewboard.mozilla.org/r/164210/#review170004

::: taskcluster/taskgraph/transforms/job/mozharness.py:114
(Diff revision 1)
>          raise NotImplementedError("Cannot disabled mh magic arg passing via"
>                                    "'use-magic-mh-args' on docker-workers")
>  
>      # running via mozharness assumes desktop-build (which contains build.sh)
> +    if not taskdesc['worker']['docker-image']:
> -    taskdesc['worker']['docker-image'] = {"in-tree": "desktop-build"}
> +        taskdesc['worker']['docker-image'] = {"in-tree": "desktop-build"}

the comment here should be updated too
Attachment #8893186 - Flags: review?(dustin) → review+
Comment on attachment 8893187 [details]
Bug 1383973 - "Rebase" android-gradle-build back onto desktop-build.

https://reviewboard.mozilla.org/r/164212/#review170010
Attachment #8893187 - Flags: review?(dustin) → review+
Comment on attachment 8893188 [details]
Bug 1383973 - Introduce single build script for `run: using: mozharness`.

https://reviewboard.mozilla.org/r/164214/#review170014
Attachment #8893188 - Flags: review?(dustin) → review+
Comment on attachment 8893189 [details]
Bug 1383973 - Use `run: using` and build transforms to avoid tc-vcs in android-* tasks.

https://reviewboard.mozilla.org/r/164216/#review170018

And just to confirm, this leaves us with no uses of tc-vcs related to android?  I know last time we talked there would be one task left, but it seems that is no longer the case?

::: taskcluster/ci/build/kind.yml:20
(Diff revision 1)
>     - taskgraph.transforms.job:transforms
>     - taskgraph.transforms.task:transforms
>  
>  jobs-from:
>      - android.yml
> +    - android-stuff.yml

I sort of made up "android-stuff" because I didn't know what it was.  Is there a better name?
Attachment #8893189 - Flags: review?(dustin) → review+
Comment on attachment 8893189 [details]
Bug 1383973 - Use `run: using` and build transforms to avoid tc-vcs in android-* tasks.

https://reviewboard.mozilla.org/r/164216/#review170018

Yes, that's correct: no uses of tc-vcs (and, I think Node.js!) specific to Android.  When we last talked, I thought that `android-gradle-dependencies` was different enough that it would be hard to handle -- but it wasn't, as you can see :)

> I sort of made up "android-stuff" because I didn't know what it was.  Is there a better name?

A fine question.  I'm happy any which way, but right now, I somewhat like the "obviously temporary" name `android-stuff`.  You'll note that I made all of the jobs `kind: build`.  That was required to get sufficient TC scopes (I don't claim to understand the details here), but it's not semantically correct -- these jobs are syntactically `kind: build` but semantically `kind: test` (with one `kind: toolchain`, if you will).  I decided not to try to change the naming at this time, since I expect to follow-up the `kind: test` jobs eventually, and to move the `kind: toolchain` job to an actual toolchain job.

All that to say, I don't know if there's a better name right this moment, and I expect all of the jobs to migrate elsewhere eventually.
That all sounds great then :)
Pushed by nalexander@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e534aae10ab1
Allow `docker-image:` that is not 'desktop-build'. r=dustin
https://hg.mozilla.org/integration/autoland/rev/a0c9f426defa
"Rebase" android-gradle-build back onto desktop-build. r=dustin
https://hg.mozilla.org/integration/autoland/rev/e7daba76bf31
Introduce single build script for `run: using: mozharness`. r=dustin
https://hg.mozilla.org/integration/autoland/rev/f59d2660bccf
Use `run: using` and build transforms to avoid tc-vcs in android-* tasks. r=dustin
Blocks: 1409260
Product: TaskCluster → Firefox Build System
Assignee: nobody → nalexander
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: