1105439 - Decision tasks for gecko

Status: RESOLVED FIXED

(Taskcluster :: General, defect)

Description

[James Lal [:lightsofapollo]]

Decision task logic for gecko deals with pulling initial repository and running the initial commit logic.

Comment 1

[James Lal [:lightsofapollo]]

Attachment: MozReview Request: bz://1105439/lightsofapollo

Comment 2

[James Lal [:lightsofapollo]]

/r/1093 - Bug 1105439 - Part 1 New gaia_props.py util for getting gaia version needed by the gecko r=garndt, wcosta
/r/1095 - Bug 1105439 - Part 2 Update builder to use tc-vcs for b2g-desktop/mulet (and related repos) r=wcosta, garndt
/r/1097 - Bug 1105439 -Part 3  Initial decision task logic for gecko r=wcosta, garndt
/r/1099 - Bug 1105439 - Part 4 Framework for inheriting one task from another. r=wcosta, garndt
/r/1101 - Bug 1105439 - Part 5 Use inheritance instead of additional-properties r=wcosta, garndt
/r/1103 - Bug 1105439 - Part 7 Update scopes and debug properties for emulator builds r=wcosta, garndt
/r/1105 - Bug 1105439 - First pass at speeding up emulators r=wcosta, garndt
/r/1107 - Bug 1105439 - Finalize decision task and simplify emulator caching logic r=wcosta, garndt

Pull down these commits:

hg pull review -r 63a74503ea583e7fdd3120f133448b5c27ea61ad

Comment 3

[James Lal [:lightsofapollo]]

Note this is based off alder _not_ mozilla-central or b2g-inbound.

Comment 4

[Wander Lairson Costa]

https://reviewboard.mozilla.org/r/1091/#review653

::: testing/docker/builder/Dockerfile
(Diff revision 1)
> +ADD bin /home/worker/bin

Any particular reason why we don't use a setup script anymore?

::: testing/docker/builder/git.env
(Diff revision 1)
> +GECKO_HEAD_REPOSITORY=https://github.com/mozilla/gecko-dev

Do we use github ot git.mozilla.org for builds?

::: testing/taskcluster/taskcluster_graph/templates.py
(Diff revision 1)
> +    for key in source:

I think it is better to do:

```
for key, value in source.items():
```

Since source[key] is done a lot inside the for loop.

Comment 5

[Wander Lairson Costa]

Comment on attachment 8531105 [details]
MozReview Request: bz://1105439/lightsofapollo

lgtm, just some minor comments

Comment 6

[Greg Arndt [:garndt]]

https://reviewboard.mozilla.org/r/1091/#review655

::: testing/taskcluster/tests/test_templates.py
(Diff revision 1)
> +

nit: extra line

::: testing/taskcluster/mach_commands.py
(Diff revision 1)
> +class InheritTryme(object):

What is this used for?

::: testing/taskcluster/mach_commands.py
(Diff revision 1)
> +        help='URL for "base" repository to clone')

help text is the same as --base-repository.  Won't this be the repostiroty where the commit is?

::: testing/taskcluster/mach_commands.py
(Diff revision 1)
>      @CommandArgument('--owner',

Should owner be marked as required here as well since it was changed up above?

::: testing/taskcluster/mach_commands.py
(Diff revision 1)
> -    @CommandArgument('--b2g-config',
> +        help='URL for "base" repository to clone')

Same comment about help text above.

Comment 7

[Greg Arndt [:garndt]]

https://reviewboard.mozilla.org/r/1091/#review657

::: testing/docker/builder/git.env
(Diff revision 1)
> +GECKO_BASE_REPOSITORY=https://github.com/mozilla/gecko-dev

Where are these env files used or are they just there for defaults if someone wants to setup their environment variables prior to running the mach commands?

Comment 8

[Greg Arndt [:garndt]]

https://reviewboard.mozilla.org/r/1091/#review659

::: testing/taskcluster/entrypoints/try.yml
(Diff revision 1)
> +        image: ubuntu:13.10

This is based off of ubuntu image that does not have the entrypoint scripts and gecko like the decision docker image that's built elsewhere.

Also noticed that this is doing something similar to taskcluster/tasks/decision/try.yml, which uses decision:0.0.3

Not sure if both should exist or one should have been removed.

::: testing/taskcluster/tasks/decision/try.yml
(Diff revision 1)
> +    - 'docker-worker:cache:tc-vcs-public-sources'

Do we need to cache gecko?  Entrypoint looks to see if it exists and if not, will do a clone.  Or will we just rely on it pulling the s3 archive each time?

Comment 9

[Greg Arndt [:garndt]]

Comment on attachment 8531105 [details]
MozReview Request: bz://1105439/lightsofapollo

r+ with addressing the issues raised in reviewboard, specifically about multiple try.yml definitions and prototype code that should be removed.

Comment 10

[James Lal [:lightsofapollo]]

(In reply to Wander Lairson Costa [:wcosta] from comment #4)
> https://reviewboard.mozilla.org/r/1091/#review653
> 
> ::: testing/docker/builder/Dockerfile
> (Diff revision 1)
> > +ADD bin /home/worker/bin
> 
> Any particular reason why we don't use a setup script anymore?

Putting the commands directly in the Dockerfile greatly increases the chance you can hit the cache which docker uses which means _much_ faster docker build times.

> 
> ::: testing/docker/builder/git.env
> (Diff revision 1)
> > +GECKO_HEAD_REPOSITORY=https://github.com/mozilla/gecko-dev
> 
> Do we use github ot git.mozilla.org for builds?

We do this file is just for testing the docker image file as needed.

> 
> ::: testing/taskcluster/taskcluster_graph/templates.py
> (Diff revision 1)
> > +    for key in source:
> 
> I think it is better to do:
> 
> ```
> for key, value in source.items():
> ```
> 
> Since source[key] is done a lot inside the for loop.

Done

Comment 11

[James Lal [:lightsofapollo]]

https://reviewboard.mozilla.org/r/1091/#review673

> This is based off of ubuntu image that does not have the entrypoint scripts and gecko like the decision docker image that's built elsewhere.
> 
> Also noticed that this is doing something similar to taskcluster/tasks/decision/try.yml, which uses decision:0.0.3
> 
> Not sure if both should exist or one should have been removed.

Removed this was a bad idea !

> Do we need to cache gecko?  Entrypoint looks to see if it exists and if not, will do a clone.  Or will we just rely on it pulling the s3 archive each time?

We now cache the underlying "sources" (tarballs of the clones) in all images so this will clone the revision each time without caching gecko (somewhat intentionally) so decision tasks always are in as clean of state as possible.

Comment 12

[James Lal [:lightsofapollo]]

https://reviewboard.mozilla.org/r/1091/#review671

> help text is the same as --base-repository.  Won't this be the repostiroty where the commit is?

Actually "base" revision is what we but the "head" is where the revision is (this is taken from how github describes this type of workflow not sure how right the names are otherwise)

Comment 13

[James Lal [:lightsofapollo]]

http://hg.mozilla.org/projects/alder/rev/deadeba968b5

Comment 14

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/59ed4da143d3
https://hg.mozilla.org/mozilla-central/rev/01b7b89e24c6
https://hg.mozilla.org/mozilla-central/rev/c91ca9b89bba
https://hg.mozilla.org/mozilla-central/rev/0697d74236f6
https://hg.mozilla.org/mozilla-central/rev/45a45e412520
https://hg.mozilla.org/mozilla-central/rev/42d96044ef47
https://hg.mozilla.org/mozilla-central/rev/5b1f2e0fda35
https://hg.mozilla.org/mozilla-central/rev/6cc2fd0206da

Comment 16

[James Lal [:lightsofapollo]]

Attachment: MozReview Request: Bug 1105439 - Part 1 New gaia_props.py util for getting gaia version needed by the gecko r=garndt, wcosta

Comment 17

[James Lal [:lightsofapollo]]

Attachment: MozReview Request: Bug 1105439 - Part 2 Update builder to use tc-vcs for b2g-desktop/mulet (and related repos) r=wcosta, garndt

Comment 18

[James Lal [:lightsofapollo]]

Attachment: MozReview Request: Bug 1105439 -Part 3 Initial decision task logic for gecko r=wcosta, garndt

Comment 19

[James Lal [:lightsofapollo]]

Attachment: MozReview Request: Bug 1105439 - Part 4 Framework for inheriting one task from another. r=wcosta, garndt

Comment 20

[James Lal [:lightsofapollo]]

Attachment: MozReview Request: Bug 1105439 - Part 5 Use inheritance instead of additional-properties r=wcosta, garndt

Comment 21

[James Lal [:lightsofapollo]]

Attachment: MozReview Request: Bug 1105439 - Part 7 Update scopes and debug properties for emulator builds r=wcosta, garndt

Comment 22

[James Lal [:lightsofapollo]]

Attachment: MozReview Request: Bug 1105439 - First pass at speeding up emulators r=wcosta, garndt

Comment 23

[James Lal [:lightsofapollo]]

Attachment: MozReview Request: Bug 1105439 - Finalize decision task and simplify emulator caching logic r=wcosta, garndt

Comment 24

[Pete Moore [:pmoore][:pete]]

Resetting Version and Target Milestone that accidentally got changed...