1088232 - gecko: mach ci-docker-run

Status: RESOLVED FIXED

(Taskcluster :: General, defect, P1)

Description

[Wander Lairson Costa]

+++ This bug was initially created as a clone of Bug #1085505 +++

Add a mach command to "docker run" an image, and allow to pass a local repository for volume mount. Something like this (by :lightsofapollo):

mach ci-docker-run -e REPOSITORY=<local-mozilla-central-repo> -e REVISION=xxxx builder build-b2g-desktop.sh

Comment 1

[James Lal [:lightsofapollo]]

Note that -e means environment variable the REPOSITORY/REVISION are examples of what the current build uses we could easily have other vars (or no vars)

Comment 2

[Greg Arndt [:garndt]]

Perhaps something like this if I'm understanding correctly...

mach ci-docker-run --local-mozilla-central-repo=xxxx --revision=xxxx <image> <command>

If 'local-mozilla-central' isn't given, it will be cloned somewhere locally before running anything docker related.

Once that's done, the docker run command would be constructed as:

docker run -v <local mozilla central>:<mount point in container> -e REVISION=<revision> -e REPOSITORY=<mount point in container used in -v> <image> <command>

then whatever <command> that's called that builds the environment would just do a 'hg clone <mount point> /home/builder/mozilla-central/source or something.


Maybe I'm off base with this...

Comment 3

[Wander Lairson Costa]

(In reply to Greg Arndt [:garndt] from comment #2)
> Perhaps something like this if I'm understanding correctly...
> 
> mach ci-docker-run --local-mozilla-central-repo=xxxx --revision=xxxx <image>
> <command>
> 
> If 'local-mozilla-central' isn't given, it will be cloned somewhere locally
> before running anything docker related.
> 

I think we don't need to clone locally in this case, just let builder clone the repos, as it does today.

Comment 4

[Greg Arndt [:garndt]]

I think what I was thinking when I wrote that was to mimic some of the caching behavior we have in docker-worker so the person running the mach command doesn't need to be aware of things happening behind the scenes to save time on subsequent runs of an image. Maybe we don't need to be concerned with cloning but more concerned with providing a cached volume to mount within the container like we do in docker-worker.  In taskcluster, a task can request to have a volume cache to use to store "stuff" [1] and any tasks after that wanting to use that cache can get the benefit of some of the work being already done.  Well this mach command can just set up a temp location for those types of things and provide a pseudo-cache.

I don't know, maybe that's all too much and the easiest thing now is to just explicitly declare the local repo to use and other wise continue with normal (current) behavior.

[1] https://github.com/lightsofapollo/gecko-dev/blob/taskcluster/testing/taskcluster/tasks/builds/b2g_desktop.yml#L20

Comment 5

[Robert Wood [:rwood]]

Attachment: https://github.com/lightsofapollo/gecko-dev/pull/9

If a local repo is specified, it is mounted in the docker container, and environment vars are set. For revision, if none is specified then the latest revision is used by default. Docker run flags are also supported. Some examples:

./mach ci-docker-run --local-repo /home/rwood/mozilla-central/ centos:centos7 echo hello world!

./mach ci-docker-run --local-repo /home/rwood/mozilla-central/ --revision=xxxxxxx --flags=ti centos:centos7 /bin/bash

Comment 6

[Wander Lairson Costa]

(In reply to Robert Wood [:rwood] from comment #5)
> Created attachment 8514544 [details] [review]
> https://github.com/lightsofapollo/gecko-dev/pull/9
> 
> If a local repo is specified, it is mounted in the docker container, and
> environment vars are set. For revision, if none is specified then the latest
> revision is used by default. Docker run flags are also supported. Some
> examples:
> 
> ./mach ci-docker-run --local-repo /home/rwood/mozilla-central/
> centos:centos7 echo hello world!
> 
> ./mach ci-docker-run --local-repo /home/rwood/mozilla-central/
> --revision=xxxxxxx --flags=ti centos:centos7 /bin/bash

I made some comments in the PR. I think :garndt and :lightsofapollo might have some comments too, better adding them as reviewers as well.

Comment 7

[James Lal [:lightsofapollo]]

Comment on attachment 8514544 [details] [review]
https://github.com/lightsofapollo/gecko-dev/pull/9

Going to un-r? myself mainly I want everyone else here to be happy using whatever we come up with as their primary tool to debug/hack on this.

Comment 8

[Robert Wood [:rwood]]

PR updated accordingly after conversation on IRC (add option for local gaia repo, and mount repos directly to /worker/ folders).

Comment 9

[Greg Arndt [:garndt]]

Comment on attachment 8514544 [details] [review]
https://github.com/lightsofapollo/gecko-dev/pull/9

Looks good.  Couple of final comments about mozconfig and a rogue print statement.

Comment 10

[Greg Arndt [:garndt]]

merged:
https://github.com/lightsofapollo/gecko-dev/commit/66824a19409fceb298ffeca75a3fabd9d7aa0d81

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/8256114225e7

Comment 12

[Pete Moore [:pmoore][:pete]]

Resetting Version and Target Milestone that accidentally got changed...