1232314 - [Taskcluster] Add new script to permits community phone builds

Status: RESOLVED WONTFIX

(Taskcluster :: Services, defect)

Description

[Ronald Claveau [:sousmangoosta]]

We need to add some files on gecko to easily permit taskcluster usage by mozilla's community.

Comment 1

[Ronald Claveau [:sousmangoosta]]

Attachment: 1dc7529f30ad.patch

Treeherder job : https://treeherder.mozilla.org/#/jobs?repo=try&revision=f9f82cca91e8

Comment 2

[Wander Lairson Costa]

Comment on attachment 8698040 [details] [diff] [review]
1dc7529f30ad.patch

Review of attachment 8698040 [details] [diff] [review]:
-----------------------------------------------------------------

There are some concerns about accessing an external repositories and making blobs public available.

::: testing/taskcluster/scripts/community-phone-builder/pre-build.sh
@@ +18,5 @@
> +
> +. ../builder/setup-ccache.sh
> +
> +# Figure out where the remote manifest.
> +B2G_CONFIG="https://github.com/b2g-community-builds/b2g-config.git"

We need to check if we are ok with repositories out of git.m.o. I am not sure who to ask.

@@ +19,5 @@
> +. ../builder/setup-ccache.sh
> +
> +# Figure out where the remote manifest.
> +B2G_CONFIG="https://github.com/b2g-community-builds/b2g-config.git"
> +BLOBS="http://b2g-community-builds.github.io/blobs-zips/$TARGET.tar.bz2"

What are these blobs? If they are vendor blobs, I am afraid we may have legal issues here. I didn't participate on community builds discussions, so I might be missing something here.

@@ +25,5 @@
> +git clone $B2G_CONFIG $WORKSPACE/b2g-config
> +cp $WORKSPACE/b2g-config/$TARGET/sources.xml $WORKSPACE/sources.xml
> +
> +git config --global --add color.ui auto
> +git clone https://git.mozilla.org/b2g/B2G.git $WORKSPACE/B2G

Depending on the number of builds we do per minute, we might cause a DoS attack on git.m.o, better using tc-vcs for closing repos.

@@ +35,5 @@
> +git commit -m manifest
> +git branch -m master
> +cd $WORKSPACE/B2G
> +./repo init -b master -u $WORKSPACE/B2G/.tc-vcs-manifest -m sources.xml --repo-url https://git.mozilla.org/external/google/gerrit/git-repo.git --repo-branch master
> +./repo sync -j100 -q

I am not sure what this tc-vcs trick is for, but tc-vcs has a "repo-checkout" command iirc.

Comment 3

[Ronald Claveau [:sousmangoosta]]

(In reply to Wander Lairson Costa [:wcosta] from comment #2)
> Comment on attachment 8698040 [details] [diff] [review]
> 1dc7529f30ad.patch
> 
> Review of attachment 8698040 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> There are some concerns about accessing an external repositories and making
> blobs public available.
> 
> ::: testing/taskcluster/scripts/community-phone-builder/pre-build.sh
> @@ +18,5 @@
> > +
> > +. ../builder/setup-ccache.sh
> > +
> > +# Figure out where the remote manifest.
> > +B2G_CONFIG="https://github.com/b2g-community-builds/b2g-config.git"
> 
> We need to check if we are ok with repositories out of git.m.o. I am not
> sure who to ask.

No pb to use git.m.o, but need more flexibility on community's file push.

> 
> @@ +19,5 @@
> > +. ../builder/setup-ccache.sh
> > +
> > +# Figure out where the remote manifest.
> > +B2G_CONFIG="https://github.com/b2g-community-builds/b2g-config.git"
> > +BLOBS="http://b2g-community-builds.github.io/blobs-zips/$TARGET.tar.bz2"
> 
> What are these blobs? If they are vendor blobs, I am afraid we may have
> legal issues here. I didn't participate on community builds discussions, so
> I might be missing something here.

Yes there are, when you start building, for blobfree or not, you need those blobs.
They will not be publicly available on moz platforms, only private on TC.

> 
> @@ +25,5 @@
> > +git clone $B2G_CONFIG $WORKSPACE/b2g-config
> > +cp $WORKSPACE/b2g-config/$TARGET/sources.xml $WORKSPACE/sources.xml
> > +
> > +git config --global --add color.ui auto
> > +git clone https://git.mozilla.org/b2g/B2G.git $WORKSPACE/B2G
> 
> Depending on the number of builds we do per minute, we might cause a DoS
> attack on git.m.o, better using tc-vcs for closing repos.
>

I'm ok with that, but needs to launch tc-vcs builds before, and community can't have rights to do that. For hacking new devices, it should not be usable.

> 
> @@ +35,5 @@
> > +git commit -m manifest
> > +git branch -m master
> > +cd $WORKSPACE/B2G
> > +./repo init -b master -u $WORKSPACE/B2G/.tc-vcs-manifest -m sources.xml --repo-url https://git.mozilla.org/external/google/gerrit/git-repo.git --repo-branch master
> > +./repo sync -j100 -q
> 
> I am not sure what this tc-vcs trick is for, but tc-vcs has a
> "repo-checkout" command iirc.

It's only same name of command used by official phone-builder script, not really tc.

Comment 4

[Ronald Claveau [:sousmangoosta]]

Elvin, Peter,

Can we have your point of view about those security concerns.

Comment 5

[Wander Lairson Costa]

(In reply to Ronald Claveau [:sousmangoosta] from comment #3)
> (In reply to Wander Lairson Costa [:wcosta] from comment #2)
> > Comment on attachment 8698040 [details] [diff] [review]
> > 1dc7529f30ad.patch
> > 
> > Review of attachment 8698040 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > There are some concerns about accessing an external repositories and making
> > blobs public available.
> > 
> > ::: testing/taskcluster/scripts/community-phone-builder/pre-build.sh
> > @@ +18,5 @@
> > > +
> > > +. ../builder/setup-ccache.sh
> > > +
> > > +# Figure out where the remote manifest.
> > > +B2G_CONFIG="https://github.com/b2g-community-builds/b2g-config.git"
> > 
> > We need to check if we are ok with repositories out of git.m.o. I am not
> > sure who to ask.
> 
> No pb to use git.m.o, but need more flexibility on community's file push.
> 

I am not sure who you should talk to, someone on #releng might help.

> > 
> > @@ +19,5 @@
> > > +. ../builder/setup-ccache.sh
> > > +
> > > +# Figure out where the remote manifest.
> > > +B2G_CONFIG="https://github.com/b2g-community-builds/b2g-config.git"
> > > +BLOBS="http://b2g-community-builds.github.io/blobs-zips/$TARGET.tar.bz2"
> > 
> > What are these blobs? If they are vendor blobs, I am afraid we may have
> > legal issues here. I didn't participate on community builds discussions, so
> > I might be missing something here.
> 
> Yes there are, when you start building, for blobfree or not, you need those
> blobs.
> They will not be publicly available on moz platforms, only private on TC.
> 

Ok, if blob images will not be public, I think that's fine, but better hearing that from someone from legal team. Maybe :mahe can help you on that.

> > 
> > @@ +25,5 @@
> > > +git clone $B2G_CONFIG $WORKSPACE/b2g-config
> > > +cp $WORKSPACE/b2g-config/$TARGET/sources.xml $WORKSPACE/sources.xml
> > > +
> > > +git config --global --add color.ui auto
> > > +git clone https://git.mozilla.org/b2g/B2G.git $WORKSPACE/B2G
> > 
> > Depending on the number of builds we do per minute, we might cause a DoS
> > attack on git.m.o, better using tc-vcs for closing repos.
> >
> 
> I'm ok with that, but needs to launch tc-vcs builds before, and community
> can't have rights to do that. For hacking new devices, it should not be
> usable.
> 

You can talk to :selenamarie or :garndt about tc-vcs stuff you need.

> > 
> > @@ +35,5 @@
> > > +git commit -m manifest
> > > +git branch -m master
> > > +cd $WORKSPACE/B2G
> > > +./repo init -b master -u $WORKSPACE/B2G/.tc-vcs-manifest -m sources.xml --repo-url https://git.mozilla.org/external/google/gerrit/git-repo.git --repo-branch master
> > > +./repo sync -j100 -q
> > 
> > I am not sure what this tc-vcs trick is for, but tc-vcs has a
> > "repo-checkout" command iirc.
> 
> It's only same name of command used by official phone-builder script, not
> really tc.

Ok, fine.

Comment 6

[Ronald Claveau [:sousmangoosta]]

(In reply to Wander Lairson Costa [:wcosta] from comment #5)
> (In reply to Ronald Claveau [:sousmangoosta] from comment #3)
> > (In reply to Wander Lairson Costa [:wcosta] from comment #2)
> > > Comment on attachment 8698040 [details] [diff] [review]
> > > 1dc7529f30ad.patch
> > > 
> > > Review of attachment 8698040 [details] [diff] [review]:
> > > -----------------------------------------------------------------
> > > 
> > > There are some concerns about accessing an external repositories and making
> > > blobs public available.
> > > 
> > > ::: testing/taskcluster/scripts/community-phone-builder/pre-build.sh
> > > @@ +18,5 @@
> > > > +
> > > > +. ../builder/setup-ccache.sh
> > > > +
> > > > +# Figure out where the remote manifest.
> > > > +B2G_CONFIG="https://github.com/b2g-community-builds/b2g-config.git"
> > > 
> > > We need to check if we are ok with repositories out of git.m.o. I am not
> > > sure who to ask.
> > 
> > No pb to use git.m.o, but need more flexibility on community's file push.
> > 
> 
> I am not sure who you should talk to, someone on #releng might help.
>

Ok I'll try to have something there.
 
> > > 
> > > @@ +19,5 @@
> > > > +. ../builder/setup-ccache.sh
> > > > +
> > > > +# Figure out where the remote manifest.
> > > > +B2G_CONFIG="https://github.com/b2g-community-builds/b2g-config.git"
> > > > +BLOBS="http://b2g-community-builds.github.io/blobs-zips/$TARGET.tar.bz2"
> > > 
> > > What are these blobs? If they are vendor blobs, I am afraid we may have
> > > legal issues here. I didn't participate on community builds discussions, so
> > > I might be missing something here.
> > 
> > Yes there are, when you start building, for blobfree or not, you need those
> > blobs.
> > They will not be publicly available on moz platforms, only private on TC.
> > 
> 
> Ok, if blob images will not be public, I think that's fine, but better
> hearing that from someone from legal team. Maybe :mahe can help you on that.
> 

Elvin, Mahendra,
Need you on this legal possible issue.

Wander,
Can't we use a worker in Francfort (no software patents) to avoid this point ?

> > > 
> > > @@ +25,5 @@
> > > > +git clone $B2G_CONFIG $WORKSPACE/b2g-config
> > > > +cp $WORKSPACE/b2g-config/$TARGET/sources.xml $WORKSPACE/sources.xml
> > > > +
> > > > +git config --global --add color.ui auto
> > > > +git clone https://git.mozilla.org/b2g/B2G.git $WORKSPACE/B2G
> > > 
> > > Depending on the number of builds we do per minute, we might cause a DoS
> > > attack on git.m.o, better using tc-vcs for closing repos.
> > >
> > 
> > I'm ok with that, but needs to launch tc-vcs builds before, and community
> > can't have rights to do that. For hacking new devices, it should not be
> > usable.
> > 
> 
> You can talk to :selenamarie or :garndt about tc-vcs stuff you need.

I think you're right on this, do you have a way to get the last TC task id ?

Comment 7

[Elvin Lee [:ellee] (no longer with Mozilla)]

Hi Ron, let's set up a time to chat briefly to make sure I understand what is being proposed/asked. Peter, I'll loop you in as well.

Comment 8

[Mahendra Potharaju [:mahe]]

Elvin, Please add me to the meeting, will be quite helpful. Thanks

Comment 9

[Peter Dolanjski [:pdol]]

(In reply to Elvin Lee [:ellee] from comment #7)
> Hi Ron, let's set up a time to chat briefly to make sure I understand what
> is being proposed/asked. Peter, I'll loop you in as well.

Elvin, can we set this meeting up?

Comment 10

[Pete Moore [:pmoore][:pete]]

Moving to TaskCluster component...