447666 - ffxbld/build network needs access to hg.mozilla.org

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: Account Requests, task)

Description

[[e-mail] (:bhearsum)]

While doing the Firefox 3.1a1 release I noticed two things:
1) The build network does not have ssh access to hg.mozilla.org. We need this for doing version bumps, and other release oriented things.
2) I can't test this, because of the above, but I'm pretty sure cltbld doesn't have an account that works with hg.m.o. We'll need this for future releases.

This isn't super urgent, but we definitely need it done before the next 3.1 release.

Comment 1

[Aravind Gottipati [:aravind]]

Would this be a good time to switch from the all en-compassing cltbld users to a ffxbld user (and the other app specific users).  Is using http://hg.m.o not good enough?

Do you need ssh access to the box?  I'd strongly suggest going the http route and checking stuff out as anonymous users.  If its really critical, I'd go so far as to put an ssl cert on hg.m.o if you need some kind of verification that you are indeed pulling from hg.m.o.

Comment 2

[Reed Loden [:reed]]

The automated release stuff needs to be able to push changes, so http-only access isn't good enough here.

Comment 3

[[e-mail] (:bhearsum)]

(In reply to comment #1)
> Would this be a good time to switch from the all en-compassing cltbld users to
> a ffxbld user (and the other app specific users).

Yeah, that's a good point. I'll update the summary. Eventually we'll need a tbirdbld key, maybe others, but I think MoMo will be dealing with that.

> Is using http://hg.m.o not
> good enough?
> 
> Do you need ssh access to the box?  I'd strongly suggest going the http route
> and checking stuff out as anonymous users.  If its really critical, I'd go so
> far as to put an ssl cert on hg.m.o if you need some kind of verification that
> you are indeed pulling from hg.m.o.
> 

It's not about verification. As reed mentions, our build automation does some check-ins. We bump versions, update configs files, etc.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Changing ffxbld from r/o to r/w for VCS access is something we should discuss a bit. One of the reasons for stopping using cltbld (bug 419651) is that it had been used for a long time and for every job under the sun. Moving away from it gives us a chance to lock down write-access to VCS systems, so we can enable access for just the machines that need it. That would require some new key (eg releng) that's used for releases (tagging, version bumps etc) and ancillary jobs like updating the blocklist (bug 426214). 

Machines that are just checking out would continue to use ffxbld, could pull from hg using http, and can push access to stage. Anything doing double-duty as a release builder would require both ffxbld and releng keys to be present, since tasks are allocated at random (not yet, but I'm assuming they will be). So we don't really gain much, except for machines like the unit testers using only ffxbld (bug 449901), and will have to make adjustments so we select the right key when doing releases.

Ben/Coop/John/Lukas - what's your take on the good security/PITA balance for a new key ?

Comment 5

[[e-mail] (:bhearsum)]

I agree with most everything you said, Nick. I think we use the ffxbld/tbirdbld/whateverbld keys for dep/nightlies. We should give this key to Firefox unittest machines as well. (Does Talos need an ssh key?). I'm torn on whether we should have a 'release' key. I suppose it would be good to completely restrict dep/nightly builds from being able to check-in to Mercurial so I think I support that. I also feel that we should have a separate 'services' key - or something like that. Syncing configure, blocklist, etc, are distinctly different tasks than dep/nightly/release.

I totally agree that we should be pulling sources via http://.

To summarize:
* ffxbld for Firefox builds/tests - read-only CVS/HG access, ssh access to stage.
* "releasebld" for releases - read/write CVS/HG access
* "servicebld" for configure/blocklist syncing, other misc. jobs - read/write CVS/HG access.

I'm not totally convinced we need both of the latter two, but if we use one key for both I'd like to think up a better name. It's non-obvious to me that a key named "releasebld" would be used for non-release activities like updating configure/blocklist.

Comment 6

[Lukas Blakk [:lsblakk] use ?needinfo]

I'm in favour of using the ffxbld key (also stgbld) - see bug 449901

In the next week or two both the unittest production masters are being moved over to the build network, and they will all use the new keys.

Comment 7

[Aravind Gottipati [:aravind]]

So.. what do you need from us at this point?

http(s)://hg.m.o should be open from the build network.  So you should be able to pull from there as needed.

If you want me to grant releasebld and servicebld accounts r/w access to hg.m.o, I need ssh keys for them.

Comment 8

[[e-mail] (:bhearsum)]

We haven't figured out what we're doing here yet. For regular dep/nightly we don't need a key, so we don't need to figure this out until we have an automated release. We'll take this bug back for now and pass it back once we know what we're doing

Comment 9

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #4)
> Changing ffxbld from r/o to r/w for VCS access is something we should discuss a
> bit. One of the reasons for stopping using cltbld (bug 419651) is that it had
> been used for a long time and for every job under the sun. Moving away from it
> gives us a chance to lock down write-access to VCS systems, so we can enable
> access for just the machines that need it. That would require some new key (eg
> releng) that's used for releases (tagging, version bumps etc) and ancillary
> jobs like updating the blocklist (bug 426214). 
> 
> Machines that are just checking out would continue to use ffxbld, could pull
> from hg using http, and can push access to stage. Anything doing double-duty as
> a release builder would require both ffxbld and releng keys to be present,
> since tasks are allocated at random (not yet, but I'm assuming they will be).
> So we don't really gain much, except for machines like the unit testers using
> only ffxbld (bug 449901), and will have to make adjustments so we select the
> right key when doing releases.
> 
> Ben/Coop/John/Lukas - what's your take on the good security/PITA balance for a
> new key ?

I think that the "use new releng" key approach will hit the same problem over time... namely that it would be used for a long time, and use for every release. Unless we were having a dedicating "checkout-and-tag" slave, we'd need to put the new releng key on all the slaves also. Adds to be complexity, and doesnt feel like its really solving the problem, imho.

Personally, I'd be in favour of either:
- just using ffxbld, with r/w permissions. In a couple of years, talk about ffxbld like we did about cltbld.
...or...
- leaving ffxbld as r/o, and using the LDAP-based-human-id for doing r/w, tagging, etc. So, whomever is doing the actual release will use their id. This will always keep us current, and feels like the right way to me.

Thats my $0.02... what do others think?

Comment 10

[[e-mail] (:bhearsum)]

(In reply to comment #9)
> - leaving ffxbld as r/o, and using the LDAP-based-human-id for doing r/w,
> tagging, etc. So, whomever is doing the actual release will use their id. This
> will always keep us current, and feels like the right way to me.
> 

I don't think this is viable. It would require us to either a) run automation that does automated checkins on our laptops. Or b) Put our private keys on build machines.

Unless I'm missing something.

(I'll respond to the rest later.)

Comment 11

[[e-mail] (:bhearsum)]

I'm getting ready to set-up a staging release automation environment and would like to get *something* in place here soon. From re-reading this bug I see that both nthomas and I are in favour of creating a 'releasebld' key for things that require r/w access to CVS/HG. John, can we get your agreement here? I really don't feel like purposely putting 'ffxbld' down the cltbld path is the right way to go here. It's a very small task to put the new key on all of the slaves - no harder than it is to put the existing 'ffxbld' key on them.

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Maybe I'm missing something.

We are moving all build and unittest machines into the same shared pool. So any of those machines could be used for doing:
- an incremental (hourly) build
- a full (nightly) build
- a release build
- running unittests. 

What key are you proposing be put on those machines? ffxbld with read-only access? releasebld with read-write access?

(Note: The talos machines are separate and not part of this shared pool. Unclear if they need any key at all, or if they only use anonymous access to cvs. )

Comment 13

[[e-mail] (:bhearsum)]

(In reply to comment #12)
> What key are you proposing be put on those machines? ffxbld with read-only
> access? releasebld with read-write access?
> 

I was proposing ffxbld read-only to stage, aus2-staging, CVS, etc. *and* releasebld w/ read-write to hg/cvs.

> (Note: The talos machines are separate and not part of this shared pool.
> Unclear if they need any key at all, or if they only use anonymous access to
> cvs. )

I checked, and they use anonymous@cvs-mirror.m.o


So, if we're working on the assumption that all of our machines that do *builds* will be in a single pool I am OK with simply giving ffxbld read-write to cvs/hg, unless there's a hidden benefit to a separate key for that.

Nick, what do you think?

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

I think there is some benefit to setting up ffxbld as the default key on the boxes, r/o as it is now, and explicitly using a r/w releasebld key for only the operations that need to do VCS writes (eg hg <operation> -e 'ssh -i ~/.ssh/releasebld.dsa' ...). That'd prevent accidental pushes at least, by users and scripts unaware of the write key.

I don't particularly like repeating the all-our-eggs-in-one-basket aspect of cltbld, but Hg at least gives us much better tools to revert damage, and we need to not block bhearsum here. Lets go ahead and grant ffxbld access to hg.m.o.

Comment 15

[[e-mail] (:bhearsum)]

Aravind,

Can you go ahead and create an ffxbld account on hg.mozilla.org? Can you enable build network -> hg.mozilla.org:22 access, too?

Comment 16

[Reed Loden [:reed]]

(In reply to comment #15)
> Can you go ahead and create an ffxbld account on hg.mozilla.org? Can you enable
> build network -> hg.mozilla.org:22 access, too?

ffxbld or releasebld?

Comment 17

[[e-mail] (:bhearsum)]

(In reply to comment #14)
> Lets go ahead and grant ffxbld access to hg.m.o.

Comment 18

[[e-mail] (:bhearsum)]

(In reply to comment #15)
> Aravind,
> 
> Can you go ahead and create an ffxbld account on hg.mozilla.org? Can you enable
> build network -> hg.mozilla.org:22 access, too?

To be clear, we would like ffxbld to have r/w access to hg.m.o.

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #18)
> To be clear, we would like ffxbld to have r/w access to hg.m.o.

... RSN.

Comment 20

[Aravind Gottipati [:aravind]]

Done.