585615 - http://stage.mozilla.org/pub/mozilla.org/firefox/pvt-builds/ is publicly viewable

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Lukas Blakk [:lsblakk] use ?needinfo]

The location of the pvt-builds upload dir is viewable by anyone and thus, the builds are downloadable outside of the ftp auth site that was intended for use.

Can this directory (http://stage.mozilla.org/pub/mozilla.org/firefox/pvt-builds/) be accessible only to machines in the build network so that the automation can continue to upload and download to it but the http addresses can't be viewed or trolled for?

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Ideally we should enumerate the machines that serve .../pub/mozilla.org/firefox and check them all. Looks like
  http://dm-download02.mozilla.org/pub/mozilla.org/firefox/
is OK via permissions, and dm-download03/04 only have firefox/releases/ (manual rsync IIRC).

Comment 2

[Marcia Knous [:marcia]]

Lukas: I think this should probably be a security sensitive websites bug since it showing up in our security triage, which deals with browser vulnerabilities.

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to comment #2)
> Lukas: I think this should probably be a security sensitive websites bug since
> it showing up in our security triage, which deals with browser vulnerabilities.

Marcia, I only have permission to mark it in Security-Sensitive Core Bug like its tracking bug 551756

Comment 4

[Reed Loden [:reed]]

(In reply to comment #3)
> Marcia, I only have permission to mark it in Security-Sensitive Core Bug like
> its tracking bug 551756

Just add "[sg:nse]" to the whiteboard, and it'll disappear from the triage queue.

Comment 5

[Dave Miller [:justdave]]

Umm...   whose idea was it to put this in the main ftp tree?  This directory structure gets shared out to our mirrors if they ask nicely for it.

Comment 6

[Dave Miller [:justdave]]

IIRC this type of stuff is exactly what the releases.mozilla.com site (note the .com not .org) was intended for (which is also staged on stage.mozilla.org, and only certain accounts have access to it).  It should be going there instead.

Comment 8

[Lukas Blakk [:lsblakk] use ?needinfo]

Adding what dveditz said in bug 587194:

bug 553728 created a private, password-protected, ftp location to upload
shadow-central builds:

https://ftp.mozilla.org/pvt-builds/firefox/shadow-central-builds/

But these also show up in public at:

http://ftp.mozilla.org/pub/mozilla.org/firefox/pvt-builds/

(Actually I'm guessing about the "also" -- my LDAP credentials no longer work
for the non-pub location so it's possible they ONLY show up in public).

Comment 9

[Daniel Veditz [:dveditz]]

I don't care where it goes, if releases.mozilla.com is better that's fine by me if the build automation can deal with it.

My only requirements are that it be password protected and only available over SSL (http: requests should redirect to https:).

Comment 10

[Mike Shaver (:shaver emeritus)]

http requests should give 404, so that people fix them to be https, instead of leaving them as a place to hook an attack.

Comment 11

[Shyam Mani [:fox2mike]]

Re-assigning to Aravind, to make sure this is done right. He was involved in the initial setup, I guess.

Comment 12

[Aravind Gottipati [:aravind]]

Okay, the only urls that will work are those that hit https://ftp.mozilla.org/pvt-builds/

The rest of them will give you a 403 forbidden.  Please re-open if you manage to sneak in using some other route (over http or https without auth).

Comment 13

[Lukas Blakk [:lsblakk] use ?needinfo]

Can we allow for 

test-master01.build.mozilla.org
test-master02.build.mozilla.org
talos-master02.build.mozilla.org

To access the stage urls (http://stage.mozilla.org/pub/mozilla.org/firefox/pvt-builds/shadow-central-builds)?  The 403 has now broken the ability to run tests and talos on these builds since we need to download them to our test masters to unpack and run tests.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Uh, don't you mean the slaves need to get the builds rather than the masters ? I don't think there's any problem with slaves using ftp.m.o, it's just a historical leftover that we pull from stage for other branches.

Comment 15

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to comment #14)
> Uh, don't you mean the slaves need to get the builds rather than the masters ?
> I don't think there's any problem with slaves using ftp.m.o, it's just a
> historical leftover that we pull from stage for other branches.

Yes, you're right - it's the slaves.  Sorry for the mistake.  This dir is not mirrored to ftp though and so the slaves do in fact need to pull from stage to get the builds which is what they were doing until it was discovered that anyone could see the stage dir of these builds.

So in fact, can we open up access to build network slaves to be able to pull from http://stage.mozilla.org/pub/mozilla.org/firefox/pvt-builds/shadow-central-builds ?

Comment 16

[Aravind Gottipati [:aravind]]

Dan: Comments?  okay to open up to the entire build network.

Comment 17

[Daniel Veditz [:dveditz]]

Any other alternative to opening up "the entire" build network? How is the build network defined? Do we know how many machines that opens it up to? Is it all in one location or subnet?

If the directory you need is "not mirrored to ftp" then what am I getting at https://ftp.mozilla.org/pvt-builds/ ? Is the problem that you can't put a password in there, or that hardcoding the password into the scripts is worse than poking a hole in the protection to let the build network through?

Comment 18

[Lukas Blakk [:lsblakk] use ?needinfo]

Aravind - it would be great if you could give Dan info on what "entire build network" means.  Afaik it means only slaves (and masters) that Release Engineering uses for our build and test infra and therefore comprises exactly the machines we want to be able to access the shadow-central dir with. I don't know the answer about location or subnet.

Dan - when I said "not mirrored" I meant that because of the https://ftp the slaves cannot download without using authentication, which is not part of how the automation works.  

We've run up against this in other parts of trying to set up this branch.  I want to be clear that this part of running the builds was working fine until it was discovered that stage.m.o urls were publicly visible.  The solution as I see it is to allow build network machines to access stage.m.o directly in order to download the packages needed for tests without making those urls visible in the browser so that the only way to browse to shadow-central results it to use the https://ftp.m.o/pvt-builds area.

Comment 19

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Aravind:

The original purpose of this bug (in comment#0) was to have this directory on stage.m.o visible to build machines, yet not visible to the outside world. 

aiui, comment#15-#18 are further discussion, also asking to have all *.build.m.o machines (which are the slaves used for building, and testing) be able to access that location.

What other info do you need to proceed granting access to these *.build.m.o machines?



(dan, chime in if I'm missing something).

Comment 20

[Aravind Gottipati [:aravind]]

Okay, I think I should be able to do this.  My only concern is that we might be exposing security sensitive information, should the build network be compromised.  However, it's probably not that big a deal to leak the build binaries themselves..

@Dan:  All of the build machines are just in one giant network.

@Lukas: you are right that this was open in the past, before we locked it down.

Like I said, I think I can reason (with myself) that opening it up to the build network without authentication is probably okay.

I'd like Dan to confirm that this is okay, and I will go ahead and set this up.

@Lukas:  What hostname/url do you use to fetch these builds?

Comment 21

[Lukas Blakk [:lsblakk] use ?needinfo]

We try to fetch the builds using ['stage_base_path'] = '/home/ftp/pub/firefox/pvt-builds' which looks like this for a particular build:

http://stage.mozilla.org/pub/mozilla.org/firefox/pvt-builds/shadow-central-builds/shadow-central-linux64/1283893350/firefox-3.7a5pre.en-US.linux-x86_64.tar.bz2

Comment 22

[Daniel Veditz [:dveditz]]

> However, it's probably not that big a deal to leak the build
> binaries themselves..

Eventually that will be equivalent to leaking the patches which is why we're trying to hide it. If the build machines are compromised we've got bigger problems so I guess if you're confident that non-auth/non-ssl access is limited to the build network we can worry about hypothetical leaks later.

Comment 23

[Aravind Gottipati [:aravind]]

(In reply to comment #22)
> Eventually that will be equivalent to leaking the patches which is why we're
> trying to hide it. If the build machines are compromised we've got bigger
> problems so I guess if you're confident that non-auth/non-ssl access is limited
> to the build network we can worry about hypothetical leaks later.

I am more worried about unauthorized access into the network either through a compromised build machine or through a compromised gateway machine.

However, I won't drag this out further.  Will work on opening it up to the build network.  If folks feel like its a risk, let me know and I will remove that access.

Comment 24

[Aravind Gottipati [:aravind]]

I started digging into this and short of redoing the backend or writing some awfull(er) re-write rules or some such this is not possible - at least on stage.mozilla.org.  Would it be possible for you to switch the url you pull from to ftp.mozilla.org instead of stage.mozilla.org.  stage proxies back to ftp anyway for the firefox builds..

And while you are at it, switch it to https://ftp.mozilla.org (from http).

Comment 25

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to comment #24)
> Would it be possible for you to switch the url you pull
> from to ftp.mozilla.org instead of stage.mozilla.org. 
 
> And while you are at it, switch it to https://ftp.mozilla.org (from http).

no problem, we can point the configs to https://ftp.mozilla.org/pub/firefox/pvt-builds if you give the build machines access to that location

Comment 26

[Aravind Gottipati [:aravind]]

(In reply to comment #25)
> no problem, we can point the configs to
> https://ftp.mozilla.org/pub/firefox/pvt-builds if you give the build machines
> access to that location

Great, I will let you know once thats ready.. and btw, that path is wide open to the net right now.  Sigh.. one more path to close...

Comment 27

[Aravind Gottipati [:aravind]]

All leaks plugged hopefully and opened to the build network(s).

Comment 28

[Lukas Blakk [:lsblakk] use ?needinfo]

Using https://ftp.m.o our build machines are hitting 401 authorization errors.

Comment 29

[Mike Shaver (:shaver emeritus)]

Good!

Is there a way to do this without putting them on stage at all?  Seems like they shouldn't have to leave the build/test network at all for now, as I understand things, and putting them on stage means we're just one configuration error away from undoing all the hard work.

Comment 30

[Aravind Gottipati [:aravind]]

(In reply to comment #28)
> Using https://ftp.m.o our build machines are hitting 401 authorization errors.
Can I get more information about this?  Which machine is having problems accessing this?  Are all the machines failing?

@Shaver:  We need a place to upload these builds, we need to allow (a select group of) community folks to access these builds, we need all the builders to access these builds.  No matter where we put them, we will still be one configuration error away from failure.

Comment 31

[Lukas Blakk [:lsblakk] use ?needinfo]

They all fail (not one slave) with output like the following:

--16:23:37--  http://stage.mozilla.org/pub/mozilla.org/firefox/pvt-builds/shadow-central-builds/shadow-central-macosx/1285018366/firefox-3.7a5pre.en-US.mac.dmg
           => `firefox-3.7a5pre.en-US.mac.dmg'
Resolving stage.mozilla.org... 10.2.74.116
Connecting to stage.mozilla.org|10.2.74.116|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
16:23:37 ERROR 403: Forbidden.

Comment 32

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to comment #30)
> (In reply to comment #28)
> We need a place to upload these builds, we need to allow (a select
> group of) community folks to access these builds, we need all the builders to
> access these builds.  No matter where we put them, we will still be one
> configuration error away from failure.

Unless you don't want test coverage... if you only want builds, those are almost ready to go (waiting on win32 ssh batch mode landing in next downtime for win32 to build)

Comment 33

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to comment #31)
> They all fail (not one slave) with output like the following:
> 
> --16:23:37-- 
> http://stage.mozilla.org/pub/mozilla.org/firefox/pvt-builds/shadow-central-builds/shadow-central-macosx/1285018366/firefox-3.7a5pre.en-US.mac.dmg
>            => `firefox-3.7a5pre.en-US.mac.dmg'
> Resolving stage.mozilla.org... 10.2.74.116
> Connecting to stage.mozilla.org|10.2.74.116|:80... connected.
> HTTP request sent, awaiting response... 403 Forbidden
> 16:23:37 ERROR 403: Forbidden.

I realize this is the wrong URL, let me grab the right log output.

Comment 34

[Lukas Blakk [:lsblakk] use ?needinfo]

--2010-09-21 10:00:53--  https://ftp.mozilla.org/pub/mozilla.org/firefox/pvt-builds/shadow-central-builds/shadow-central-linux64/1285084335/firefox-3.7a5pre.en-US.linux-x86_64.tar.bz2
Resolving ftp.mozilla.org... 10.2.74.10
Connecting to ftp.mozilla.org|10.2.74.10|:443... connected.
HTTP request sent, awaiting response... 401 Authorization Required
Authorization failed.
program finished with exit code 1
elapsedTime=0.104128

Comment 35

[Mike Shaver (:shaver emeritus)]

(Do you not see the auth errors in the HTTP error log, to tell you which machines are getting them and what paths they're trying to access?)

Comment 36

[Lukas Blakk [:lsblakk] use ?needinfo]

Just chatting with Shaver and the new idea here is to have a completely different base name for the storage location of the shadow central builds, keeping them away from the public ftp altogether.  Our build machines can use that base name in up/downloading of builds and test suites and access to it (as well as backup regimen, cleanup scripts, etc.) can be independent from the public ftp configs.

Comment 37

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Shaver & I met late last week about this. Even if we get the privs correct here for pvt-build, the concern is around risk of accidentally exposing this share while doing other maintenance changes on the same stage. Hence a change of direction to put these pvt-builds onto a separate machine. This separate machines would have fewer changes to its privs, and would be treated with caution, because of its security sensitive nature. 

Mechanically that would mean no longer doing http://stage.mozilla.org/pub/mozilla.org/firefox/pvt-builds/ , and instead setting up http://pvt-stage.mozilla.org which would be used only for storing builds from shadow-central.


(shaver, chime in if I missed anything)

Comment 38

[Mike Shaver (:shaver emeritus)]

That is entirely in accordance with my recollection.

Comment 39

[Lukas Blakk [:lsblakk] use ?needinfo]

Aravind - eta on having a location to upload/download shadow-central builds?

Comment 40

[Aravind Gottipati [:aravind]]

Working on it, should have it within a week.

Comment 41

[Aravind Gottipati [:aravind]]

Okay, a new server to upload these builds is ready.  I am not sure how paranoid folks want to be with these builds, so I have a few questions.

1) Should shells be allowed?  (I can restrict stuff to sftp only).
2) Should I open up access to folks/buildbots/testbots etc via https, or should I limit it to scp/sftp only?

Also, I am not mounting this volume anywhere else, except on this one box (dm-pvtbuilds01).  From the last few comments, that was my assumption, but please correct me if I am wrong.

Comment 42

[Daniel Veditz [:dveditz]]

does RelEng need shells to do their thing? I don't. I just care that we can share builds with with trusted community members, that they can get to without having to be on our network.

https: seems easiest to deal with for that scenario. I don't care if it's LDAP controlled or http auth to a generic user:pass that we update periodically, but if it's LDAP I'd like a unique group for it so we're not granting extra privs just to share test builds.

Comment 43

[Aravind Gottipati [:aravind]]

(In reply to comment #42)
> does RelEng need shells to do their thing? I don't. I just care that we can
> share builds with with trusted community members, that they can get to without
> having to be on our network.

I need releng folks to answer that.

> https: seems easiest to deal with for that scenario. I don't care if it's LDAP
> controlled or http auth to a generic user:pass that we update periodically, but
> if it's LDAP I'd like a unique group for it so we're not granting extra privs
> just to share test builds.

I plan on using a brand new group in ldap for this.

Comment 44

[Lukas Blakk [:lsblakk] use ?needinfo]

I don't think we need shells for this location, but the machines in the build network do need to be able to run 'make upload' which uses scp to place the completed builds on the storage mount location, then we use http: to download them to the test slaves.

Comment 45

[Chris AtLee [:catlee]]

Current behaviour relies on being able to run something like 'ssh remotehost post_upload.py /tmp/tmpdir'

Comment 46

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to comment #41)
> Okay, a new server to upload these builds is ready.  

can I get a path to this location? it would be great to start testing it out.

Comment 47

[Aravind Gottipati [:aravind]]

@Derek: I need an external ip linking back to dm-pvtbuild01 with ports 22 and 443 open.

Comment 48

[Derek Moore [:dmoore]]

aravind: created at 63.245.208.160

Comment 49

[Aravind Gottipati [:aravind]]

dm-pvtbuild01.mozilla.org is now ready.  It lets you browse stuff via https (with ldap auth), and allows ssh logins (ldap auth).  There is a separate group in ldap called cn=pvtbuilds,ou=groups,dc=mozilla that controls access to the box.

I am not sure if we decided to use a separate builder account (or key) to upload stuff to this box.  I granted dveditz, lukas, jesse and catlee access to this box.  Also anyone in the build network should be able to browse stuff (over https) without a password.

All of the pvt builds should be uploaded to /mnt/pvt_builds.

Please let me know if you want me to change anything here.

Comment 50

[Lukas Blakk [:lsblakk] use ?needinfo]

please add ffxbld to the group, and also I cannot connect (as lsblakk) to dm-pvtbuild01.m.o when connected to the build-vpn

Comment 51

[Aravind Gottipati [:aravind]]

I added ffxbld to the group.  It looks like you were able to login from an external network.  What happens when you try to login from the build machines?

Is the connection blocked?

Comment 52

[Chris AtLee [:catlee]]

(In reply to comment #51)
> I added ffxbld to the group.  It looks like you were able to login from an
> external network.  What happens when you try to login from the build machines?
> 
> Is the connection blocked?

I can't hit it from moz2-linux-slave23 (looks like firewall issue?), but it works from linux-ix-slave30.

Comment 53

[Aravind Gottipati [:aravind]]

Derek: Network changes..?

Comment 54

[Lukas Blakk [:lsblakk] use ?needinfo]

Aravind can you please add stage-ffxbld to the group as well so that we can do staging runs?

Comment 55

[Derek Moore [:dmoore]]

Access has been granted from the build network to the internal address for dm-pvtbuild01 on ports 22 and 443.

Comment 56

[Lukas Blakk [:lsblakk] use ?needinfo]

great, I can confirm that connecting from moz2-linux-slave23 works now -- just need stage-ffxbld access:

(from moz2-linux-slave04)
ssh [e-mail]
You must be a member of cn=pvtbuilds,ou=groups,dc=mozilla to login.
Connection closed by 10.2.74.95

Comment 57

[Aravind Gottipati [:aravind]]

added stage-ffxbld to the group.  Please give it 10 minutes to propagate.

Comment 58

[Lukas Blakk [:lsblakk] use ?needinfo]

quick request, in the interest of keeping things in the right place can you please rename /mnt/pvt_builds/firefox to be /mnt/pvt_builds/shadow-central this will be useful in how we manage the uploading and placing of builds.  thank you.

Comment 59

[Aravind Gottipati [:aravind]]

Done.