Closed Bug 450648 Opened 16 years ago Closed 16 years ago

Move all webapps away from hg.mozilla.org

Categories

(mozilla.org Graveyard :: Server Operations, task)

task
Not set
major

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: BenB, Assigned: aravind)

References

Details

hgwebdir is running on hg.mozilla.org. That's a bad idea - the more servers and webapps are running on a server, the more likely break-ins to a server are. Each server app adds to the risk.

hg.mozilla.org is the most valuable and highest-risk resource of the whole Mozilla project. If somebody manages to break in there, he can subtly alter the source code to insert deliberate security holes, blame them on anybody or hide them in a big commit of related source etc.. If he does that in trunk, there's almost no chance to detect that.

Things like that (insertion of deliberate security holes in the source) did happen in the past, even much more elaborate hacks than the one outlines here.
Debian systems have been rooted last years because they ran too many services.

So, please move off all server apps, which are not absolutely necessary to fetch (pull/clone) and commit/push for devs, to other servers. hgwebdir and co can run on a different physical server which works on a copy of the data and has no write access to the core repo server.
OS: Linux → All
Hardware: PC → All
I tend to think that this is a good idea, but that the risk is overstated. With DCVS, everyone has the history and it's protected by cryptographic checksums, so an attacker can't just go silently changing history.
FWIW, hgwebdir *is* what lets you clone from hg.mozilla.org over http. Presumably we could run it on a separate machine, with read-only access to the repository, but keep ssh access to hg.mozilla.org as the way to push changes.
I like the idea of running hgweb on a different server with r/o copy of the repositories.

I still think we should allow browsing stuff in http.
Assignee: server-ops → aravind
Yes, though our build systems should use https, IMO.
In fact, we could do something like cvs-mirror, and have the machine that serves hg.mozilla.org over http simply have a clone of hg.mozilla.org, which it refreshes by pulling from the real repo (over ssh, with a read-only account) on a frequent cron job.
No need for a cron job... a postcommit hook can push the changes to the mirror server. This is in fact the recommended layout for repos with complex postcommit validation hooks.
I'd rather have it push-triggered, if we could, to reduce latency?
All of the hgweb stuff now lives on a different server.  There is no latency involved, since the fs holding the repos is nfs mounted (read-only).  This is limited to read-only both on the client side and the server side.

We are also now using the wsgi apache module to serve the repositories, so hopefully we will get better performance.
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
push-triggered is possible, and probably what bsmedberg meant, anyway, since presumably people can't commit directly at either of the server repos.
I had to add the old servers back into rotation to keep up with the load.  So re-opening this bug, until I can find replacements that can handle the load.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Okay, moved all the hgweb stuff over to two new VMs.  They seem like they are keeping up and handling the load okay.
Status: REOPENED → RESOLVED
Closed: 16 years ago16 years ago
Resolution: --- → FIXED
This broke the pushlog:
http://hg.mozilla.org/mozilla-central/pushloghtml is just giving me the same thing as http://hg.mozilla.org/mozilla-central/
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Should be working now.  Sorry about that.
Status: REOPENED → RESOLVED
Closed: 16 years ago16 years ago
Resolution: --- → FIXED
no idea how, but this breaks json-pushes, still, see http://hg.mozilla.org/mozilla-central/json-pushes?startID=0&endID=20

I just get an server error back, no idea what about specifically.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Should be working now.  Was missing an rpm on the server side.
Status: REOPENED → RESOLVED
Closed: 16 years ago16 years ago
Resolution: --- → FIXED
Reverting this since changes are impacting build.  Have to figure out a different way to accomplish this.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
One more shot, this time with a real server in the back end.
Status: REOPENED → RESOLVED
Closed: 16 years ago16 years ago
Resolution: --- → FIXED
Depends on: 460054
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.