Closed Bug 633478 Opened 15 years ago Closed 15 years ago

A Jenkins public server for Services

Categories

(Cloud Services :: Operations: Miscellaneous, task)

Product:
Cloud Services
Component:
Operations: Miscellaneous
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: tarek, Assigned: mcoates)

References

Details

The build system in all Services server apps relies on: - pypi.python.org Python packages repository (mandatory) - moz.ziade.org for upgrading the build system (optional) I'd like to set up a box that can be reached from inside and outside, that will: - provide a mirror of pypi.python.org when build occurs (even if the builder is not inside our network) - replace moz.ziade.org This box could also be used to: - migrate the Hudson system we have in tarek1. - generate RPMs for all projects and publish them in a directory accessible from apache
Oh, and also we can publish there the HTML version of the guide that leaves here for now: http://sync.ziade.org/doc/
Severity: normal → major
Note that we'll need both CentOS 5 (phx, dev, stage) and RHEL 6 (scl2) builds of the RPMs.
(In reply to comment #2) > Note that we'll need both CentOS 5 (phx, dev, stage) and RHEL 6 (scl2) builds > of the RPMs. In that case we'll need a second box with RHEL6, where a hudson slave can run to build those RPMS.
(In reply to comment #1) > Oh, and also we can publish there the HTML version of the guide that leaves > here for now: http://sync.ziade.org/doc/ As we start to share our code, the doc is starting to get linked (like https://wiki.mozilla.org/Security/Users_and_Logs#How_to_Log_Events_-_CEF_Library) I think it would be good to have that box soon, with a dedicated CNAME. Maybe docs.services.mozilla.com ?
Not sure what's the next step here. Do we wait for an OK from QA then Ops can go ahead, or do I need to add IT to that ticket ?
Do we want to conflate the doc site with the jenkins instance? I don't think we do, really, but I don't know what the plus/minus is there. Also, random 2 cents: let's not do builds on the master, if possible, and have a slave per target environment. Then we don't need to reconfig the master if we change reqs/platforms for something.
Not sure why we'd need anything from QA here. Think this is a straight req to ops people.
(In reply to comment #6) > Do we want to conflate the doc site with the jenkins instance? I don't think > we do, really, but I don't know what the plus/minus is there. Right, the doc site can be anywhere. It's just handier, because I can manage myself that box, and maintain all those things. > > Also, random 2 cents: let's not do builds on the master, if possible, and have > a slave per target environment. Then we don't need to reconfig the master if > we change reqs/platforms for something. I've been talking with Gozer lately about this for the f1 project, and he said he could work w/ me on setting up Mock -- that tool would let us build stuff for all the CentOS/RedHat/Fedora flavors on the same box. But at some point, we would definitely want another slave box. I propose that we start with 2 box and see how the mock stuff goes.
Blocks: 635367
(In reply to comment #7) > Not sure why we'd need anything from QA here. Think this is a straight req to > ops people. ooops typo - I meant infrasec - related to the fact that the box will be reachable from outside.
Separated the discussion about the doc website at bug 637669
Separated the discussion about PyPI mirroring at bug 637676
cc'ing zandr, as from a similar discussion with infrasec + philipp, this might be suited for Labs.
Yikes. Nobody said anything to me about Hudson/Jenkins. My understanding from hallway conversations with clyon is that publicly exposing Hudson is a non-starter.
Can we expose the data somehow, without the interface? Hudson has feeds... maybe we can publish just that part, and generate static content with just the data?
(In reply to comment #13) > Yikes. Nobody said anything to me about Hudson/Jenkins. My understanding from > hallway conversations with clyon is that publicly exposing Hudson is a > non-starter. We took another look at hudson today and I would say that we won't block on this but would like specific things if this was going to be deployed for services. Complete isolation is one of them. I am not going to go into deals on a public bug the findings that we have seen but we have reported all of them but not all of them are fixed. I will take with zandr to discuss some of the details.
Chris, any news on this ? Anything I can do to move this forward ? Note that webdev already runs a public Hudson (http://hudson.mozilla.org)
Severity: major → critical
(In reply to comment #16) > Chris, any news on this ? Anything I can do to move this forward ? > > Note that webdev already runs a public Hudson (http://hudson.mozilla.org) yes, we are aware of their public instance, which is why we are hesitant. It has some significant security issues. Adding in mcoates who can re-eval this.
Whiteboard: [pending secreview] [go-live: now]
The latest version of hudson is 1.398 do we have a stage version of this setup? I'd like to review some key security issues that I've identified in previous tests.
Tarek, can we do a private buildbox initially, with the understanding that we'll continue the sec reviews and so forth regarding a public buildbox? I don't want to block having something very useful for dev (a buildbox) on something not so relevant to us at MoCo (anonymous access).
(In reply to comment #19) > Tarek, can we do a private buildbox initially, with the understanding that > we'll continue the sec reviews and so forth regarding a public buildbox? I > don't want to block having something very useful for dev (a buildbox) on > something not so relevant to us at MoCo (anonymous access). I already have that box (the current hudson we use) so I am not sure if it's very useful to re-create it, if we need to create yet another one public. unless the same VM can become public once the review has been done ? What I can do is upgrade my hudson to the latest jenkins for the security review.
So, after a discussion with atoll on IRC, it occured to us that we could have: - a private build server that creates the RPMs for us - a public hudson server that just runs the tests The private hudson can push the RPMs to a public read-only space once they are built, so people can grab them. So we could split this discussion in two parts, and reduce the security review for the public hudson. If this sounds good I can create another bug for the "private hudson"
(In reply to comment #21) > If this sounds good I can create another bug for the "private hudson" I like that idea.
What else needs to happen in this bug?
Assignee: nobody → mcoates
Whiteboard: [pending secreview] [go-live: now] → [in-progress secreview] [go-live: now]
The buildbox will be build as a private VM, and we'll just want to have a public Jenkins instance that does not build any RPMs for us, but just run the tests continuously for our projects
Summary: A buildbox for Services → A Jenkins public server for Services
(In reply to comment #24) > The buildbox will be build as a private VM, and we'll just want to have a > public Jenkins instance that does not build any RPMs for us, but just run > the tests continuously for our projects Does this bug need to be open still? Any other security review needs? Should it be handed off to someone else?
It still need to be open. The security review is now just concerning the setup of a jenkins server that will run tests against our public repositories.
New bugs will be open for the necessary security reviews.
Whiteboard: [in-progress secreview] [go-live: now]
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.