Closed Bug 527038 Opened 15 years ago Closed 11 years ago

Bugzilla's continuous integration testing needs to report to Jenkins instead of Tinderbox-server.

Categories

(Bugzilla :: bugzilla.org, defect)

Product:
Bugzilla
Component:
bugzilla.org
Type:
defect
Priority:
Not set
Severity:
major

Tracking

()

Status:
RESOLVED WONTFIX

People

(Reporter: mkanat, Assigned: dkl)

References

Details

Tinderbox = old and busted. Hudson = new hotness. http://hudson-ci.org/ It should work with our Test::More scripts if we use something like: http://taint.org/2008/03/26/124602a.html Also, Smolder is a Continuous Integration framework that natively understands TAP (the output of Perl test scripts): http://sourceforge.net/projects/smolder/ And so may be better for us in general than Tinderbox.
Severity: normal → enhancement
Bugzilla is one of the last projects using Tinderbox. There's a good chance if we stop using it, they can shut it down. Mozilla has a Jenkins instance they use for all this stuff nowadays. Nothing for us to set up on our own, we can just use theirs, just like we currently use their Tinderbox (we'd probably need them to give us a page on it to submit to, but that's easily done). 11:54:11 <@dkl> http://www.slideshare.net/nachbaur/automating-perl-deployments-with-hudson 11:54:25 <@dkl> http://wiki.hudson-ci.org/display/HUDSON/Bazaar+Plugin 12:04:19 < solarce> justdave: if you just need a place to run tests and report on failures, use our jenkins, ci.mozilla.org -- http://blogs.perl.org/users/confuseacat/2011/09/perl-testing-with-jenkinshudson-avoiding-some-pitfalls.html
Summary: Consider using Hudson or Smolder instead of Tinderbox for Bugzilla → Investigate using something other than Tinderbox for Bugzilla's continuous integration testing
https://ci.mozilla.org is self service other than requiring an LDAP login. You should be able to just make a new job and get going
Monitor an external job This type of job allows you to record the execution of a process run outside Jenkins, even on a remote machine. This is designed so that you can use Jenkins as a dashboard of your existing automation system. See the documentation[1] for more details. [1] http://wiki.jenkins-ci.org/display/JENKINS/Monitoring+external+jobs ^^ that would probably be the easiest thing to make our existing stuff talk to.
Blocks: tinderbox-death
(In reply to Dave Miller [:justdave] from comment #1) > Bugzilla is one of the last projects using Tinderbox. There's a good chance > if we stop using it, they can shut it down. Yes, the few remaining projects that still use Tinderbox server are moving off asap. Raising from "enhancement" to "major", given the short time lines remaining here. > Mozilla has a Jenkins instance they use for all this stuff nowadays. > Nothing for us to set up on our own, we can just use theirs, just like we > currently use their Tinderbox (we'd probably need them to give us a page on > it to submit to, but that's easily done). > > 11:54:11 <@dkl> > http://www.slideshare.net/nachbaur/automating-perl-deployments-with-hudson > 11:54:25 <@dkl> http://wiki.hudson-ci.org/display/HUDSON/Bazaar+Plugin > 12:04:19 < solarce> justdave: if you just need a place to run tests and > report on failures, use our jenkins, ci.mozilla.org -- > http://blogs.perl.org/users/confuseacat/2011/09/perl-testing-with- > jenkinshudson-avoiding-some-pitfalls.html This is certainly one valid option. Happy to help unblock things if needed, or discuss if there are other options you'd prefer to explore. Who from Bugzilla team should be point on this transition?
Severity: enhancement → major
If you discover you need anything added to the ci.mozilla.org build environment, please open a new bug in https://bugzilla.mozilla.org/buglist.cgi?product=mozilla.org;component=Server%20Operations%3A%20Web%20Operations;resolution=---;list_id=5860567 and I'll add to puppet , as I am removing myself as a cc Thanks
(In reply to John O'Duinn [:joduinn] from comment #4) > (In reply to Dave Miller [:justdave] from comment #1) > > Bugzilla is one of the last projects using Tinderbox. There's a good chance > > if we stop using it, they can shut it down. > Yes, the few remaining projects that still use Tinderbox server are moving > off asap. Raising from "enhancement" to "major", given the short time lines > remaining here. > > > > > Mozilla has a Jenkins instance they use for all this stuff nowadays. > > Nothing for us to set up on our own, we can just use theirs, just like we > > currently use their Tinderbox (we'd probably need them to give us a page on > > it to submit to, but that's easily done). > > > > 11:54:11 <@dkl> > > http://www.slideshare.net/nachbaur/automating-perl-deployments-with-hudson > > 11:54:25 <@dkl> http://wiki.hudson-ci.org/display/HUDSON/Bazaar+Plugin > > 12:04:19 < solarce> justdave: if you just need a place to run tests and > > report on failures, use our jenkins, ci.mozilla.org -- > > http://blogs.perl.org/users/confuseacat/2011/09/perl-testing-with- > > jenkinshudson-avoiding-some-pitfalls.html > > This is certainly one valid option. Happy to help unblock things if needed, > or discuss if there are other options you'd prefer to explore. > > Who from Bugzilla team should be point on this transition? :mkanat, :justdave: ping? Any suggestions who is best person to contact in bugzilla team?
Flags: needinfo?(mkanat)
Flags: needinfo?(justdave)
Flags: needinfo?(wicked)
Flags: needinfo?(mkanat)
Flags: needinfo?(justdave)
Flags: needinfo?(LpSolit)
The easiest fix is probably to fix our existing tests to format the output the way jenkins expects and post it to jenkins instead of tinderbox. We have machines running all the tests already, so configuring ci to run our tests probably isn't necessary.
Summary: Investigate using something other than Tinderbox for Bugzilla's continuous integration testing → Bugzilla's continuous integration testing needs to report to Jenkins instead of Tinderbox-server.
(In reply to Dave Miller [:justdave] from comment #7) > The easiest fix is probably to fix our existing tests to format the output > the way jenkins expects and post it to jenkins instead of tinderbox. We > have machines running all the tests already, so configuring ci to run our > tests probably isn't necessary. If the tests mentioned in comment#7 need to be modified to post to jenkins, who owns driving this, and what is the ETA? Does this block decommissioning tinderbox.m.o?
(In reply to John O'Duinn [:joduinn] from comment #8) > (In reply to Dave Miller [:justdave] from comment #7) > > The easiest fix is probably to fix our existing tests to format the output > > the way jenkins expects and post it to jenkins instead of tinderbox. We > > have machines running all the tests already, so configuring ci to run our > > tests probably isn't necessary. > > If the tests mentioned in comment#7 need to be modified to post to jenkins, > who owns driving this, and what is the ETA? Does this block decommissioning > tinderbox.m.o? FWIW the scripts run prove which default to TAP formatted output. But if we install TAP::Formatter::JUnit on the testing systems and change the output of prove using --formatter=TAP::Formatter::JUnit, we should be able to push the resulting XML to the Jenkins server. Currently the scripts use a Tinderbox client module to send the results to Tinderbox. Shouldn't be too difficult to use the above instead. Just need to figure out where/how to post the results. Historically wicked has maintained the tinderbox testing host so he may have some more input on the difficulty of this. dkl
(In reply to David Lawrence [:dkl] from comment #9) > (In reply to John O'Duinn [:joduinn] from comment #8) > > (In reply to Dave Miller [:justdave] from comment #7) > > > The easiest fix is probably to fix our existing tests to format the output > > > the way jenkins expects and post it to jenkins instead of tinderbox. We > > > have machines running all the tests already, so configuring ci to run our > > > tests probably isn't necessary. > > > > If the tests mentioned in comment#7 need to be modified to post to jenkins, > > who owns driving this, and what is the ETA? Does this block decommissioning > > tinderbox.m.o? > > FWIW the scripts run prove which default to TAP formatted output. But if we > install TAP::Formatter::JUnit on the testing systems and change the output > of prove using > --formatter=TAP::Formatter::JUnit, we should be able to push the resulting > XML to > the Jenkins server. > > Currently the scripts use a Tinderbox client module to send the results to > Tinderbox. Shouldn't be too difficult to use the above instead. Just need to > figure out where/how to post the results. > > Historically wicked has maintained the tinderbox testing host so he may have > some more input on the difficulty of this. > > dkl dkl, wicked: how quickly can you remove your dependency on the tinderbox.m.o server? More specifically, if tinderbox server is powered off by OpSec, would you be ok with that? Time is ticking...
(In reply to John O'Duinn [:joduinn] from comment #10) > More specifically, if tinderbox server is powered off by OpSec, > would you be ok with that? Time is ticking... We are certainly not ok with that. The meta bug 843383 has been filed only one month ago, and you expect us to be ready to move to something else. That's not the case.
Flags: needinfo?(LpSolit)
(In reply to David Lawrence [:dkl] from comment #9) > Currently the scripts use a Tinderbox client module to send the results to > Tinderbox. Shouldn't be too difficult to use the above instead. Just need to > figure out where/how to post the results. I can try to see if we can get your Jenkins install working for us. Where can we send the JUnit formatted build output? An email address would be easiest but I doubt Jenkins does that. Is it possible to get Bazaar and TAP Plugins installed on your Jenkins so that we can automate our tests? Can we give it our own builder that executes only our tests? Do we need some kind of admin access to manage our corner on Jenkins server?
Flags: needinfo?(wicked)
(In reply to Frédéric Buclin from comment #11) > (In reply to John O'Duinn [:joduinn] from comment #10) > > More specifically, if tinderbox server is powered off by OpSec, > > would you be ok with that? Time is ticking... > > We are certainly not ok with that. The meta bug 843383 has been filed only > one month ago, and you expect us to be ready to move to something else. > That's not the case. I totally understand. I note that this was an oversight, and just provided some background in tracking bug: https://bugzilla.mozilla.org/show_bug.cgi?id=843383#c16 (In reply to Teemu Mannermaa (:wicked) from comment #12) > (In reply to David Lawrence [:dkl] from comment #9) > > Currently the scripts use a Tinderbox client module to send the results to > > Tinderbox. Shouldn't be too difficult to use the above instead. Just need to > > figure out where/how to post the results. > > I can try to see if we can get your Jenkins install working for us. Where > can we send the JUnit formatted build output? An email address would be > easiest but I doubt Jenkins does that. > > Is it possible to get Bazaar and TAP Plugins installed on your Jenkins so > that we can automate our tests? Can we give it our own builder that executes > only our tests? Do we need some kind of admin access to manage our corner on > Jenkins server? Who can best answer these questions for you and keep this possible-migration-path unblocked? In case it helps, here are some alternative suggestions I explored with other groups: 0) migrate away from using tinderbox server (the changes-to-tests described above). 1) setup and support your own different instance of tinderbox server, hosted externally or on your own hardware. 2) move the existing tinderbox.m.o server within VPN, and limit access to specific LDAP users (unknown if this is really a workable option for your project - it depends on the profile of the contributors involved, and how they interact with tinderbox. Listing for completeness.) Hope those ideas help. I know we all have a lot going on, so if you need anything, let me know and I'll do what I can to find help.
(In reply to John O'Duinn [:joduinn] from comment #13) > > In case it helps, here are some alternative suggestions I explored with > other groups: > 0) migrate away from using tinderbox server (the changes-to-tests described > above). We should definitely go this route as everything I have seen, there should not be a technical reason Bugzilla's current test suites could not run under Jenkins. The only key is getting the proper Perl/Bazaar/etc. support and then formatting the results for Jenkins. > 1) setup and support your own different instance of tinderbox server, hosted > externally or on your own hardware. Non-starter IMO. See BzAPI where this is not a good idea for long term. > 2) move the existing tinderbox.m.o server within VPN, and limit access to > specific LDAP users (unknown if this is really a workable option for your > project - it depends on the profile of the contributors involved, and how > they interact with tinderbox. Listing for completeness.) Also not a good choice as I am not sure what Mozilla's policy on giving VPN access and this really just prolongs the eventual need to move off of Tinderbox anyway. > Hope those ideas help. I know we all have a lot going on, so if you need > anything, let me know and I'll do what I can to find help. Thanks dkl
(In reply to David Lawrence [:dkl] from comment #14) > (In reply to John O'Duinn [:joduinn] from comment #13) > > > > In case it helps, here are some alternative suggestions I explored with > > other groups: > > 0) migrate away from using tinderbox server (the changes-to-tests described > > above). > > We should definitely go this route as everything I have seen, there should > not be a technical reason Bugzilla's current test suites could not run under > Jenkins. The only key is getting the proper Perl/Bazaar/etc. support and > then formatting the results for Jenkins. Yep, this would be my preferred solution also. Sounds like we're thinking the same way, so lets go this way. The alternates below were only in case this "right" solution would take a long time... more time then OpSec were comfortable with. Now of course, the question is "how long would this take to do"? :-) > > 1) setup and support your own different instance of tinderbox server, hosted > > externally or on your own hardware. > Non-starter IMO. See BzAPI where this is not a good idea for long term. > > > 2) move the existing tinderbox.m.o server within VPN, and limit access to > > specific LDAP users (unknown if this is really a workable option for your > > project - it depends on the profile of the contributors involved, and how > > they interact with tinderbox. Listing for completeness.) > Also not a good choice as I am not sure what Mozilla's policy on giving VPN > access and this really just prolongs the eventual need to move off of > Tinderbox anyway. If this helps as short term option, and reduces time pressure on the test-changes above, I'd do my utmost to help make this VPN access happen. The bugzilla folks could then do their migration work without the sound of a clock ticking loudly... > > Hope those ideas help. I know we all have a lot going on, so if you need > > anything, let me know and I'll do what I can to find help. > Thanks > dkl tc John.
(In reply to John O'Duinn [:joduinn] from comment #13) > 2) move the existing tinderbox.m.o server within VPN, and limit access to > specific LDAP users (unknown if this is really a workable option for your > project - it depends on the profile of the contributors involved, and how > they interact with tinderbox. Listing for completeness.) LDAP could mitigate security issues and could be used as a short term solution. The list of people who need access to Tinderbox for Bugzilla purposes is pretty short: justdave, glob, dkl, wicked and myself.
(In reply to John O'Duinn [:joduinn] from comment #13) > Who can best answer these questions for you and keep this > possible-migration-path unblocked? Yes, that's the question for you. I know nothing how Mozilla has organized Jenkins service support and administration internally. Is it solarce that hinted in comment 5 that we could just file bugs for changing Jenkins install? The code in question is at http://bzr.mozilla.org/bugzilla/misc/tinderbox-client/files so a patch against that would be ideal to make this progress faster. I can certainly deploy such patches (or test any promising changes).
(In reply to Teemu Mannermaa (:wicked) from comment #17) > (In reply to John O'Duinn [:joduinn] from comment #13) > > Who can best answer these questions for you and keep this > > possible-migration-path unblocked? > > Yes, that's the question for you. I know nothing how Mozilla has organized > Jenkins service support and administration internally. Is it solarce that > hinted in comment 5 that we could just file bugs for changing Jenkins > install? :solarce, I think you know more about the jenkins server install at ci.m.o, so can you directly chat with :wicked to answer his questions? > The code in question is at > http://bzr.mozilla.org/bugzilla/misc/tinderbox-client/files so a patch > against that would be ideal to make this progress faster. I can certainly > deploy such patches (or test any promising changes). Fixing those bugzilla-project files is best done by someone in the bugzilla project who has context. dkl, wicked: unclear from comment#9 which of you is best - who do you think is best to own this work?
Flags: needinfo?(dkl)
Flags: needinfo?(bburton)
(In reply to Teemu Mannermaa (:wicked) from comment #17) > (In reply to John O'Duinn [:joduinn] from comment #13) > > Who can best answer these questions for you and keep this > > possible-migration-path unblocked? > > Yes, that's the question for you. I know nothing how Mozilla has organized > Jenkins service support and administration internally. Is it solarce that > hinted in comment 5 that we could just file bugs for changing Jenkins > install? Depending on the level of change, WebOps can likely accommodate it. Mostly we manage the Jenkins service, what plugins are installed, and the environment of the host OS for builds, but we are limited in the variety of packages we can provide, particularly if it's something that shipped in RHEL 6 base, however we do have Vagrant support enabled, so you can use that to use other distros if needed. What we can't necessarily provide a lot of expertise on is the specifics of build jobs and running tests for particularly languages. As mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=527038#c2 you can make a job and get going with your LDAP login If you need an additional Jenkins plugin or packages installed into the base builder OS, please file a bug in https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org&component=Server%20Operations%3A%20Web%20Operations If I've missed a question directed at me please let me know Thanks
Flags: needinfo?(bburton)
(In reply to John O'Duinn [:joduinn] from comment #18) > (In reply to Teemu Mannermaa (:wicked) from comment #17) > > (In reply to John O'Duinn [:joduinn] from comment #13) ... > > The code in question is at > > http://bzr.mozilla.org/bugzilla/misc/tinderbox-client/files so a patch > > against that would be ideal to make this progress faster. I can certainly > > deploy such patches (or test any promising changes). > > Fixing those bugzilla-project files is best done by someone in the bugzilla > project who has context. dkl, wicked: unclear from comment#9 which of you is > best - who do you think is best to own this work? dlk, wicked: ping?
(In reply to John O'Duinn [:joduinn] from comment #20) > (In reply to John O'Duinn [:joduinn] from comment #18) > > (In reply to Teemu Mannermaa (:wicked) from comment #17) > > > (In reply to John O'Duinn [:joduinn] from comment #13) > ... > > > The code in question is at > > > http://bzr.mozilla.org/bugzilla/misc/tinderbox-client/files so a patch > > > against that would be ideal to make this progress faster. I can certainly > > > deploy such patches (or test any promising changes). > > > > Fixing those bugzilla-project files is best done by someone in the bugzilla > > project who has context. dkl, wicked: unclear from comment#9 which of you is > > best - who do you think is best to own this work? > > dlk, wicked: ping? (bah - sorry for typo) dkl, wicked: ping?
I am finishing up a couple of quarter goals atm. Wicked, do you have time to look into this further, if not I can pick it up when I get done. dkl
Flags: needinfo?(dkl) → needinfo?(wicked)
(In reply to Frédéric Buclin from comment #16) > LDAP could mitigate security issues and could be used as a short term > solution. The list of people who need access to Tinderbox for Bugzilla > purposes is pretty short: justdave, glob, dkl, wicked and myself. ldap protection looks like a reasonable short-term fix, to buy us time to sort out the migration to jenkins. joduinn: is there an issue with implementing that?
(In reply to Brandon Burton [:solarce] from comment #19) > As mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=527038#c2 you > can make a job and get going with your LDAP login I can't see anything on the site to create a new job and if I go to the /createItem URL, it says " Access Denied - wicked@sci.fi is missing the Create permission" so I guess I'm not allowed to add new jobs? (In reply to David Lawrence [:dkl] from comment #22) > I am finishing up a couple of quarter goals atm. Wicked, do you have time to > look into this further, if not I can pick it up when I get done. It'll take at least until next week/weekend for me to have time to try to figure out Jenkins. After a brief look through it's API help pages, I didn't see any obvious ones that allow me to submit new build logs so that's first thing that I need to figure out. (In reply to Byron Jones ‹:glob› from comment #23) > ldap protection looks like a reasonable short-term fix, to buy us time to > sort out the migration to jenkins. Especially since 3 of the 5 people needing access are employees it shouldn't be that hard to grant special access.
Flags: needinfo?(wicked) → needinfo?(bburton)
(In reply to Teemu Mannermaa (:wicked) from comment #24) > (In reply to Brandon Burton [:solarce] from comment #19) > > As mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=527038#c2 you > > can make a job and get going with your LDAP login > > I can't see anything on the site to create a new job and if I go to the > /createItem URL, it says " Access Denied - wicked@sci.fi is missing the > Create permission" so I guess I'm not allowed to add new jobs? > Ah, there is another level of permissions to actually make and manage jobs, which I gave your user, let me know the other community folks who'd need this level of access
Flags: needinfo?(bburton)
How is this going? We still need to keep moving on decommissioning the Tinderbox server. This remains a high priority. Please let me know if you have any questions about the urgency or need to shutdown the tinderbox server. -- Joe Stevensen Operations Security Manager
:wicked, it appears I misunderstood your last jenkins question in #it, you can delete jobs whenever you want, they just stick around unless you explicitly delete one. As far as the naming, there isn't a strict convention, something to do with the project at the beginning is helpful, e.g. bugzilla-master
The last sections look promising: https://wiki.jenkins-ci.org/display/JENKINS/Monitoring+external+jobs At first glance, it looks like basically the tests scripts that execute on cg-bugs01 should be converted to use TAP::Formatter::JUnit to output the results as XML. Example: prove --formatter=TAP::Formatter::JUnit -l t/*.t > /tmp/test_results_<uniq_id>.xml curl -X POST -d @/tmp/test_results_<uniq_id>.xml \ http://user:pass@ci.mozilla.org/jenkins/job/<job_name>/postBuildResult rm /tmp/test_results_<uniq_id>.xml This is all in theory. dkl
(In reply to Teemu Mannermaa (:wicked) from comment #24) > (In reply to Byron Jones ‹:glob› from comment #23) > > ldap protection looks like a reasonable short-term fix, to buy us time to > > sort out the migration to jenkins. > > Especially since 3 of the 5 people needing access are employees it shouldn't > be that hard to grant special access. Everyone in the project who would need access already has an LDAP account in order to get bzr commit access. You could certainly restrict it to people in the bzr_bugzilla group rather than employees since Bugzilla's the only people left using it.
i've filed bug 862665 to discuss/implement restricting access by ldap group, so this bug can focus on the jenkins work.
(In reply to Byron Jones ‹:glob› from comment #30) > i've filed bug 862665 to discuss/implement restricting access by ldap group, > so this bug can focus on the jenkins work. Posted in bug 862665; crossposting here to make sure this is seen by all involved. If we are going to use the fallback plan of moving this server behind LDAP, we need to change the name from tinderbox.m.o to something else. Anything else. Only the bugzilla folks are using it, so they can choose the new name. We're using the announcements of EOL-of-tinderbox to train people to use new tree.m.o servers, and have bugs blocked for setting up redirects, etc from tinderbox.m.o to the new server to help with those announcements. See dep bugs linked to bug#843383 for details. Moving tinderbox behind ldap while keeping the same tinderbox.m.o name would block that work and not be ok. I note this means bugzilla folks need to change what email address they send test results to. Not sure how easy/hard this is. And sorry for not thinking of this when I suggested moving-behind-ldap as a fallback plan in comment#13.
(In reply to John O'Duinn [:joduinn] from comment #31) > We're using the announcements of EOL-of-tinderbox to train people to use new > tree.m.o servers (Sorry for more offtopic) Similar to in the other bugs, we cannot move TBPL to tree.m.o - we need to redirect tree.m.o to tbpl.m.o (for now), otherwise it will break our rollout plan for treeherder. Please don't change TBPL fqdn, or use tree.m.o is the EOL announcement without speaking to me first.
I'll sync up with dkl on Monday and see how I can help make the bugzilla CI move from tinderbox to Jenkins. I took a look today and this seems fairly easily doable. I'll confirm once I check with dkl on Monday.
Assignee: website → shyam
(In reply to Joe Stevensen [:joes] from comment #26) > How is this going? We still need to keep moving on decommissioning the > Tinderbox server. This remains a high priority. And you have to realize we are doing this as volunteers and have other things we need to so we can pay our bills. If you are in a hurry then you should pay somebody to do the conversion. That's how things work in open-source projects. That said, I'm on vacation now and have more time to look into this. (In reply to Brandon Burton [:solarce] from comment #27) > :wicked, it appears I misunderstood your last jenkins question in #it, you > can delete jobs whenever you want, they just stick around unless you > explicitly delete one. Yeah, that's why I wanted to verify the situation. It's a good thing to know jobs can be deleted so I can more freely test Jenkins. Thanks for that and also answering the naming question. I'll probably use bz- or bugzilla- as the prefix for our jobs. (In reply to David Lawrence [:dkl] from comment #28) > prove --formatter=TAP::Formatter::JUnit -l t/*.t > > /tmp/test_results_<uniq_id>.xml > curl -X POST -d @/tmp/test_results_<uniq_id>.xml \ > http://user:pass@ci.mozilla.org/jenkins/job/<job_name>/postBuildResult > rm /tmp/test_results_<uniq_id>.xml Yes, I read the same and there's also the post URL I was looking for. However, problem I just realized is that there's a higher level logic that lives in Tinderbox::Client, which runs commands like prove and sends emails to the Tinderbox Server address. So the prove/TAP output is only small part of only some jobs. For example, the docs build jobs have no prove/TAP commands and instead only record pure console output. (Jenkins external job monitoring does handle exactly this situation so we are not doomed yet.) I think we have two options now. First is to build a mail gateway that receives the tinderbox mail output, converts it to Jenkins JUnit format and submits it to the post URL. Second is to hack Tinderbox::Client to add posting of Jenkins compatible output that gets done instead of sending mail (basically instead of Tinderbox::Client::Mailer package use a new Tinderbox::Client::Jenkins package). It's actually good thing we can't use TAP::Formatter::JUnit since there's no RPM for that and it would also require Moose. :) Instead there's RPM for TAP::Harness::JUnit that could have been used to produce JUnit output from any TAP::Harness based test scripts.
<mdas> dkl: but davehunt has lots of experience with jenkins as well, perhaps he can help you <dkl> mdas, ok. cool. I dont know anything about autolog but I can look into that as well. I will ping dave hunt and see if he has any other info too. Thanks <mdas> dkl: oh autolog is like a homebrew version of the tbpl display. autolog gathers test data and displays it like so: http://brasstacks.mozilla.com/autolog/?tree=gaia&source=autolog <mdas> but it's most likely not waht you need <davehunt|mtg> Jenkins uses the exit code, but can also analyze JUnit XML output in a post build process <mdas> the way we used jenkins was pretty much "did our suite fail? yes or no?" using exit codes then look at the output of the log, which was just pushed to stdout I think <dkl> mdas, ah i see. nice. So the test scripts are initiated by Jenkins? And when they done the full results are sent to autolog? <davehunt|mtg> autolog submission is within Marionette test runner, I'm not sure who follows that though <mdas> davehunt|mtg: dkl: yeah the submission to autolog is done completely by the marionette test runner <dkl> mdas, davehunt|mtg: in our scenario (Bugzilla) we would be running the test suite on a dedicated system. We can output the test results as JUnit XML, but we want to push that to the jenkins server for reporting purposes, etc. <davehunt|mtg> dkl: so Jenkins wouldn't be running the tests? <dkl> davehunt|mtg, right. i suppose we would need to have the job(s) defined in Jenkins but then the builds would be initiated on the test server based on polling of the SCM repo <davehunt|mtg> Jenkins can run jobs on remote servers <dkl> davehunt|mtg, yeah saw that in the config but we wouldnt need to go that far i dont think <davehunt|mtg> dkl: when I'm out of this meeting, I'd be happy to talk about what you need <dkl> davehunt|mtg, cool thanks <davehunt|mtg> In case I can help <davehunt> dkl: so, what are you planning on using Jenkins for? <davehunt> okay, so if Jenkins is triggering your builds you should be fine with the reporting in Jenkins <mcote> yeah I don't think we were talking about generating builds via jenkins, but I'm not sure <mcote> I will let dkl answer that :) <davehunt> by 'build' I mean execution of a job defined in Jenkins, not actually building Bugzilla, necessarily <mcote> ah <davehunt> the terminology can be confusing :) <mcote> as usual :) <dkl> davehunt, what mcote said. We are fine with initiating the builds on the separate server and then feeding the results to Jenkins for recording purposes. I guess the question should be is that a possible and useful scenario. <dkl> Or is it necessary for Jenkins to control when the new test run is initiated. <davehunt> It would be a bit strange to use Jenkins just as a place for your reports <dkl> ah <dkl> cause basically that was how Tinderbox is being used <davehunt> Right, but Tinderbox is a reporting tool, essentially, right? <dkl> yes. it also has ties into email IRC etc. <dkl> davehunt, http://tinderbox.mozilla.org/showbuilds.cgi?tree=Bugzilla <davehunt> Right, so you want to take advantage of Jenkins post build actions.. <dkl> yep <davehunt> hmm So how is the build currently triggered? I think if you could get Jenkins involved in the triggering, that would work much better for you <dkl> davehunt, bzr checkout and then a cron that looks for commits davehunt, i realize that jenkins can do that as well. But then we would need a way for Jenkins to send a message to the server to run the tests and then report back. <davehunt> dkl: It can do that using a slave node and SSH <dkl> davehunt, a quick win would be to just utilize what is already working and just send the results to Jenkins I imagined <davehunt> dkl: afaik Jenkins can't simply listen for results, it needs a trigger and then collects data <dkl> davehunt, ah. good to know <davehunt> dkl: so you _could_ trigger via REST API and submit the results in a parameter or publish them elsewhere for Jenkins to grab <dkl> davehunt, one issue is that the test server being used already has all of the dependencies, etc. needed to run a test Bugzilla site <dkl> also it has the databases with test data. <davehunt> dkl: it could still be the machine that runs the build, if the Jenkins master can communicate with it over SSH <dkl> davehunt, cool. how would the test scripts communicate back to Jenkins when it is completed and send the XML output? <dkl> or would that be ssh as well? <davehunt> dkl: all taken care of via the SSH connection <dkl> ah <dkl> davehunt, any example code anywhere? <davehunt> dkl: this shows you the basics for setting up a master/slave config in Jenkins https://wiki.jenkins-ci.org/display/JENKINS/Step+by+step+guide+to+set+up+master+and+slave+machines <davehunt> dkl: Basically you define the repository to clone, your shell commands to run your build and/or tests, and any post build actions (email, IRC, etc) in a Jenkins job. You can then say to only run this on a certain node. That node is your build machine running the slave.jar or with SSH open so it can be brought up on demand. <davehunt> Then when the job is triggered (perhaps when a change in the repo is detected, or on a schedule) it copies everything it needs to the node, runs the job, and archives results and any artifacts on the master. <davehunt> dkl: I have to head out, but feel free to ping me (or point others towards me) if you have any Jenkins related questions.
:dkl, wicked, fox2mike: Ping? Its been 2 weeks since the last update here, and from reading comment#33-35, I'm unclear on who is doing what, or what the ETA is.
Flags: needinfo?(wicked)
Flags: needinfo?(shyam)
Flags: needinfo?(dkl)
I haven't been able to get in touch with dkl to actually spend sometime on this. I also believe he's unwell today, so might be a few days before I can touch base.
Flags: needinfo?(shyam)
(In reply to John O'Duinn [:joduinn] from comment #36) > :dkl, wicked, fox2mike: > > Ping? Its been 2 weeks since the last update here, and from reading > comment#33-35, I'm unclear on who is doing what, or what the ETA is. ping?
Way more complex than I thought. It needs some work to move to Jenkins.
Assignee: shyam → dkl
Wicked. Can you give me sudo on the community box when you get a chance? Ping me when it is done. Thanks dkl
Flags: needinfo?(dkl)
We never heard back on bug 862665, which, while obviously not a permanent solution, would be a stop-gap measure while we switch over to a new CI system.
I spent the day yesterday on this and I have a semi-working client working on the community system being used for Tinderbox testing. I was able to duplicate the current Tinderbox client (Renaming to Jenkins::Client) and made the needed changes to post the data to Jenkins instead of sending formatted emails. Currently it sends the log output and end result in XML to the preconfigured Job name. It is not formatting the log output any differently than before but you can view the full log contents in the Jenkins build. At the moment, even though the data is being sent, Jenkins is crashing with a Java error before finishing that I need to ping one of the Jenkins maintainers about. After, all that needs to be done is to setup all of the different jobs in Jenkins and then configure the community server to loop and monitor the Bugzilla repos for changes. You can see some sample runs at: https://ci.mozilla.org/job/bugzilla-tip/ dkl
(In reply to David Lawrence [:dkl] from comment #42) > I spent the day yesterday on this and I have a semi-working client working > on the community system being used for Tinderbox testing. w00t! thats great to hear. > I was able to duplicate the current Tinderbox client (Renaming to > Jenkins::Client) and made the needed changes to post the data to Jenkins > instead of sending formatted emails. > > Currently it sends the log output and end result in XML to the preconfigured > Job name. It is not formatting the log output any differently than before > but you can view the full log contents in the Jenkins build. > > At the moment, even though the data is being sent, Jenkins is crashing with > a Java error before finishing that I need to ping one of the Jenkins > maintainers about. Urgh. Any news? > After, all that needs to be done is to setup all of the different jobs in > Jenkins and then configure the community server to loop and monitor the > Bugzilla repos for changes. Who is on the hook to do this? > You can see some sample runs at: > https://ci.mozilla.org/job/bugzilla-tip/ I see some green (as well as red!) jobs there, which is encouraging. Who can evaluate these for accuracy compared to the jobs running on tinderbox server? > dkl Thanks dkl! John.
Flags: needinfo?(dkl)
(In reply to John O'Duinn [:joduinn] from comment #43) > (In reply to David Lawrence [:dkl] from comment #42) > > I spent the day yesterday on this and I have a semi-working client working > > on the community system being used for Tinderbox testing. > w00t! thats great to hear. > > > > I was able to duplicate the current Tinderbox client (Renaming to > > Jenkins::Client) and made the needed changes to post the data to Jenkins > > instead of sending formatted emails. > > > > Currently it sends the log output and end result in XML to the preconfigured > > Job name. It is not formatting the log output any differently than before > > but you can view the full log contents in the Jenkins build. > > > > At the moment, even though the data is being sent, Jenkins is crashing with > > a Java error before finishing that I need to ping one of the Jenkins > > maintainers about. > Urgh. Any news? An upstream Jenkins plugin patch was available to fix the error. Solarce installed the updated plugin with the fix last week which fixed the issue for me. But unfortunately it caused others to have problems and the new plugin had to be downgraded until the issue is resolved :( Solarce has mentioned it was in his plan to setup a staging environment for Jenkins and he is happy to install the updated plugin there once it is up and running. We could then push our test results to it and monitor that. It is of course not ideal for the long term as staging systems tend to be wiped and refreshed from time to time. But hopefully production would be fixed at some point soon after. Solarce has no timeframe for when the staging server will be installed and available. The other option which is more complex is to make the current Bugzilla community server a Jenkins slave host and have Jenkins initiate the test runs via SSH and the pipe the results back for reporting. This will involve getting the proper Java jars and support installed on the community server and then have IT open up the right flows. There is also talk of moving off of Jenkins completely and move to an outsourced solution such as Travis CI or similar. We would have to port again to whatever is chosen. > > After, all that needs to be done is to setup all of the different jobs in > > Jenkins and then configure the community server to loop and monitor the > > Bugzilla repos for changes. > Who is on the hook to do this? I have created the different jobs right now and each one will contain the different sub builds eventually. > > You can see some sample runs at: > > https://ci.mozilla.org/job/bugzilla-tip/ > I see some green (as well as red!) jobs there, which is encouraging. Who can > evaluate these for accuracy compared to the jobs running on tinderbox server? Unfortunately the reds occur due to the error generated by Jenkins when posting the results even though the test runs themselves were all PASS.
Flags: needinfo?(dkl)
(In reply to David Lawrence [:dkl] from comment #44) > (In reply to John O'Duinn [:joduinn] from comment #43) > > (In reply to David Lawrence [:dkl] from comment #42) > > > I spent the day yesterday on this and I have a semi-working client working > > > on the community system being used for Tinderbox testing. > > w00t! thats great to hear. > > > > > > > I was able to duplicate the current Tinderbox client (Renaming to > > > Jenkins::Client) and made the needed changes to post the data to Jenkins > > > instead of sending formatted emails. > > > > > > Currently it sends the log output and end result in XML to the preconfigured > > > Job name. It is not formatting the log output any differently than before > > > but you can view the full log contents in the Jenkins build. > > > > > > At the moment, even though the data is being sent, Jenkins is crashing with > > > a Java error before finishing that I need to ping one of the Jenkins > > > maintainers about. > > Urgh. Any news? > > An upstream Jenkins plugin patch was available to fix the error. Solarce > installed the updated plugin with the fix last week which fixed the issue > for me. But unfortunately > it caused others to have problems and the new plugin had to be downgraded > until the issue > is resolved :( > Yes, unfortunately the plugin that's causing the API :dkl needs to use return a 500 broke when we upgraded to the version that supposed to fix his issue and made all our most important builds stop working. > Solarce has mentioned it was in his plan to setup a staging environment for > Jenkins and he is happy to install the updated plugin there once it is up > and running. We could then push our > test results to it and monitor that. It is of course not ideal for the long > term as staging > systems tend to be wiped and refreshed from time to time. But hopefully > production would be > fixed at some point soon after. Solarce has no timeframe for when the > staging server will be installed and available. > > The other option which is more complex is to make the current Bugzilla > community server a Jenkins slave host and have Jenkins initiate the test > runs via SSH and the pipe the results back for reporting. This will involve > getting the proper Java jars and support installed on the community server > and then have IT open up the right flows. > The current Jenkins setup does not let us use slaves because of how we're using LDAP authentication and how our Puppet module is built, this is something I also need a staging environment to work on and make happen, but before we invest that time we want to explore if we should just use Hosted CI > There is also talk of moving off of Jenkins completely and move to an > outsourced solution such as Travis CI or similar. We would have to port > again to whatever is chosen. > > > > After, all that needs to be done is to setup all of the different jobs in > > > Jenkins and then configure the community server to loop and monitor the > > > Bugzilla repos for changes. > > Who is on the hook to do this? > > I have created the different jobs right now and each one will contain the > different sub builds eventually. > > > > You can see some sample runs at: > > > https://ci.mozilla.org/job/bugzilla-tip/ > > I see some green (as well as red!) jobs there, which is encouraging. Who can > > evaluate these for accuracy compared to the jobs running on tinderbox server? > > Unfortunately the reds occur due to the error generated by Jenkins when > posting the results even though the test runs themselves were all PASS. Jenkins needs a fair amount of care and feeding at this point, so we're discussing with webdev (the biggest jenkins users) the feasibility of moving to Travis/Hosted Jenkins via email right now I would recommend you take a look at if you could just use Travis for your needs
So, here's a thought on how to secure the existing tinderbox if we need to lock it down short term until the jenkins stuff gets worked out. Assumptions: 1. The main pages that need to be publicly-visible are the waterfall pages. There are static cached versions of those pages which are generated by tinderbox to allow for speedy page delivery and so forth. 2. The people who need to access the pages for notes and tree control and such all have committer rights and thus LDAP access. The idea: 1. Move the tinderbox server behind the firewall. 2. Point the tinderbox domain name at a proxy server. 3. Restrict the proxy server so that it will only pass through the URLs to the static waterfall pages, and maybe an index page 4. People who need to edit things can log into the VPN (perhaps on the contributor VLAN, we have one there, right?) to access the real machine directly. Thoughts?
Severity: major → enhancement
> 2. Point the tinderbox domain name at a proxy server. Or whatever domain it ends up sitting behind - per comment 31 it won't be t.m.o :)
damnit, I did not touch the severity. /me yells at the Firefox cache
Severity: enhancement → major
Actually, if we go with the proxy route, we *could* keep the t.m.o domain name, as the proxy could redirect everything other than the Bugzilla-specific URLs to the new site.
(In reply to Dave Miller [:justdave] from comment #46) > So, here's a thought on how to secure the existing tinderbox if we need to > lock it down short term until the jenkins stuff gets worked out. that's bug 862665. i'll copy your comments there.
(In reply to Dave Miller [:justdave] from comment #46) > So, here's a thought on how to secure the existing tinderbox if we need to > lock it down short term until the jenkins stuff gets worked out. > > Assumptions: > 1. The main pages that need to be publicly-visible are the waterfall pages. > There are static cached versions of those pages which are generated by > tinderbox to allow for speedy page delivery and so forth. > 2. The people who need to access the pages for notes and tree control and > such all have committer rights and thus LDAP access. unknown. bugzilla folks could confirm these assumptions. > The idea: > 1. Move the tinderbox server behind the firewall. > 2. Point the tinderbox domain name at a proxy server. > 3. Restrict the proxy server so that it will only pass through the URLs to > the static waterfall pages, and maybe an index page > 4. People who need to edit things can log into the VPN (perhaps on the > contributor VLAN, we have one there, right?) to access the real machine > directly. > > Thoughts? joes (in OpSec) would need to comment on whether this proposal resolves his open security concerns, flagging him for needs-info. aiui, unaddressed in the above proposal are: 1) who will maintain this tinderbox server while the rest of the transition work is completed? 2) I have no info on how much work remains to complete transition from tinderbox to jenkins, so dont know if this workaround is best use of our limited human time, or whether we'd spend as-much-time on the workaround as on the "real" fix. Another proposal could be to export/move the tinderbox instance onto a bugzilla-community machine owned and hosted outside of Mozilla. This would not solve the security problems with the tinderbox instance, but would reduce the time-pressure from Opsec and RelEng. The security problems would now become those of whomever agreed to host the instance outside of Mozilla. I'm not sure this proposal is a good idea, but I want to list it in case others think it helps.
Flags: needinfo?(jstevensen)
let's keep this bug on the topic of moving bugzilla to jenkins/travis please. bug 862665 is about restricting access to the current tinderbox, i'll duplicate your comment over there.
Flags: needinfo?(jstevensen)
(In reply to Byron Jones ‹:glob› from comment #52) > let's keep this bug on the topic of moving bugzilla to jenkins/travis please. > > bug 862665 is about restricting access to the current tinderbox, i'll > duplicate your comment over there. Not much to say new on the Jenkins front. Will either need to spend some quality time seeing what it would take to make the community server a full test slave by installing the necessary Java dependencies or wait til the current Jenkins system is updated or a staging server is available. From what I can tell so far Travis CI may be a non-starter as it is tightly bound to git(hub) and does not seem to have any support for BZR repositories. We would either need to move Bugzilla's code over to github or set up some sort of mirror with BZR->Git that Travis CI would be able to monitor. Right now securing the current Tinderbox system behind LDAP will buy us the extra time we need. As glob has mentioned we are in no way volunteering to take over active maintainership of the Tinderbox system and if time and resources were not an issue, it would most definitely need to be rewritten in a more modern framework. dkl
Flags: needinfo?(wicked)
As suggested in comment 45, we are going to use travis for our CI testing (bug 983275).
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.