Setup buildbot(Linux + OSX) for dehydra and treehydra

RESOLVED WONTFIX

Status

()

Core
Rewriting and Analysis
RESOLVED WONTFIX
9 years ago
4 years ago

People

(Reporter: (dormant account), Unassigned)

Tracking

Trunk
x86
Mac OS X
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 1 obsolete attachment)

(Reporter)

Description

9 years ago
Need this to stop breaking each other's work.
I'd like to work on this. I have a little experience with setting up buildbot servers/clients and I have built gcc/dehydra before.
(Reporter)

Comment 2

9 years ago
Dirkjan,
That'd be great.
Cool. Can I use infra from Mozilla to set this up, or should I try to figure things out on one of my own boxes?

Comment 4

9 years ago
Couple of things I wanted to mention here:

1. It could be useful to invoke static checking build on mozilla sources on a regular basis and log any errors/crashes encountered.

2. It could be useful to compare what is passed to JS in de/treehydra for the test c++ source between platforms. Not sure for treehydra, but for dehydra any significant deviation between what JS gets on platforms is probably a bug. It would be cool to automate this too.
I don't think we want to involve the official mozilla build infrastructure yet. I think we were planning on hosting this on some of our personal machines for the moment (I have all three platforms available at my office and can run a local buildbot master/slave setup on them)... even if you can just set up a buildbot config file for just linux or mac, that would be very helpful.
Vlad, we're definitely going to be running mozilla with static checkin on the main tinderboxes, once de/treehydra are more stable.

Comment 7

9 years ago
We had several dehydra mac regressions caused by seemingly innocent changes in dehydra code recently which could've been caught early by a hypothetical buildbot from this bug: bug 431100 and bug 431215. I don't see a reason to expect this kind of thing won't happen again, as xhydra has to use the code from gcc, and even likelier as we get treehydra up on mac because it treehydra's still moving fast.. Just thought I'd mention this here.
Yeah, sorry I haven't gotten to this yet.

I hear there's a buildbot version that's heavily modified by Mozilla. Should I be using vanilla buildbot? If the latter, any version preferences?
For Builds/Tests/etc. we have a vendor branch in CVS. It's currently at 0.7.7 + a couple of fixes. Just pull the trunk of mozilla/tools/buildbot to get it.

Updated

9 years ago
Depends on: 434331

Comment 10

9 years ago
Created attachment 321496 [details]
master.cfg for mac-only xhydra buildbot

Here is something I've put together about a week ago when xhydra on mac build got broken almost daily by incoming changes. Perhaps it can work as a start here.

Meanwhile I suppose the buildbot from this bug should probably contain a linux and mac configuration where only dehydra is built with js-1.7 because there's nothing in dehydra that should not work with last released spidermonkey.

Watching for changes in gcc patches, applying those when needed and running gcc testsuite on them wouldn't hurt either, I guess. Probably this should trigger a full xhydra rebuild too.
(Reporter)

Comment 11

9 years ago
Vlad thanks for setting this up. The problem with older JS releases isn't so much in the API as bugs. As we use more advanced spidermonkey APIs(also less used) we run into more bugs/regressions/etc.
I do not want to officially support old(ie non-CVS head) spidermonkey releases. People can use old SpiderMonkeys if they enjoy occasional pain, but I don't want to encourage that.

Comment 12

9 years ago
Created attachment 322053 [details] [diff] [review]
master.cfg for linux and mac xhydra buidbot

Added linux slave config. There is a problem with relative path not working for --js-libs arg to configure on linux (later, when in tests the thing cannot find libjs.so which is in ../installed/lib because current dir is different) so I think it makes sense to do expandvars in addition to expanduser in the script so that $PWD can be used in the value of --js-libs. There may be other way, please enlighten me if you know one. Mac's dynamic linker works in a different way, so relative paths work there given appropriate DYLD_LIBRARY_PATH.
Attachment #321496 - Attachment is obsolete: true
I suggest doing os.path.realpath on the value of js-libs, just like we do for gccdir: http://hg.mozilla.org/users/tglek_mozilla.com/index.cgi/dehydra-gcc/file/7d2e4256ec9e/configure#l35

Comment 14

9 years ago
Created attachment 322126 [details] [diff] [review]
use os.path.realpath on jslibs

Looks like this will do.
Attachment #322126 - Flags: review?(tglek)
(Reporter)

Comment 15

9 years ago
Comment on attachment 322126 [details] [diff] [review]
use os.path.realpath on jslibs

relative paths are a pain sometimes.
Attachment #322126 - Flags: review?(tglek) → review+

Comment 16

9 years ago
pushed it.
(Reporter)

Updated

9 years ago
Blocks: 437502
(Reporter)

Updated

9 years ago
No longer blocks: 423898
(Reporter)

Comment 17

9 years ago
Looks like we finally have all of the pieces to have a working linux buildbot. So what's next?

From my POV:
a) Find way to plug in more buildbots
i) One for mac
ii) One for packaging(probably should just fold that into the linux one)
b) Setup a way to bug people if they break the build. Any examples on using hg to yank out email from commit
c) Perhaps publish the buildbot waterfall page somewhere?

Comments, suggestions? 
What's the best way for buildbots to communicate over the internet?

Comment 18

9 years ago
I support this plan :). The waterfall page should definitely be publicly available. Here are some notes specifically on mac and packaging:

Apple recently published a new revision of its gcc42 source at http://www.opensource.apple.com/darwinsource/tarballs/other/gcc_42-5553.tar.gz. Our stuff works just fine with this new version, except that the part where we modify the version string in gcc had to be changed. Not sure what should we do about it and if we should do anything at all. I don't know if there will be more revisions of Apple compiler, but I guess we may expect that given that the OS update is due soon. I, for one, keep track of different mac patches in private repository at http://hg.mozilla.org/users/vladimir.sukhoy_gmail.com/apple-gcc-moz-plugin-mq just to keep things easier. Perhaps we should split our version mod in a script and not put it in a patch: the advantage is that we don't have to modify that when the new gcc version is published by Apple.

Debian packaging can be done in a separate build restricted to Debian. Note that packaging requires obtaining the source code for gcc packaging and patching it to enable plugins and build dehydra right in because we depend tightly not just on gcc source but on files produced while gcc builds. The packaging obviously changes by itself as well with respect to debian's release system. Also the build/test sequence of dehydra there is driven by debian's package build.

Not sure if there is an actual repository where the buildbot could check for changes in gcc packaging, but if nothing else the ftp site at, say, ftp://ftp.debian.org/debian/pool/main/g/gcc-4.3/ can be checked for .dsc files on a regular basis and packaging rebuilt as new files show up there.
(Reporter)

Updated

9 years ago
Status: NEW → ASSIGNED
Depends on: 450493
(Reporter)

Comment 19

9 years ago
Here is the buildbot page http://oink:8080/
(Reporter)

Comment 20

9 years ago
I meant http://63.245.208.179:8080
This is a mass change. Every comment has "assigned-to-new" in it.

I didn't look through the bugs, so I'm sorry if I change a bug which shouldn't be changed. But I guess these bugs are just bugs that were once assigned and people forgot to change the Status back when unassigning.
Status: ASSIGNED → NEW
Dehydra and treehydra are no longer maintained by Mozilla.
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.