609413 - Add automated spidermonkey builds with various options.

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P4)

Description

[Paul Biggar]

SpiderMonkey is tested as part of the larger mozilla build, but not on its own. That means it's tested with the following configure line:

   --enable-threadsafe --enable-ctypes --with-nspr-cflags=XX --with-nspr-libs=XX --with-dist-dir=XX --prefix=XX --with-sync-build-files=XX --{dis,en}able-debug --{dis,en}able-optimize

and depending on the platform:

   --disable-shared-js and --enable-jemalloc.


I think we need probably need to test:

  --disable-methodjit
  --disable-tracejit
  --enable-dtrace
  --enable-shark

What else is important? After a few suggestions, I'll send this over to RelEng.

As for what to test, 'make check' should be a good start I think.

Comment 1

[Mike Taylor [:bear]]

In order to do this in RelEng we would need a break down of what mozconfig's would be used by platform so we can setup the appropriate project(s)

When that is ready, bounce it back to us - thanks

Comment 2

[Paul Biggar]

Spidermonkey doesnt use mozconfigs, so I guess you need a list of ./configure lines.

Let's start with the following four configurations, and we can follow-up if we need more:

  ./configure --enable-threadsafe --with-system-nspr --enable-optimize --disable-debug --disable-methodjit
  ./configure --enable-threadsafe --with-system-nspr --enable-optimize --disable-debug --disable-tracejit
  ./configure --enable-threadsafe --with-system-nspr --enable-optimize --disable-debug --enable-debug-symbols --enable-dtrace
  ./configure --enable-threadsafe --with-system-nspr --enable-optimize --disable-debug --enable-debug-symbols --enable-shark

Comment 3

[Chris Cooper [:coop] (he/him)]

(In reply to comment #2)
>   ./configure --enable-threadsafe --with-system-nspr --enable-optimize
> --disable-debug --disable-methodjit
>   ./configure --enable-threadsafe --with-system-nspr --enable-optimize
> --disable-debug --disable-tracejit
>   ./configure --enable-threadsafe --with-system-nspr --enable-optimize
> --disable-debug --enable-debug-symbols --enable-dtrace
>   ./configure --enable-threadsafe --with-system-nspr --enable-optimize
> --disable-debug --enable-debug-symbols --enable-shark

Can you break it down by platform for us too, please? Some of them are obvious (shark, dtrace), but we don't want to take anything for granted here.

Comment 4

[Paul Biggar]

Good point:

All platforms:

   ./configure --enable-threadsafe --with-system-nspr --enable-optimize  --disable-debug --disable-methodjit
   ./configure --enable-threadsafe --with-system-nspr --enable-optimize --disable-debug --disable-tracejit


Just MacOSX:

  ./configure --enable-threadsafe --with-system-nspr --enable-optimize --disable-debug --enable-debug-symbols --enable-dtrace
  ./configure --enable-threadsafe --with-system-nspr --enable-optimize --disable-debug --enable-debug-symbols --enable-shark

Comment 5

[Paul Biggar]

Do you guys need any more information from me?

Comment 6

[Chris Cooper [:coop] (he/him)]

(In reply to comment #5)
> Do you guys need any more information from me?

Sorry, I should have pointed you to this sooner:

https://wiki.mozilla.org/ReleaseEngineering:ProjectBranchPlanning

We still need to know stuff like:
* frequency of builds (weekly vs. nightly vs. depend)
* which branches to run on
* what changes would trigger a depend build, i.e. which subdirs or files should we be watching in hg if you want depend builds
* what tests do you want?

Comment 7

[Paul Biggar]

(In reply to comment #6) 
> We still need to know stuff like:
> * frequency of builds (weekly vs. nightly vs. depend)
'depend' (I interpret that as each time the js/src/ directory is touched)

> * which branches to run on
tracemonkey.

> * what changes would trigger a depend build, i.e. which subdirs or files should
> we be watching in hg if you want depend builds
js/src/ and all subdirectories.

> * what tests do you want?
None for now. Let's just check it builds, and we can go from there.


> https://wiki.mozilla.org/ReleaseEngineering:ProjectBranchPlanning

This page is for a slightly different purpose (create a new branch), but I think I've got all the relevant info below.

- These are extra builds for the tracemonkey repo.
- We'd like one build for each configure line and platform (so 8 platforms, 4 configure line = 32 builds, in addition to existing builds).
- Incremental builds are OK (the rules for clobbering should be the same as for clobbering js/src/ in mozilla builds).
- Mobile platforms too.
- No tests are required, just the builds.
- No talos required.
- Timeline: the sooner the better of course, but no rush.
- No l10n required.

Comment 8

[Paul Biggar]

Was the information in the last comment sufficient?

Comment 9

[Paul Biggar]

Ping?

Comment 10

[Paul Biggar]

> [e-mail]: Assignee: [e-mail] ⇒ [e-mail]

I'm not 100% sure, but I think this means that no-one is going to work on this anymore? Is my understanding correct?

Comment 11

[Mike Taylor [:bear]]

It means that lsblakk thought she had time to work on it but is telling the releng triage team that it no longer is the case.

Once the full team is back from holiday break I'll bring it up at the next triage meeting to see if we can find someone to work on it.

Comment 12

[Paul Biggar]

(In reply to comment #11)
> Once the full team is back from holiday break I'll bring it up at the next
> triage meeting to see if we can find someone to work on it.

Great, thanks.

Comment 13

[Chris AtLee [:catlee]]

Do you need each variant per platform reported separately in its own tinderbox column / tbpl letter, or is a single test run per platform that does each variant in sequence and reports on the overall status ok?

Comment 14

[Paul Biggar]

I think the former is definitely preferable, thanks.

Comment 15

[Chris AtLee [:catlee]]

Is --with-system-nspr required, or can we build/use the nspr that's part of the source checkout?

Comment 16

[Paul Biggar]

(In reply to comment #15)
> Is --with-system-nspr required, or can we build/use the nspr that's part of the
> source checkout?

You can use the one that's part of the source checkout, if that's more convenient to you.

Comment 17

[Paul Biggar]

Hi Chris, can I ask the progress here? Is there anything I can do to help get this done?

Comment 18

[Benjamin Smedberg]

I don't think this is a high priority compared with various releng tasks for Firefox 4 and for the subsequent release. I'm almost tempted to say we shouldn't ask releng to do this at all: just run a local buildbot on your own, if these configurations are important.

Comment 19

[Paul Biggar]

(In reply to comment #18)
> I don't think this is a high priority compared with various releng tasks for
> Firefox 4 and for the subsequent release.

It looks like this was marked as P4, which I would guess is a kiss of death, so I think RelEng agrees with you.


We _need_ this. Code which is not tested bitrots, and we have serious bitrot issues with code which we must support, but is NPOTB. Here's a list of bugs from since I filed this:

  - disable-methodjit: bug 609206, bug 617656, bug 623137, bug 623139, bug 623277, bug 624350, bug 627516
 - dtrace: bug 613127
 - shark: 625962, 625993


We can't expect developers to check all these builds before pushing (the cost in person-time is too high), so we must have automated builds to check this (or else we spend the person-time in reporting, fixing, unbitrotting).

This is also the first step to add more automatic checks, for example see bug 626706.


> I'm almost tempted to say we
> shouldn't ask releng to do this at all: just run a local buildbot on your own,
> if these configurations are important.

I have to say I can't understand this suggestion at all.

Comment 20

[Chris AtLee [:catlee]]

You can help by providing me with a set of commands for each platform that will do what you want, with the in-source copy of NSPR.  What I have so far is:

---
<checkout into src>
mkdir objdir
cd objdir
cp $VARIANT mozconfig
echo "ac_add_options --enable-application=browser" >> mozconfig

make -f ../src/client.mk MOZCONFIG=$PWD/mozconfig configure || exit 2

(cd nsprpub; make) || exit 2
cd js/src
make -j4 || exit 2
make check || exit 1
---

Where the VARIANT mozconfig contains e.g.

nomethodjit:
ac_add_options --enable-threadsafe
ac_add_options --enable-optimize 
ac_add_options --disable-debug
ac_add_options --disable-methodjit

notracejit:
ac_add_options --enable-threadsafe
ac_add_options --enable-optimize 
ac_add_options --disable-debug
ac_add_options --disable-tracejit

I've tested this code out on my linux64 laptop, and I think it works, other than the fact that at the time I was testing (daaf6a3b8782 in tracemonkey), neither variant didn't compiled.

I haven't tested this on OSX, linux32 nor on win32.

Do you need 32-bit and 64-bit on OSX?

Comment 21

[Paul Biggar]

What you have above seems right.

The configure commands above were intended to be run from the js/src directory. However, since you also need to configure NSPR, I can see why you used mozconfigs instead.

I guess the problem with OSX and linux32 is cross-compilation? Perhaps the solution is to start with the standard mozconfigs for those configurations, and echo the nomethodjit and notracejit variants into them.


> I haven't tested this on OSX, linux32 nor on win32.

Note that it's not a problem for us if these go red in tbpl (meaning they do need to be a separate letter). We'll fix them retrospectively.
 
> Do you need 32-bit and 64-bit on OSX?

Yes, the more the merrier. That said, I think we'd just prefer something right now, so if this is hard, please feel free to make a follow-on bug for other configurations instead.


Would it be straightforward to do the following:

 - add a directory js/src/mozconfig_variants/
 - add the nomethodjit variant as js/src/mozconfig_variants/NMJ.mozconfig
, where NMJ is the 'letter' that appears in TBPL
 - add the notracejit variant as js/src/mozconfig_variants/NTJ.mozconfig
 - run your steps above for each .mozconfig file in the js/src/mozconfig_variants/ directory.

This would mean that followups like bug 626706 could be solved without your intervention. (I don't want to increase the scope of this on you, so please ignore me here if you prefer).

Comment 22

[Chris AtLee [:catlee]]

(In reply to comment #21)
> What you have above seems right.
> 
> The configure commands above were intended to be run from the js/src directory.
> However, since you also need to configure NSPR, I can see why you used
> mozconfigs instead.
> 
> I guess the problem with OSX and linux32 is cross-compilation? Perhaps the
> solution is to start with the standard mozconfigs for those configurations, and
> echo the nomethodjit and notracejit variants into them.

There's no problem (yet!) with linux32, I just haven't tested it there yet.  We
have both 32-bit and 64-bit linux build environments.

> 
> 
> > I haven't tested this on OSX, linux32 nor on win32.
> 
> Note that it's not a problem for us if these go red in tbpl (meaning they do
> need to be a separate letter). We'll fix them retrospectively.
> 
> > Do you need 32-bit and 64-bit on OSX?
> 
> Yes, the more the merrier. That said, I think we'd just prefer something right
> now, so if this is hard, please feel free to make a follow-on bug for other
> configurations instead.

Ok, that helps a lot.  I don't expect many problems getting this up and running
on linux or OSX.  Windows is always the challenging one, so I'll punt on that
if it's holding things up.

> 
> 
> Would it be straightforward to do the following:
> 
>  - add a directory js/src/mozconfig_variants/
>  - add the nomethodjit variant as js/src/mozconfig_variants/NMJ.mozconfig
> , where NMJ is the 'letter' that appears in TBPL
>  - add the notracejit variant as js/src/mozconfig_variants/NTJ.mozconfig
>  - run your steps above for each .mozconfig file in the
> js/src/mozconfig_variants/ directory.
> 
> This would mean that followups like bug 626706 could be solved without your
> intervention. (I don't want to increase the scope of this on you, so please
> ignore me here if you prefer).

Very interesting idea...I would love to be able to do this!  Let me roll this
idea around a bit and see if it's doable soon.  In the meanwhile, having the
mozconfigs under js/src somewhere makes a lot of sense, and then all I need is
a list of platforms, and which varients to run on them.  Followups like 626706
would be as easy as adding another entry to these lists.

Comment 23

[Chris AtLee [:catlee]]

Attachment: Spidermonkey project branch objects

Comment 24

[Chris AtLee [:catlee]]

Attachment: Spidermonkey project branch configs

Comment 25

[Chris AtLee [:catlee]]

Attachment: Spidermonkey project branch scripts

Comment 26

[[e-mail] (:bhearsum)]

Comment on attachment 511145 [details] [diff] [review]
Spidermonkey project branch scripts

How do you feel about adding a "set -x" here? As it stands now, it's going to be very difficult to debug issues, because the commands aren't echoed anywhere.
Looks fine othewise.

Comment 27

[Chris AtLee [:catlee]]

Attachment: Spidermonkey project branch scripts

Same as before, with set -x

Comment 28

[Chris AtLee [:catlee]]

Comment on attachment 511374 [details] [diff] [review]
Spidermonkey project branch scripts

http://hg.mozilla.org/build/tools/rev/87edf68bff3d

Comment 29

[Chris AtLee [:catlee]]

Comment on attachment 511127 [details] [diff] [review]
Spidermonkey project branch objects

Checked in on default: http://hg.mozilla.org/build/buildbotcustom/rev/420b7d36bf7d

Comment 30

[Chris AtLee [:catlee]]

Comment on attachment 511129 [details] [diff] [review]
Spidermonkey project branch configs

Checked in on default: http://hg.mozilla.org/build/buildbot-configs/rev/09d71e2e4167

Comment 31

[Aki Sasaki (not active)]

Do we still need arm support?
Spidermonkey arm and nanojit arm seem to be the last two things which are keeping scratchbox alive on our linux slaves (bug 548551)

Comment 32

[Nicholas Nethercote [inactive]]

(In reply to Aki Sasaki [:aki] from comment #31)
> Do we still need arm support?

We certainly need ARM builds and testing, but I don't see why it has to be via scratchbox -- I imagine the Android machines provide sufficient testing.

(CC'ing dmandelin and brendan for confirmation.)

Comment 33

[David Mandelin [:dmandelin]]

(In reply to Nicholas Nethercote [:njn] from comment #32)
> (In reply to Aki Sasaki [:aki] from comment #31)
> > Do we still need arm support?
> 
> We certainly need ARM builds and testing, but I don't see why it has to be
> via scratchbox -- I imagine the Android machines provide sufficient testing.
> 
> (CC'ing dmandelin and brendan for confirmation.)

Agreed.