Closed Bug 442522 Opened 16 years ago Closed 16 years ago

Categories

(Release Engineering :: General, defect, P2)

x86
macOS
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: sayrer, Assigned: bhearsum)

References

Details

Attachments

(1 file)

This is relatively important to get running on one platform soon. Linux or OS X would be fine. The buildbot should run mochitest and the js testsuite.
Build requirements should be the same as mozilla-central right?
Yes.
How important? Is this something you need *today* or is sometime in the next
week ok?
Sometime this week is fine.
If it happens this week, that's fine I think.
Questions, questions:

1) when you say "js testsuite" you are referring to bc's set of tests?

2) Do we have a tinderbox this machine can report to, e.g., mozilla-central?

3) any special build/configure options required on linux? (requirements same, .mozconfigs different?)

Given the complexity of setting up the machines to do js test work, I don't expect this by end of this week. I'll need to coordinate with BC to figure out his changes for mozilla-central. Also, we've never run mochitests inside an environment that was running the js test suite, though I don't expect that to be terribly difficult, it'll require some automation tweaks.
Yes, the js/tests suite (-L lc2 lc3 spidermonkey-n.tests slow-n.tests).

We don't need a tinderbox for it to report to, reading the waterfall is fine; it shouldn't be on mozilla-central because it's not testing mozilla-central, for sure.  If it'll take more than a few days to get the JS tests running, can we get mochitest first?  (I thought we already had the JS suite reporting via tinderboxes for months, so I didn't realize it would be hard to clone that config. :/)  If it's easier to run the suite in the shell, it's OK to start with that, too.

If not everything this week, when should we expect it?  Not having test coverage is making it hard to go fast, and it takes an hour to run the test suite (even without slow-n) on Andreas' machine.
How many of the hg repositories need to have tests running? I was going to file a bug for getting vms/hardware for running on mozilla-central and perhaps actionmonkey and this bug adds tracemonkey. Anything else?
I don't know, and I don't know how we would predict.  For now, it sounds like those three need coverage.  Can we not just add them as we discover a need?  (I'm not sure actionmonkey is actually active enough to need a test coverage machine, but I'm not following it super closely -- not this bug, regardless.)
We already have unit-test coverage for mozilla-central... why do we need separate VMs for js/tests coverage, instead of just adding steps to the existing unit-test process?
This bug is not about mozilla-central, and we don't want to run more tests than we care about (reftests, Tunit, f.e.) until we no longer see the massive false-failure rate that is currently in evidence.

The bike shed can be green with sparkles; we need:

- js test suite coverage
- mochitest coverage
- on 3 platforms
- with no other tests running, if at all possible, because of the false-failure crap
- as soon as possible, incrementally if there is a research problem to solve with respect to running a given test suite under automation
- don't care how it's reported, as long as we can see "checkins in the delta that caused a test failure" and the new failures are easy to see; tinderbox, bb waterfall, twitter messages, a drum circle on speakerphone

Discussion of how to integrate the js/tests stuff into mozilla-central's test payload is not this bug, and should not be.
I can setup mochitest fairly quickly, given than it's self-contained and already runs on mozilla-central. There are a few questions about how much work Bob needs to complete to get the JS tests running in this environment. The JS test suite, as it exists now under automation on 1.9.0 handles the entire process from checkout to build and will need to be modified to work with hg. Fitting mochitest into *that* hasn't been done before, but should be easy enough, but will need some minor tweaks to get it right. I don't know how long this will take to setup or how long Bob needs to get it working.

This isn't bike-shedding. This is trying to intelligently scope a problem which is being called an immediate requirement. Mike Shaver may not care how it's reported, but I'm sure there will be people who *will* care and we'll be left with finding some mechanism for that. Sparkles and all.
(In reply to comment #12)
> I can setup mochitest fairly quickly, given than it's self-contained and
> already runs on mozilla-central.

That is a good first step.

We didn't initially plan on tracemonkey existing, it has ended up with a lot of gnarly integration work, pulling pieces of tamarin-tracing and mozilla-central. Test coverage of any kind is really needed, so the work can proceed at a fast pace. 

The urgency is rooted in the fact that a few lost days and weeks could slip tracemonkey later than we would like, and tracemonkey is really great. I hope that makes sense.

So, sooner is better, and something is better than nothing. Just getting one platform building + mochitest this week would be awesome.
fwiw, the sisyphus/js tests can run from hg.mozilla.org repositories. Unfortunately it is "different" from the normal test environments (since it predates them all), especially with respect to how it builds. I am in the final stages of documenting all of it and we can decide how to integrate it with unittests and end of life it as people desire. Lets do mochikit first and decide what to do with the js tests in the next day or so. I'll post to npm.quality when the docs are ready and you all can rip it up.
(In reply to comment #12)
> This isn't bike-shedding. This is trying to intelligently scope a problem which
> is being called an immediate requirement. Mike Shaver may not care how it's
> reported, but I'm sure there will be people who *will* care and we'll be left
> with finding some mechanism for that. Sparkles and all.

I didn't mean to say that you were bike-shedding; I was saying that I didn't care about the colour of the shed (whether it was fully integrated or two entirely different test systems, f.e).  I wanted to clarify what the requirements are here, since it is not a requirement of mine or sayrer's (AFAIK) that these tests be integrated fully into the m-c mix, or that we run the full m-c test setup.  If that's the fastest or best way to get it going, then we can hold our breath on the full-suite flakiness even.
I though that mozilla-central build infra could pull from multiple branches if the build requirements were the same?   E.g. we get slower overall throughput but just share the same infra...
(In reply to comment #16)
> I though that mozilla-central build infra could pull from multiple branches if
> the build requirements were the same?   E.g. we get slower overall throughput
> but just share the same infra...
> 

For traditional "build" things (dep builds, nightlies, leak test) this is the case. For Talos and Unit test it is not.
Specifically regarding dep/nightly/leak test: In the future we should be able to bring this up faster. The two releases going on are taking priority right now, and since this is the first branch we've added after-the-fact I'd like to test it out on staging before touching the production setup.
I'm testing this on staging today. A couple of questions:
* Do we want PGO builds on win32?
* What Tinderbox tree should they be on?
* "with PGO", pls: we're going to be messing with native code, so we want to make sure that we're onside with more aggressive optimizations
* "MozillaTest" is fine

Thanks!
Assignee: nobody → bhearsum
Priority: -- → P2
Status: NEW → ASSIGNED
Ok, I've done test builds on all three platforms. They've all failed with the same error:
nanojit.h(55) : fatal error C1189: #error :  "unknown nanojit architecture"

According to shaver, Mac builds should work if we don't do a UB, just i386. From a build perspective it's easier to do UB builds, but if we have to do i386-only so be it.

If we're doing UB builds I can add this to production very soon (today/tomorrow), albeit all builds will be busted.

If we're doing i386-only I need a little bit more time to a) test a build on one of the slaves, and b) tweak the configs to allow branch-specific mozconfigs.
We're not going to have PPC support any time very soon, so x86-only would be the way to go.  We can fix the Win/Lin builds more easily, though, so maybe go ahead and push to production, and then we can push the PPC-ectomy build stuff in a little while?
Sure, we can do that.
This is almost identical to the changeset I pushed for staging (http://hg.mozilla.org/build/buildbot-configs/index.cgi/rev/989d6f7a84b4). This one has PGO flipped on for win32.
Attachment #328498 - Flags: review?(ccooper)
Attachment #328498 - Flags: review?(ccooper) → review+
Comment on attachment 328498 [details] [diff] [review]
[checked in] turn on tracemonkey in production

landed in changeset:   127:a18b2a1c6575
Attachment #328498 - Attachment description: turn on tracemonkey in production → [checked in] turn on tracemonkey in production
buildbot master has been updated, builds are going now
Status: ASSIGNED → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
I _think_ this isn't running any tests?

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1215738455.1215738755.26565.gz&fulltext=1

or am I looking in the wrong place for the test results?
Codesighs is running on that builder (as well as the Mac one). What tests are you looking for?
We're looking for mochitests and the JS test suite; I can open specific bugs on those if that's the more righteous path.  At some point we're gonna want Tss for sure. :)
cc'ing alice, re: comment #29
Talos set up should be filed separately (Release Eng: Talos) covering:

- desired operating systems
- desired tests

It would also be good to have an idea of priority in terms of amount of machines to throw at this, how soon we want them up and how important it is to get this coverage.  That way we can balance it against other pending talos requests.
OK, so what about the mochitests and JS test suite from comment 0?  What do I have to do to get those turned on and reporting?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
1) Tracemonkey dep.builds and nightly builds are automated and are currently posted to MozillaTest. Should these builds be posted to mozilla-central waterfall, or a new tracemonkey waterfall? 

2) Mac tracemonkey builds have been failing as far back as waterfall remembers. Is tracemonkey supposed to build on mac or is this known bustage?

3) I've filed a separate bug#446101 to handle setting up unittests for tracemonkey.
Blocks: 446101
1) I think it's fine to leave them on MozillaTest; I certainly wouldn't want 5 seconds to be spent moving them that could be spent actually getting the tests from comment 0 running and reporting somewhere.

2) We've certainly been green before, though I haven't been following the waterfall in the past week or so because if we don't have it running tests it doesn't matter much whether it builds or not -- the tests help us identify what change we need address to fix a regression, but "doesn't build" regressions are trivial to address.  I'll take a look, might be a gcc-4.2 dep that snuck in.

3) Should we have filed this bug in split form originally?  The whole point of the bug was to get this test coverage, but if we need to do it in multiple steps for future trees people can be told to file a dep chain, I suppose.  If that bug is open, though, should this one be closed FIXED, in spite of comment 0's clear description not being addressed?
(In reply to comment #34)
> 2) We've certainly been green before, though I haven't been following the
> waterfall in the past week or so because if we don't have it running tests it
> doesn't matter much whether it builds or not -- the tests help us identify what
> change we need address to fix a regression, but "doesn't build" regressions are
> trivial to address.  I'll take a look, might be a gcc-4.2 dep that snuck in.
Thanks for fixing that, builds are now green.



> 3) Should we have filed this bug in split form originally?  The whole point of
> the bug was to get this test coverage, but if we need to do it in multiple
> steps for future trees people can be told to file a dep chain, I suppose.  If
> that bug is open, though, should this one be closed FIXED, in spite of comment
> 0's clear description not being addressed?

hi Shaver;

No, filing the one bug like you did was the right thing to do. During our triage, we should have filed the 2nd bug when we were triaging your bug. That was my snafu, and I'm sorry.

Its our internal RelEng problem that we've got two different infrastructures, one for doing the builds, and a different one for doing unittests. There's different networks, user accounts, buildbot custom classes and overall system configs. And different people who know how to get things working on each infrastructure. :-( We're still streamlining all that, but its not finished yet. Once thats resolved, there would be no need for a 2nd bug anyway - we'd simply enable unittest jobs on new branches go to the slave pool, just like we already can enable build jobs on new branches go to the slave pool. And one person could do it all, from one bug. 

From offline IRC, this tracemonkey project branch/repo is only going to be around for another couple of weeks, so its fine to leave it reporting on MozillaTest.

There's nothing left to do in this bug, so closing. Remaining work is in bug#446101 to setup unittests.

Sorry again
John.
Status: REOPENED → RESOLVED
Closed: 16 years ago16 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: