669428 - jetpack tests should be run everywhere, not just mozilla-central, mozilla-inbound and mozilla-aurora

Status: RESOLVED DUPLICATE of bug 854451

(Release Engineering :: General, defect, P2)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Right now we run jetpack tests on mozilla-central and mozilla-aurora.  However, we do not run them on mozilla-beta or mozilla-release, which means we're not testing what we actually ship, and we don't run them on any of the project branches that merge into mozilla-central, which means if they break on a project branch we'll have a huge amount of code to bisect (which really means anybody making changes that could affect jetpack should not be using project branches).

We should run these tests on beta, release, and project branches.

Comment 1

[Lukas Blakk [:lsblakk] use ?needinfo]

This makes sense, and will happen as soon as we have a bit more stability on the jetpack test suite which is currently going orange (or red) too frequently to be of much use on the more 'stable' beta/release branches.  bug 629263 is tracking the work on this.

Comment 2

[Lukas Blakk [:lsblakk] use ?needinfo]

fwiw, the suite also runs on try.

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

putting this to p4 since it's waiting for another bug to be fixed first.

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Can we at least run the jetpack tests on Inbound (hidden by default)? A big inbound -> central merge is causing compartment mismatch assertions in debug builds (bug 738956), and I can't tell what exactly caused it without seeing the jp test results on the individual pushes to the inbound branch.

Comment 5

[Phil Ringnalda (:philor)]

Sigh. Doesn't entirely depend on bug 629263, since as comment 4 demonstrates there's value in running a test even if that test doesn't result in immediate backouts, but since Armen is looking for builders to thrown under the bus to be able to add a few, I suspect it depends on bug 712244.

Comment 6

[Armen [:armenzg]]

kwierso, the code to add jetpack to inbound is here:
http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla-tests/config.py#l1170
but as philor points out I have to refactor some code to reduce the # of builders.
We are lucky enough to have few branches that have been abandoned and I will reach the project branch owners to remove them and give room to run jobs were we need them.

Comment 7

[Phil Ringnalda (:philor)]

Attachment: inbound

I keep watching as things make more room, and then things bounce because there's no room, and not having the slightest idea when I can get away with this. It's just one puny little builder per OS, maybe we can afford that this week?

Comment 8

[Phil Ringnalda (:philor)]

Comment on attachment 619349 [details] [diff] [review]
inbound

http://hg.mozilla.org/build/buildbot-configs/rev/25ea9b2c950a which was just fast enough to catch a reconfig, and we're now running it on inbound, https://tbpl.mozilla.org/?tree=Mozilla-Inbound&noignore=1&jobname=jetpack

Comment 9

[Phil Ringnalda (:philor)]

Attachment: ionmonkey

We'll be wanting to see what Ionmonkey does to Jetpack sooner than when it comes crashing into mozilla-central.

Comment 10

[Armen [:armenzg]]

Comment on attachment 656339 [details] [diff] [review]
ionmonkey

Thanks philor but I think a further discussion needs to happen or the owners of the Jetpack project to say on which branches to run it on.

Adding Jetpack to Ionmonkey to be hidden does buy much but not as much as taking machine cycles.

I just don't know how to feel about none closing tests when we are having such bad wait times.

I wonder if we had a tool that went to a development branch and triggered jobs on that branch to find a regression range. I will add it to my list of items to be discussed on our next meeting.

Comment 11

[David Anderson [:dvander] - inactive, e-mail if emergency]

If mozilla-central is running a set of tests to determine whether a changeset is valid, then I think IonMonkey should run the same tests so there are no surprises when we merge.

Comment 12

[Phil Ringnalda (:philor)]

My memory of 2010 is getting really fuzzy, but I think that the reason we run the Jetpack tests at all is because we found out far too late that Jaegermonkey, our last big JS engine rewrite before Ionmonkey, broke Jetpack.

If the problem is that Jetpack is hidden, then a) people need to get over that, Jetpack does exactly what it needs to do while hidden, the system works just fine, and hidden does not mean ignored, and b) because Ionmonkey is a project branch with a strong tolerance for visible test failures, I had no intention of hiding it there. It's a nearly-done rewrite of the JS engine, which needs to know what it will break, which is a very different situation than when an intentional change which breaks Jetpack's assumptions lands on mozilla-central/mozilla-inbound.

Comment 13

[Armen [:armenzg]]

Comment on attachment 656339 [details] [diff] [review]
ionmonkey

Thanks for clarifying and helping me understand it better.
Want to land? or want me to?

Comment 14

[Phil Ringnalda (:philor)]

If you can, that'd be great, I'm hours away from having a tree to push from, and with the number of reconfigs happening this week, I might luck into production at any time ;)

Comment 15

[Phil Ringnalda (:philor)]

Comment on attachment 656339 [details] [diff] [review]
ionmonkey

http://hg.mozilla.org/build/buildbot-configs/rev/abe15f89c3d8

Comment 16

[[e-mail] (:bhearsum)]

Made it to production today.

Comment 17

[Armen [:armenzg]]

Fine if we close it and open when we want to add more branches?

Comment 18

[Phil Ringnalda (:philor)]

Sure, if you don't mind closing it as WONTFIX, saying that even in the glorious future of infinite slaves in the cloud, we will never be willing to run jetpack tests on every tree where we run tests.

Comment 19

[Armen [:armenzg]]

I assume we would want to run them even if they are not tier1 to make Jetpack's life easier.
It will be awhile until we have enough capacity.
Another alternative would be to find a way of making Jetpack jobs to run on idle times except for current active branches. This is not a feature we have and would require lots of work but might be the best way of using resources.

Comment 20

[Chris Cooper [:coop] (he/him)]

Jetpack tests are still too expensive to run by default everywhere, but we can continue to enable them on branches where the branch owners will do something with the results. WONTFIXing for now. We can revisit in a few months when we (hopefully) have iX machines running tests.