1135230 - Switch mozilla-central to use mochitest-jetpack and let that ride the trains

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dave Townsend [:mossop] (PTO until July 20)]

mochitest-jetpack is now stable and passing in non-debug builds. We should turn it on on all mozilla-central based branches and let that ride the trains.

Comment 1

[Dave Townsend [:mossop] (PTO until July 20)]

Attachment: patch

Untested but this looks right to me. We should plan to hide the debug builds on treeherder for now as two bugs are causing them to file

Comment 2

[Chris AtLee [:catlee]]

Attachment: jetpack-builders.diff

Here are the differences that would happen with this patch. Look ok to you?

Comment 3

[Dave Townsend [:mossop] (PTO until July 20)]

Comment on attachment 8570461 [details]
jetpack-builders.diff

Looks like try isn't being included in the new set and I'm not sure I see why. I don't see gecko_version defined for try and items_at_least should pass then. What am I missing?

Also date seems to be excluded, I don't know what that is but I'm guessing it isn't an issue.

Comment 4

[Chris AtLee [:catlee]]

Oh, I think that's because try is already running these, so it doesn't show up in the diff.

Comment 5

[Dave Townsend [:mossop] (PTO until July 20)]

Comment on attachment 8570461 [details]
jetpack-builders.diff

Good point, yes this looks fine then (unless you care about date, I don't).

Comment 6

[Chris AtLee [:catlee]]

Comment on attachment 8567310 [details] [diff] [review]
patch

ship it!

date doesn't show up because it's pinned to version 36 right now: http://hg.mozilla.org/build/buildbot-configs/file/a4e503fcea40/mozilla/project_branches.py#l143

Comment 7

[Dave Townsend [:mossop] (PTO until July 20)]

https://hg.mozilla.org/build/buildbot-configs/rev/5b97d32617e3

Comment 8

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/5b97d32617e3

Comment 9

[Erik Vold [:erikvold] (please needinfo? me)]

I need to be able to run the old jetpack test suite still while doing uplifts, in order to make sure the uplifts are done correctly.  Since that is the only suite running all of the tests, it's also the only suite using the old loader.

So afaict this bug blocks the sdk uplift process, and we won't be able to do another uplift until this is resolved.

Comment 10

[Dave Townsend [:mossop] (PTO until July 20)]

(In reply to Erik Vold [:erikvold] (please needinfo? me) from comment #9)
> I need to be able to run the old jetpack test suite still while doing
> uplifts, in order to make sure the uplifts are done correctly.  Since that
> is the only suite running all of the tests, it's also the only suite using
> the old loader.

I've filed bug 1141767 for this.

> So afaict this bug blocks the sdk uplift process, and we won't be able to do
> another uplift until this is resolved.

I disagree with this. The addon-sdk tree runs the old tests against current fx-team, the only difference being that it uses the overloaded SDK rather than the one in Firefox itself. There should be very few cases where that misses a regression.

Comment 11

[Erik Vold [:erikvold] (please needinfo? me)]

(In reply to Dave Townsend [:mossop] from comment #10)
> (In reply to Erik Vold [:erikvold] (please needinfo? me) from comment #9)
> > So afaict this bug blocks the sdk uplift process, and we won't be able to do
> > another uplift until this is resolved.
> 
> I disagree with this. The addon-sdk tree runs the old tests against current
> fx-team, the only difference being that it uses the overloaded SDK rather
> than the one in Firefox itself. There should be very few cases where that
> misses a regression.

I haven't looked in to why, but my understanding is that the mochi-jp test suite is skipping the same module tests which our travis test suite skips due to code like https://github.com/mozilla/addon-sdk/blob/master/test/test-ui-toggle-button.js#L1380-L1384 this would suggest that the mochi-jp test suite is using a different loader than our old jp cfx based test suite, and that it is using a loader that is more similar to the one used on travis with jpm.

This is why I feel like we need to continue using the old jp cfx test suite for uplifts.

I filed bug 1141776 for tracking progress on deleting cfx.

Comment 12

[Dave Townsend [:mossop] (PTO until July 20)]

(In reply to Erik Vold [:erikvold] (please needinfo? me) from comment #11)
> (In reply to Dave Townsend [:mossop] from comment #10)
> > (In reply to Erik Vold [:erikvold] (please needinfo? me) from comment #9)
> > > So afaict this bug blocks the sdk uplift process, and we won't be able to do
> > > another uplift until this is resolved.
> > 
> > I disagree with this. The addon-sdk tree runs the old tests against current
> > fx-team, the only difference being that it uses the overloaded SDK rather
> > than the one in Firefox itself. There should be very few cases where that
> > misses a regression.
> 
> I haven't looked in to why, but my understanding is that the mochi-jp test
> suite is skipping the same module tests which our travis test suite skips
> due to code like
> https://github.com/mozilla/addon-sdk/blob/master/test/test-ui-toggle-button.
> js#L1380-L1384 this would suggest that the mochi-jp test suite is using a
> different loader than our old jp cfx based test suite, and that it is using
> a loader that is more similar to the one used on travis with jpm.

Yes but https://tbpl.mozilla.org/?tree=Jetpack still runs the old cfx suite. And the testaddons part of mochitest-jetpack still use the old loader (the add-ons are build with cfx).

> This is why I feel like we need to continue using the old jp cfx test suite
> for uplifts.

I'll get a patch up for bug 1141767 shortly