It should be possible to disable jetpack tests via a manifest

RESOLVED FIXED

Status

P1
normal
RESOLVED FIXED
5 years ago
4 years ago

People

(Reporter: emorley, Assigned: evold)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

5 years ago
At the moment to disable a jetpack test (in the case of high intermittent failure rate), the disabling has to occur upstream and then another import of jetpack's repo performed.

We really need a disabled tests manifest in mozilla-central (outside of the imported source directory), so sheriffs can disable misbehaving tests (eg bug 882867) without having to resort to upstream changes (or making changes to mozilla-central that will just get overwritten on next import).

ie jetpack only partially adheres to:
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#9.29_Supports_the_disabling_of_individual_tests
(Reporter)

Updated

5 years ago
Blocks: 629263

Updated

5 years ago
Priority: -- → P1
(Reporter)

Updated

4 years ago
Blocks: 1020473
Depends on: 1035512
Assignee: nobody → evold
Blocks: 1117922
Irakli what do you propose here?

I was suggesting using a `jetpack-test-options.json` file in bug 915376 https://github.com/mozilla/addon-sdk/pull/1241, which would have covered this, but you vetoed that and implemented preferences instead because you said you didn't want to see extra files in the add-on, so I'm not sure what you want to see here.
Flags: needinfo?(rFobic)
Note that the manifests added in bug 1035512 fix this when we turn those tests on.
Looks like we have solution as suggested by mossop above.
Flags: needinfo?(rFobic)
(In reply to Ed Morley [:edmorley] from comment #0)
> We really need a disabled tests manifest in mozilla-central (outside of the
> imported source directory), 

is this "outside of the imported source directory" part really necessary?  we uplift changes from m-c to github.
Flags: needinfo?(emorley)
(Reporter)

Comment 5

4 years ago
The reasoning being, that people can disable problematic tests immediately (and in a repo they gave access to/checked out locally), rather than needing to wait for the next uplift.

That said, a possible workflow might be: disable test in mozilla-central's copy of the addonsdk directory manifest, but also make sure someone makes the equivalent change in the external repo, so the change doesn't get clobbered on next uplift.

Either way, I'm happy to defer this call to RyanVM now, since I'm now on the treeherder team, rather than the sheriffs :-)
Flags: needinfo?(emorley)
(In reply to Ed Morley [:edmorley] from comment #5)
> That said, a possible workflow might be: disable test in mozilla-central's
> copy of the addonsdk directory manifest, but also make sure someone makes
> the equivalent change in the external repo, so the change doesn't get
> clobbered on next uplift.

This is what we already do as Erik said.
(Reporter)

Comment 7

4 years ago
(In reply to Dave Townsend [:mossop] from comment #6)
> (In reply to Ed Morley [:edmorley] from comment #5)
> > That said, a possible workflow might be: disable test in mozilla-central's
> > copy of the addonsdk directory manifest, but also make sure someone makes
> > the equivalent change in the external repo, so the change doesn't get
> > clobbered on next uplift.
> 
> This is what we already do as Erik said.

That's not how comment 4 read to me, the key part is the addonsdk team being happy for the simultaneous landing both in both repos, not that things are just uplifted. If that's already happening, great :-) 

Either way I don't really have a horse in this race any more, I'm no longer a sheriff
(Reporter)

Updated

4 years ago
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → INCOMPLETE
we should still complete this bug, it seems like we don't have any requirement on where the file is now though.
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Summary: It should be possible to disable jetpack tests via a manifest in mozilla-central and not just upstream changes → It should be possible to disable jetpack tests via a manifest
Now mochitest-jetpack is running the tests can be disabled via the manifest so I'd be inclined to close this as fixed and open other bugs if other manifests for jpm/travis are needed. Make sense Erik?
Flags: needinfo?(evold)
(In reply to Dave Townsend [:mossop] from comment #9)
> Now mochitest-jetpack is running the tests can be disabled via the manifest
> so I'd be inclined to close this as fixed and open other bugs if other
> manifests for jpm/travis are needed. Make sense Erik?

Sure that sounds fine to me.
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Flags: needinfo?(evold)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.