906844 - Removing javascript error and warning filters from tests when running on tinderbox

Status: RESOLVED INCOMPLETE

(Add-on SDK Graveyard :: General, defect, P3)

Description

[Erik Vold [:erikvold] (please needinfo? me)]

I think we should remove these filters https://github.com/mozilla/addon-sdk/blob/master/lib/sdk/test/harness.js#L425-L439 because they could obviously filter out real problems instead of just annoying ones.

To me this seems like a time bomb waiting to go off.

Comment 1

[Dave Townsend [:mossop]]

The problem here is that there are a lot of errors that come out of the Firefox code that we don't have a lot of control over. Some are because test harnesses tend to do things that normally wouldn't happen, like opening and closing windows very quickly. If we took out hiding of these completely then the output that developers see would get polluted by things that are useless. We do have to be careful that we aren't hiding too much though, some of the entries in there seem very broad.

It would be great to get a list of the errors that happen during test runs right now, maybe we can work out ways around them or get bugs fixed in the platform to stop them. We may also be able to retire entries in the list that are no longer useful or make them more specific to ensure we're only hiding the single error we know is ignorable.

One thing we could also do is never hide these things when running on tinderbox so that we have as much logging as possible there where it is most important.

What do you think?

Comment 2

[Erik Vold [:erikvold] (please needinfo? me)]

(In reply to Dave Townsend (:Mossop) from comment #1)
> The problem here is that there are a lot of errors that come out of the
> Firefox code that we don't have a lot of control over. Some are because test
> harnesses tend to do things that normally wouldn't happen, like opening and
> closing windows very quickly. If we took out hiding of these completely then
> the output that developers see would get polluted by things that are
> useless. We do have to be careful that we aren't hiding too much though,
> some of the entries in there seem very broad.
> 
> It would be great to get a list of the errors that happen during test runs
> right now, maybe we can work out ways around them or get bugs fixed in the
> platform to stop them. We may also be able to retire entries in the list
> that are no longer useful or make them more specific to ensure we're only
> hiding the single error we know is ignorable.
> 
> One thing we could also do is never hide these things when running on
> tinderbox so that we have as much logging as possible there where it is most
> important.
> 
> What do you think?

Turning off the filtering for tinderbox sounds reasonable, also providing an opt-in for developers to turn this off would be enough to make me happy.

Also making bugs for the things that we can fix and getting them resolved rather than adding them to an ignore list seems most appropriate to me.

Comment 3

[Dave Townsend [:mossop]]

Ok, let's drop them from tinderbox for now. Erik if you could get some bugs on file for the problems that'd be great.

Comment 4

[BMO Automation]

Add-on SDK is no longer supported so resolving bugs as INCOMPLETE