Closed Bug 650646 Opened 9 years ago Closed 9 years ago

Full error stack messages in frame are not exposed anymore due to non-enumerable Error properties

Categories

(Testing Graveyard :: Mozmill, defect)

defect
Not set
critical

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: aaronmt, Assigned: harth)

References

Details

(Keywords: regression, Whiteboard: [mozmill-2.0+][mozmill-next?][mozmill-1.5.4+])

Attachments

(1 file)

In debugging a test this morning, I noticed that my install of Mozmill 1.5.3 was not exposing fully detailed errors (regardless of type of issue) to console and regardless of the flags --show-errors and --show-all, no line numbers of issue, etc. 

Output: ERROR | Test Failure: {"exception": {}}

I did a full cleanup and install of Mozmill 1.5.3

This commit looks suspicious 

https://github.com/mozautomation/mozmill/commit/924ae27401c0bdf70b68b75671be2a7951f81840 -- did this land in 1.5.3?
Summary: Full error stack messages in frame are not exposed anymore → Full error stack messages in frame are not exposed anymore (1.5.3)
For what it's worth, after a clean 1.5.3 install, in my /Library/Python/2.6/site-packages/mozmill/extension/resource directory containing 1.5.3, I have a frame.js.orig and frame.js.rej reject file
Sounds to be the same as bug 627422.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 627422
Not entirely sure, but I can reproduce by simply changing failing any assert or creating a typo in a function call, not be changing interface names.
Can you give a testcase please?
Example:

http://hg.mozilla.org/qa/mozmill-tests/file/cfa98c5dffdf/tests/functional/testTabbedBrowsing/testNewTab.js#l89

Change to '!='

mozmill -t tests/functional/testTabbedBrowsing/testNewTab.js -b ~/Downloads/Nightly.app/ --show-errors

TEST-START | /Users/AaronMT/Downloads/mozmill-tests/tests/functional/testTabbedBrowsing/testNewTab.js | setupModule
TEST-PASS | /Users/AaronMT/Downloads/mozmill-tests/tests/functional/testTabbedBrowsing/testNewTab.js | setupModule
TEST-START | /Users/AaronMT/Downloads/mozmill-tests/tests/functional/testTabbedBrowsing/testNewTab.js | testNewTab
ERROR | Test Failure: {"exception": {}}
TEST-UNEXPECTED-FAIL | /Users/AaronMT/Downloads/mozmill-tests/tests/functional/testTabbedBrowsing/testNewTab.js | testNewTab
INFO Passed: 1
INFO Failed: 1
INFO Skipped: 0
Guess it's something with my install somewhere on Mac. I cleanly tested on Ubuntu and I can't reproduce. 

I'll post back if I find out what it is.
Note: Am able to reproduce as well in a virtualenv in Mac
Works fine for me. I would assume your Mozmill installation is broken. Some global installed files probably cause this behavior. Everything looks fine for me on OS X with 1.5.3.
Resolution: DUPLICATE → INVALID
yolk -l output under a virtualenv with --no-site-packages

(ENV)AaronMT@Aaron-Trains-MacBook-Pro~/Desktop/ENV: yolk -l
ManifestDestiny - 0.2.2        - active 
Python          - 2.6.1        - active development (/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload)
jsbridge        - 2.4.3        - active 
mozmill         - 1.5.3        - active 
mozrunner       - 2.5.4        - active 
pip             - 1.0          - active 
setuptools      - 0.6c11       - active 
wsgiref         - 0.1.2        - active development (/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6)
yolk            - 0.4.1        - active
I can reproduce this 100% in the latest nightly and mozmill master.  I'm getting an empty 'Exception' no matter what error happens in the test.

The odd thing is that it works normally in Fx 3.6 but fails in the latest nightly.  I'll try to find a better regression range in Firefox.
Status: RESOLVED → REOPENED
Resolution: INVALID → ---
So this also works fine in fx4 and the latest Aurora. It looks like something landed in the nightlies around the 16th that caused this bustage.
OS: Mac OS X → All
Hardware: x86 → All
Whiteboard: [mozmill-2.0?][mozmill-next?]
Version: unspecified → Trunk
We got the regression range down to the nightly.  The last working nightly was on the 13th.  Started breaking on the 14th.

Changesets:
6bcaec19d09e - aa200a803e07
(In reply to comment #10)
> I can reproduce this 100% in the latest nightly and mozmill master.  I'm
> getting an empty 'Exception' no matter what error happens in the test.

Just as note, Aaron was testing with Mozmill 1.5.3 and not Mozmill 2.0. Whatever happened with his setup could it have mixed files from both versions? It's pretty unclear for the hotfix-1.5 state.
(In reply to comment #13)
> The offending changeset: http://hg.mozilla.org/mozilla-central/rev/9743d95d473e

As this changeset tells me this is really an issue on master only and not hotfix-1.5.

Aaron, can you really make sure that you have had mozmill 1.5.3 running and no 2.0 build?
(In reply to comment #15)
> (In reply to comment #13)
> > The offending changeset: http://hg.mozilla.org/mozilla-central/rev/9743d95d473e
> 
> As this changeset tells me this is really an issue on master only and not
> hotfix-1.5.
> 
> Aaron, can you really make sure that you have had mozmill 1.5.3 running and no
> 2.0 build?

I do not have master anywhere. In comment #9, I isolated mozmill into virtualenv with --no-site-packages, I have completely cleared all egg's and related files in /Library/Python/2.6/site-packages/, I dont know where else there could be anything.
Do we definitely have two different issues here and I think we should not use this bug for the regression we have on master. Andrew, can you please file a new bug? Thanks.
(In reply to comment #15)
> (In reply to comment #13)
> > The offending changeset: http://hg.mozilla.org/mozilla-central/rev/9743d95d473e
> 
> As this changeset tells me this is really an issue on master only and not
> hotfix-1.5.

How does it tell you that?  This is a firefox changeset.. not a mozmill one..

This should be an issue on all versions of Mozmill.. it's an intentional change in Firefox that's causing this.  Basically Errors aren't allowed to be enumerable in Firefox anymore.

We'll have to fix on master and probably backport to 1.5 as well.
(In reply to comment #18)
> > As this changeset tells me this is really an issue on master only and not
> > hotfix-1.5.
> 
> How does it tell you that?  This is a firefox changeset.. not a mozmill one..
> 
> This should be an issue on all versions of Mozmill.. it's an intentional change
> in Firefox that's causing this.  Basically Errors aren't allowed to be
> enumerable in Firefox anymore.

Our daily tests are running fine for any 6.0 build. Same locally. IMO this failure comes from subclassing the Error class which we only have on 2.0.
(In reply to comment #19)
> Our daily tests are running fine for any 6.0 build. Same locally. IMO this
> failure comes from subclassing the Error class which we only have on 2.0.

I think you don't get the point of this bug... it's not that something is causing a failure.. it's that *if* there is a failure no information is getting dumped out.  Try modifying one of your tests to have a syntax error, you'll see an empty exception instead of a stack trace.
Also when I say dumped out I don't mean in reporting, I mean in the terminal.
Oh wait. Looks like when I have tested it the last time I had an outdated Nighly build which didn't contain this change. So while testing it again, I can also see this behavior. Sorry for the noise.

So this is kinda important now and would also block us from investigation failures like what we have on bug 652799. We should  try to get out 1.5.4 as soon as we can.
Blocks: 652799
Status: REOPENED → NEW
Whiteboard: [mozmill-2.0?][mozmill-next?] → [mozmill-2.0?][mozmill-next?][mozmill-1.5.4?]
Whiteboard: [mozmill-2.0?][mozmill-next?][mozmill-1.5.4?] → [mozmill-2.0+][mozmill-next?][mozmill-1.5.4+]
Assignee: nobody → fayearthur+bugs
Heather, were you able to look into this? It seems like there's a bunch of things relying on a hotfix 1.5.4 at the moment
Probably not until next week.
What we decided on though was creating a MozmillError class that inherits from Error and makes 'message', 'stack', etc. enumerable again.
(In reply to comment #24)
> Probably not until next week.

Clint, any change to get someone else working on it? We are blocked on getting our broken tests we currently have for Firefox 6 fixed.
(In reply to comment #26)
> (In reply to comment #24)
> > Probably not until next week.
> 
> Clint, any change to get someone else working on it? We are blocked on getting
> our broken tests we currently have for Firefox 6 fixed.

Jhammel is sick, Harth's out the rest of the week.  And that leaves either you (who is traveling) or me.  If I can get to it this week I will.  It's on the list.
Ok, so I will be back later today. If you have time it would be really appreciated. Otherwise I will help out later today - even when I will be totally tired.
Severity: normal → major
We are getting more and more test failures on trunk which we can't investigate due to this problem.

Heather, can you please give us an ETA when this issue will be fixed? We really need this fix ASAP.
Severity: major → critical
Okay, I tried making a new MozmillError class inheriting from Error, but I couldn't figure out how to make it not report the fileName/lineNumber as errors.js:4 (where I defined the MozmillError class and assigned the prototype to new Error()).

I ran through a lot of possibilities, but came to the conclusion that the best way to approach this is really to just check for Errors right before they're reported. This is an easy, one-place fix. Might be hacky, but errors will be reported from the mozmill file they were thrown from.
Attachment #531850 - Flags: review?(ctalbert)
Comment on attachment 531850 [details] [diff] [review]
reattach exception properties before reporting

Awesome Harth!  This works great, tested on Master. Can I get a patch for 1.5 as well?
Attachment #531850 - Flags: review?(ctalbert) → review+
Blocks: 637207
Summary: Full error stack messages in frame are not exposed anymore (1.5.3) → Full error stack messages in frame are not exposed anymore due to non-enumerable Error properties
Verified fixed with 1.5.4pre

Thanks Heather
Status: RESOLVED → VERIFIED
I think this needs to be back-ported to 1.4.2b1 because this is used on the try servers/builds of Thunderbird. See e.g. http://tinderbox.mozilla.org/showlog.cgi?log=ThunderbirdTry/1305741961.1305742742.23061.gz&fulltext=1

I patched my local installation of mozmill and it worked as expected.

(And then the update has to be installed on them, at least if they use mozilla-central)
Standard9 said on IRC that Thunderbird (tryservers) will be updated to mozmill 1.5 series, probably within the next two weeks. So I guess the backport is not necessary.
Ok cool.
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.