650646 - Full error stack messages in frame are not exposed anymore due to non-enumerable Error properties

Status: VERIFIED FIXED

(Testing Graveyard :: Mozmill, defect)

Description

[Aaron Train [:aaronmt]]

In debugging a test this morning, I noticed that my install of Mozmill 1.5.3 was not exposing fully detailed errors (regardless of type of issue) to console and regardless of the flags --show-errors and --show-all, no line numbers of issue, etc. 

Output: ERROR | Test Failure: {"exception": {}}

I did a full cleanup and install of Mozmill 1.5.3

This commit looks suspicious 

https://github.com/mozautomation/mozmill/commit/924ae27401c0bdf70b68b75671be2a7951f81840 -- did this land in 1.5.3?

Comment 1

[Aaron Train [:aaronmt]]

For what it's worth, after a clean 1.5.3 install, in my /Library/Python/2.6/site-packages/mozmill/extension/resource directory containing 1.5.3, I have a frame.js.orig and frame.js.rej reject file

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Sounds to be the same as bug 627422.

Comment 3

[Aaron Train [:aaronmt]]

Not entirely sure, but I can reproduce by simply changing failing any assert or creating a typo in a function call, not be changing interface names.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Can you give a testcase please?

Comment 5

[Aaron Train [:aaronmt]]

Example:

http://hg.mozilla.org/qa/mozmill-tests/file/cfa98c5dffdf/tests/functional/testTabbedBrowsing/testNewTab.js#l89

Change to '!='

mozmill -t tests/functional/testTabbedBrowsing/testNewTab.js -b ~/Downloads/Nightly.app/ --show-errors

TEST-START | /Users/AaronMT/Downloads/mozmill-tests/tests/functional/testTabbedBrowsing/testNewTab.js | setupModule
TEST-PASS | /Users/AaronMT/Downloads/mozmill-tests/tests/functional/testTabbedBrowsing/testNewTab.js | setupModule
TEST-START | /Users/AaronMT/Downloads/mozmill-tests/tests/functional/testTabbedBrowsing/testNewTab.js | testNewTab
ERROR | Test Failure: {"exception": {}}
TEST-UNEXPECTED-FAIL | /Users/AaronMT/Downloads/mozmill-tests/tests/functional/testTabbedBrowsing/testNewTab.js | testNewTab
INFO Passed: 1
INFO Failed: 1
INFO Skipped: 0

Comment 6

[Aaron Train [:aaronmt]]

Guess it's something with my install somewhere on Mac. I cleanly tested on Ubuntu and I can't reproduce. 

I'll post back if I find out what it is.

Comment 7

[Aaron Train [:aaronmt]]

Note: Am able to reproduce as well in a virtualenv in Mac

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Works fine for me. I would assume your Mozmill installation is broken. Some global installed files probably cause this behavior. Everything looks fine for me on OS X with 1.5.3.

Comment 9

[Aaron Train [:aaronmt]]

yolk -l output under a virtualenv with --no-site-packages

(ENV)AaronMT@Aaron-Trains-MacBook-Pro~/Desktop/ENV: yolk -l
ManifestDestiny - 0.2.2        - active 
Python          - 2.6.1        - active development (/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload)
jsbridge        - 2.4.3        - active 
mozmill         - 1.5.3        - active 
mozrunner       - 2.5.4        - active 
pip             - 1.0          - active 
setuptools      - 0.6c11       - active 
wsgiref         - 0.1.2        - active development (/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6)
yolk            - 0.4.1        - active

Comment 10

[Andrew Halberstadt [:ahal]]

I can reproduce this 100% in the latest nightly and mozmill master.  I'm getting an empty 'Exception' no matter what error happens in the test.

The odd thing is that it works normally in Fx 3.6 but fails in the latest nightly.  I'll try to find a better regression range in Firefox.

Comment 11

[Andrew Halberstadt [:ahal]]

So this also works fine in fx4 and the latest Aurora. It looks like something landed in the nightlies around the 16th that caused this bustage.

Comment 12

[Andrew Halberstadt [:ahal]]

We got the regression range down to the nightly.  The last working nightly was on the 13th.  Started breaking on the 14th.

Changesets:
6bcaec19d09e - aa200a803e07

Comment 13

[Andrew Halberstadt [:ahal]]

The offending changeset: http://hg.mozilla.org/mozilla-central/rev/9743d95d473e

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #10)
> I can reproduce this 100% in the latest nightly and mozmill master.  I'm
> getting an empty 'Exception' no matter what error happens in the test.

Just as note, Aaron was testing with Mozmill 1.5.3 and not Mozmill 2.0. Whatever happened with his setup could it have mixed files from both versions? It's pretty unclear for the hotfix-1.5 state.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #13)
> The offending changeset: http://hg.mozilla.org/mozilla-central/rev/9743d95d473e

As this changeset tells me this is really an issue on master only and not hotfix-1.5.

Aaron, can you really make sure that you have had mozmill 1.5.3 running and no 2.0 build?

Comment 16

[Aaron Train [:aaronmt]]

(In reply to comment #15)
> (In reply to comment #13)
> > The offending changeset: http://hg.mozilla.org/mozilla-central/rev/9743d95d473e
> 
> As this changeset tells me this is really an issue on master only and not
> hotfix-1.5.
> 
> Aaron, can you really make sure that you have had mozmill 1.5.3 running and no
> 2.0 build?

I do not have master anywhere. In comment #9, I isolated mozmill into virtualenv with --no-site-packages, I have completely cleared all egg's and related files in /Library/Python/2.6/site-packages/, I dont know where else there could be anything.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Do we definitely have two different issues here and I think we should not use this bug for the regression we have on master. Andrew, can you please file a new bug? Thanks.

Comment 18

[Andrew Halberstadt [:ahal]]

(In reply to comment #15)
> (In reply to comment #13)
> > The offending changeset: http://hg.mozilla.org/mozilla-central/rev/9743d95d473e
> 
> As this changeset tells me this is really an issue on master only and not
> hotfix-1.5.

How does it tell you that?  This is a firefox changeset.. not a mozmill one..

This should be an issue on all versions of Mozmill.. it's an intentional change in Firefox that's causing this.  Basically Errors aren't allowed to be enumerable in Firefox anymore.

We'll have to fix on master and probably backport to 1.5 as well.

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #18)
> > As this changeset tells me this is really an issue on master only and not
> > hotfix-1.5.
> 
> How does it tell you that?  This is a firefox changeset.. not a mozmill one..
> 
> This should be an issue on all versions of Mozmill.. it's an intentional change
> in Firefox that's causing this.  Basically Errors aren't allowed to be
> enumerable in Firefox anymore.

Our daily tests are running fine for any 6.0 build. Same locally. IMO this failure comes from subclassing the Error class which we only have on 2.0.

Comment 20

[Andrew Halberstadt [:ahal]]

(In reply to comment #19)
> Our daily tests are running fine for any 6.0 build. Same locally. IMO this
> failure comes from subclassing the Error class which we only have on 2.0.

I think you don't get the point of this bug... it's not that something is causing a failure.. it's that *if* there is a failure no information is getting dumped out.  Try modifying one of your tests to have a syntax error, you'll see an empty exception instead of a stack trace.

Comment 21

[Andrew Halberstadt [:ahal]]

Also when I say dumped out I don't mean in reporting, I mean in the terminal.

Comment 22

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Oh wait. Looks like when I have tested it the last time I had an outdated Nighly build which didn't contain this change. So while testing it again, I can also see this behavior. Sorry for the noise.

So this is kinda important now and would also block us from investigation failures like what we have on bug 652799. We should  try to get out 1.5.4 as soon as we can.

Comment 23

[Aaron Train [:aaronmt]]

Heather, were you able to look into this? It seems like there's a bunch of things relying on a hotfix 1.5.4 at the moment

Comment 24

[Heather Arthur [:harth]]

Probably not until next week.

Comment 25

[Heather Arthur [:harth]]

What we decided on though was creating a MozmillError class that inherits from Error and makes 'message', 'stack', etc. enumerable again.

Comment 26

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #24)
> Probably not until next week.

Clint, any change to get someone else working on it? We are blocked on getting our broken tests we currently have for Firefox 6 fixed.

Comment 27

[cmtalbert]

(In reply to comment #26)
> (In reply to comment #24)
> > Probably not until next week.
> 
> Clint, any change to get someone else working on it? We are blocked on getting
> our broken tests we currently have for Firefox 6 fixed.

Jhammel is sick, Harth's out the rest of the week.  And that leaves either you (who is traveling) or me.  If I can get to it this week I will.  It's on the list.

Comment 28

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, so I will be back later today. If you have time it would be really appreciated. Otherwise I will help out later today - even when I will be totally tired.

Comment 29

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We are getting more and more test failures on trunk which we can't investigate due to this problem.

Heather, can you please give us an ETA when this issue will be fixed? We really need this fix ASAP.

Comment 30

[Heather Arthur [:harth]]

Attachment: reattach exception properties before reporting

Okay, I tried making a new MozmillError class inheriting from Error, but I couldn't figure out how to make it not report the fileName/lineNumber as errors.js:4 (where I defined the MozmillError class and assigned the prototype to new Error()).

I ran through a lot of possibilities, but came to the conclusion that the best way to approach this is really to just check for Errors right before they're reported. This is an easy, one-place fix. Might be hacky, but errors will be reported from the mozmill file they were thrown from.

Comment 31

[cmtalbert]

Comment on attachment 531850 [details] [diff] [review]
reattach exception properties before reporting

Awesome Harth!  This works great, tested on Master. Can I get a patch for 1.5 as well?

Comment 32

[Heather Arthur [:harth]]

master:
https://github.com/mozautomation/mozmill/commit/198251db6273e5ec0d4b6dbd425c0519a9ab98ac

hotfix-1.5:
https://github.com/mozautomation/mozmill/commit/a139b30524b271510b9a6e523975de33e52f9614

Comment 33

[Aaron Train [:aaronmt]]

Verified fixed with 1.5.4pre

Thanks Heather

Comment 34

[Joachim Herb]

I think this needs to be back-ported to 1.4.2b1 because this is used on the try servers/builds of Thunderbird. See e.g. http://tinderbox.mozilla.org/showlog.cgi?log=ThunderbirdTry/1305741961.1305742742.23061.gz&fulltext=1

I patched my local installation of mozmill and it worked as expected.

(And then the update has to be installed on them, at least if they use mozilla-central)

Comment 35

[Joachim Herb]

Standard9 said on IRC that Thunderbird (tryservers) will be updated to mozmill 1.5 series, probably within the next two weeks. So I guess the backport is not necessary.

Comment 36

[cmtalbert]

Ok cool.