1206203 - ./mach crashtest and reftest fail with "ImportError: No module named fix_linux_stack"

Status: RESOLVED FIXED

(Testing :: Reftest, defect)

Description

[Daniel Holbert [:dholbert]]

My up-to-date mozilla-inbound clone is giving me this output when I try to run "./mach crashtest" or "./mach reftest" right now:

==========
$ ./mach crashtest
 0:00.15 Checking for orphan ssltunnel processes...
 0:00.21 Checking for orphan xpcshell processes...
 0:00.33 REFTEST REFTEST INFO | runreftest.py | Running tests: start.

Error running mach:

    ['crashtest']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

ImportError: No module named fix_linux_stack

  File "/scratch/work/builds/mozilla-inbound/mozilla/layout/tools/reftest/mach_commands.py", line 267, in run_crashtest
    return self._run_reftest(**kwargs)
  File "/scratch/work/builds/mozilla-inbound/mozilla/layout/tools/reftest/mach_commands.py", line 281, in _run_reftest
    return reftest.run_desktop_test(**kwargs)
  File "/scratch/work/builds/mozilla-inbound/mozilla/layout/tools/reftest/mach_commands.py", line 213, in run_desktop_test
    rv = runreftest.run(**kwargs)
  File "/scratch/work/builds/mozilla-inbound/mozilla/layout/tools/reftest/runreftest.py", line 719, in run
    return reftest.runTests(options.tests, options)
  File "/scratch/work/builds/mozilla-inbound/mozilla/layout/tools/reftest/runreftest.py", line 400, in runTests
    return self.runSerialTests(manifests, options, cmdlineArgs)
  File "/scratch/work/builds/mozilla-inbound/mozilla/layout/tools/reftest/runreftest.py", line 679, in runSerialTests
    debuggerInfo=debuggerInfo)
  File "/scratch/work/builds/mozilla-inbound/mozilla/layout/tools/reftest/runreftest.py", line 603, in runApp
    dump_screen_on_timeout=not debuggerInfo)
  File "/scratch/work/builds/mozilla-inbound/mozilla/layout/tools/reftest/runreftest.py", line 537, in __init__
    self.stack_fixer_function = self.stack_fixer()
  File "/scratch/work/builds/mozilla-inbound/mozilla/layout/tools/reftest/runreftest.py", line 558, in stack_fixer
    return get_stack_fixer_function(self.utilityPath, self.symbolsPath)
  File "/scratch/work/builds/mozilla-inbound/mozilla/testing/mozbase/mozrunner/mozrunner/utils.py", line 289, in get_stack_fixer_function
    'fix_linux_stack')
  File "/scratch/work/builds/mozilla-inbound/mozilla/testing/mozbase/mozrunner/mozrunner/utils.py", line 262, in import_stack_fixer_module
    module = __import__(module_name, globals(), locals(), [])
  File "/scratch/work/builds/mozilla-inbound/mozilla/build/mach_bootstrap.py", line 358, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
==========

Not sure when this started happening. I returned from a several-week vacation this past Monday, and I don't think I've run mach crashtests or reftests on this particular machine since I've been back, so this might be something that broke anytime in the past few weeks.

This is a Dell Precision T1600 desktop running 64-bit Ubuntu 15.04. My mozconfig is pretty trivial, too:

mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../obj
ac_add_options --enable-debug --disable-optimize
mk_add_options AUTOCLOBBER=1
mk_add_options MOZ_MAKE_FLAGS='-s -j14'
CC="distcc $CC"
CXX="distcc $CXX"
ac_add_options --enable-warnings-as-errors
ac_add_options --enable-profiling

Comment 1

[Daniel Holbert [:dholbert]]

I'm seeing this on all of my trees, on two different machines (the other one is a 64-bit linux laptop).

Comment 2

[Daniel Holbert [:dholbert]]

froydnj suggests (and I agree) this is likely to be a regression from bug 1196430, which modified e.g. stack_fixer (mentioned in the backtrace here) in this commit:
  http://hg.mozilla.org/mozilla-central/rev/95b4e5d5b14c#l2.29

(Not sure if that's exactly the regressing commit; there were other related commits there as well.)

Comment 3

[Daniel Holbert [:dholbert]]

(In reply to Daniel Holbert [:dholbert] from comment #2)
> froydnj suggests (and I agree) this is likely to be a regression from bug 1196430

Actually, local testing suggests this didn't break until more recently. I'm using "hg bisect" to track down exactly when it broke right now, stay tuned...

Comment 4

[Daniel Holbert [:dholbert]]

Current regression range from hg bisect (still narrowing locally, slowly) is:
https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=9efbf9792f34&tochange=655a465bcc04

jgraham has several reftest-harness changes in that range, so right now my suspicion is leaning in his direction. --> CC'd.

Comment 5

[Daniel Holbert [:dholbert]]

My hg bisect testing says:
{
The first bad revision is:
changeset:   262976:967ad761ec8c
user:        James Graham <[e-mail]>
date:        Tue Aug 25 14:07:23 2015 +0100
summary:     Bug 1181516 - Allow reftests to take paths to multiple directories containing tests on the command line, r=jmaher
}

Comment 6

[Daniel Holbert [:dholbert]]

And just as a sanity-check that I didn't mess up during 'hg bisect', I verified that this runs correctly:
  hg up -r 87bdb5f6b1e6 # parent of regressing changeset
  ./mach build
  ./mach crashtest

...whereas this triggers the error from comment 0:
  hg up -r 967ad761ec8c # regressing changeset
  ./mach build
  ./mach crashtest

So, 967ad761ec8c is definitely the first bad revision.

Comment 7

[James Graham [:jgraham]]

I'm pretty sure that http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/reftestcommandline.py#88 is wrong and should be default=build_obj.bindir or similar, but I don't have a working build to test on at the moment (and obviously this was working for me in general so I don't know that I could trivially reproduce). Can you test for me?

Comment 8

[Daniel Holbert [:dholbert]]

That doesn't seem to help; with that change, I still get the same exception.

Comment 9

[Daniel Holbert [:dholbert]]

Attachment: (patch tested for comment 7 which didn't help)

(For the record, here's a patch showing the change that I tested.)

Comment 10

[James Graham [:jgraham]]

OK, I'll have a deeper look then.

Comment 11

[James Graham [:jgraham]]

Oh, that shouldn't have been a string i.e. it should have been literally

build_obj.bindir

not

"build_obj.bindir"

(I still haven't actually tested anything).

Comment 12

[Daniel Holbert [:dholbert]]

(In reply to James Graham [:jgraham] from comment #11)
> Oh, that shouldn't have been a string i.e. it should have been literally
> 
> build_obj.bindir

That doesn't quite work, but self.build_obj.bindir seems to work!

Comment 13

[Daniel Holbert [:dholbert]]

Attachment: fix v1

In the interests of getting this fixed ASAP so others don't trip over it, I'm posting the patch that I've got locally, which is based on jgraham's suggestion [which I don't fully understand but have verified fixes the issue], for jgraham to r+.

I think this is decently kosher, given the trivialness of the change and the severity of the bustage. (primary testsuites not working)

Comment 14

[James Graham [:jgraham]]

Comment on attachment 8663232 [details] [diff] [review]
fix v1

Review of attachment 8663232 [details] [diff] [review]:
-----------------------------------------------------------------

::: layout/tools/reftest/reftestcommandline.py
@@ +84,5 @@
>          self.add_argument("--utility-path",
>                            action="store",
>                            type=str,
>                            dest="utilityPath",
> +                          default=self.build_obj.bindir,

self.buid_obj can also be None (e.g. in automation), so I think the full change that will work is

self.build_obj.bindir if self.build_obj else None

Sorry for taking so long to get to the right fix.

Comment 15

[Daniel Holbert [:dholbert]]

That makes sense -- though FWIW, if I *explicitly* specify None (without if/else), just to test what happens if we hit that "None" fallback case in your suggestion, I get the following failure output:
{
AttributeError: 'NoneType' object has no attribute 'startswith'

  File "/scratch/work/builds/mozilla-inbound/mozilla/layout/tools/reftest/mach_commands.py", line 267, in run_crashtest
    return self._run_reftest(**kwargs)
  File "/scratch/work/builds/mozilla-inbound/mozilla/layout/tools/reftest/mach_commands.py", line 281, in _run_reftest
    return reftest.run_desktop_test(**kwargs)
  File "/scratch/work/builds/mozilla-inbound/mozilla/layout/tools/reftest/mach_commands.py", line 213, in run_desktop_test
    rv = runreftest.run(**kwargs)
  File "/scratch/work/builds/mozilla-inbound/mozilla/layout/tools/reftest/runreftest.py", line 718, in run
    parser.validate(options, reftest)
  File "/scratch/work/builds/mozilla-inbound/mozilla/layout/tools/reftest/reftestcommandline.py", line 349, in validate
    options.utilityPath = reftest.getFullPath(options.utilityPath)
  File "/scratch/work/builds/mozilla-inbound/mozilla/layout/tools/reftest/runreftest.py", line 225, in getFullPath
    return os.path.normpath(os.path.join(self.oldcwd, os.path.expanduser(path)))
  File "/usr/lib/python2.7/posixpath.py", line 254, in expanduser
    if not path.startswith('~'):
}

So it seems like the "None" fallback may not actually help much here. (Though maybe this is just a default that would hopefully be overridden in that case.)

But anyway, it's better than what we've got currently, so I'll proceed with your suggestion from comment 14. Thanks! Try run:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=2fe3ab90337f

Comment 16

[James Graham [:jgraham]]

Oh, I think we get away with it in automation by providing the path explicitly. It still seems worthwhile to fix that case to at least not crash in a lower priority followup though.

(on an unrelated note, this try run probably ought to have been limited to reftest-harness using suites to save us some machine time/money and help with the try backlog).

Comment 17

[Daniel Holbert [:dholbert]]

(Yup, I'm normally pretty conservative with Try resources, but I get a little uneasy touching python files that are used in automation, and didn't want to trust myself to guess exactly which platforms/harnesses use this file.)

Comment 18

[Daniel Holbert [:dholbert]]

Landed over the weekend:
 https://hg.mozilla.org/integration/mozilla-inbound/rev/b275a63010de
 https://hg.mozilla.org/mozilla-central/rev/b275a63010de

I had two numbers swapped in the bug number though, so it actually got posted on bug 1206230. (I left a comment over there linking here.)

Comment 19

[James Graham [:jgraham]]

Thanks for fixing this.

Comment 20

[Daniel Holbert [:dholbert]]

Sure! Thanks for figuring out what needed fixing.