1289124 - Autophone should skip missing tests

Status: RESOLVED WONTFIX

(Testing Graveyard :: Autophone, defect, P3)

Description

[Dan Minor [:dminor]]

Twice now I've been bitten by tests that are removed from dom/media/tests but are still listed in the Autophone WebRTC job manifest, which seems to result in the job hanging until it is force killed (see e.g. https://autophone.s3.amazonaws.com/pub/mobile/tinderbox-builds/mozilla-central-android-api-15/1469110938/mochitest-webrtc-mochitests-webrtc.ini-1-nexus-4-7-178802e9-aa97-4696-a598-737c499578e8.log)

It would be nice if the missing tests were skipped with a warning instead rather than wasting time waiting to be killed.

Comment 1

[Geoff Brown [:gbrown]]

|mach mochitest| detects and reports such cases:

gbrown@mozpad:~/src$ ./mach mochitest testing/mochitest/tests/Harness_sanity
Build configuration changed. Regenerating backend.
Reticulating splines...
Traceback (most recent call last):
  File "config.status", line 972, in <module>
    config_status(**args)
  File "/home/gbrown/src/python/mozbuild/mozbuild/config_status.py", line 147, in config_status
    definitions = list(definitions)
  File "/home/gbrown/src/python/mozbuild/mozbuild/frontend/emitter.py", line 179, in emit
    objs = list(self.emit_from_context(out))
  File "/home/gbrown/src/python/mozbuild/mozbuild/frontend/emitter.py", line 959, in emit_from_context
    for obj in self._process_test_manifests(context):
  File "/home/gbrown/src/python/mozbuild/mozbuild/frontend/emitter.py", line 1082, in _process_test_manifests
    for obj in self._process_test_manifest(context, info, path, manifest):
  File "/home/gbrown/src/python/mozbuild/mozbuild/frontend/emitter.py", line 1199, in _process_test_manifest
    context)
mozbuild.frontend.reader.SandboxValidationError: 
==============================
ERROR PROCESSING MOZBUILD FILE
==============================

The error occurred while processing the following file or one of the files it includes:

    /home/gbrown/src/testing/mochitest/tests/moz.build

The error occurred when validating the result of the execution. The reported error is:

    Error processing test manifest file /home/gbrown/src/testing/mochitest/tests/Harness_sanity/mochitest.ini: Traceback (most recent call last):
    
      File "/home/gbrown/src/python/mozbuild/mozbuild/frontend/emitter.py", line 1131, in _process_test_manifest
        path, ', '.join(missing)), context)
    
    SandboxValidationError: 
    ==============================
    ERROR PROCESSING MOZBUILD FILE
    ==============================
    
    The error occurred while processing the following file or one of the files it includes:
    
        /home/gbrown/src/testing/mochitest/tests/moz.build
    
    The error occurred when validating the result of the execution. The reported error is:
    
        Test manifest (/home/gbrown/src/testing/mochitest/tests/Harness_sanity/mochitest.ini) lists test that does not exist: test_gbrown.html
    
    


make: *** [backend.RecursiveMakeBackend] Error 1
Error running mach:

    ['mochitest', 'testing/mochitest/tests/Harness_sanity']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 2: [u'/usr/bin/make', u'-j8', u'-s', u'run-tests-deps']

  File "/home/gbrown/src/testing/mochitest/mach_commands.py", line 408, in run_mochitest_general
    tests = mochitest.resolve_tests(test_paths, test_objects, cwd=self._mach_context.cwd)
  File "/home/gbrown/src/testing/mochitest/mach_commands.py", line 163, in resolve_tests
    resolver = self._spawn(TestResolver)
  File "/home/gbrown/src/python/mozbuild/mozbuild/base.py", line 674, in _spawn
    topobjdir=self.topobjdir)
  File "/home/gbrown/src/python/mozbuild/mozbuild/testing.py", line 184, in __init__
    print_directory=False)
  File "/home/gbrown/src/python/mozbuild/mozbuild/base.py", line 602, in _run_make
    return fn(**params)
  File "/home/gbrown/src/python/mozbuild/mozbuild/base.py", line 657, in _run_command_in_objdir
    return self.run_process(cwd=self.topobjdir, **args)
  File "/home/gbrown/src/python/mach/mach/mixin/process.py", line 147, in run_process
    raise Exception('Process executed with non-0 exit code %d: %s' % (status, args))

Comment 2

[Bob Clary [:bc] (inactive)]

Autophone is going away. Resolving these to wontfix.