Open Bug 1882899 Opened 1 year ago Updated 7 months ago

[perfdocs] Can't use test names with a '.'

Categories

(Testing :: Performance, defect, P3)

Product:
Testing
Component:
Performance
Type:
defect

Tracking

(Not tracked)

People

(Reporter: kshampur, Unassigned)

References

(Blocks 1 open bug)

Details

(Whiteboard: [fxp])

Example error when trying to use motionmark-1.3 as a test name in the browesrtime-*.yml :

 0:35.88 CRITICAL No tests exist for the following name (obtained from config.yml): motionmark-1.3
 0:36.24 CRITICAL No tests exist for the following name (obtained from config.yml): motionmark-1.3
Traceback (most recent call last):
Traceback (most recent call last):
  File "/Users/kshampur/gecko/python/mozlint/mozlint/roller.py", line 72, in _run_worker
    res = func(paths, config, **lintargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kshampur/gecko/python/mozlint/mozlint/types.py", line 57, in __call__
    return self._lint(paths, config, **lintargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kshampur/gecko/python/mozlint/mozlint/types.py", line 200, in _lint
    func(files, config, logger, **lintargs)
  File "/Users/kshampur/gecko/python/mozlint/mozlint/roller.py", line 72, in _run_worker
    res = func(paths, config, **lintargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kshampur/gecko/tools/lint/perfdocs/__init__.py", line 13, in lint
    return perfdocs.run_perfdocs(config, logger=logger, paths=paths, generate=fix)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kshampur/gecko/python/mozlint/mozlint/types.py", line 57, in __call__
    return self._lint(paths, config, **lintargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kshampur/gecko/tools/lint/perfdocs/perfdocs.py", line 95, in run_perfdocs
    generator.generate_perfdocs()
  File "/Users/kshampur/gecko/python/mozlint/mozlint/types.py", line 200, in _lint
    func(files, config, logger, **lintargs)
  File "/Users/kshampur/gecko/tools/lint/perfdocs/generator.py", line 266, in generate_perfdocs
    perfdocs_tmpdir = self._create_perfdocs()
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kshampur/gecko/tools/lint/perfdocs/__init__.py", line 13, in lint
    return perfdocs.run_perfdocs(config, logger=logger, paths=paths, generate=fix)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kshampur/gecko/tools/lint/perfdocs/perfdocs.py", line 95, in run_perfdocs
    generator.generate_perfdocs()
  File "/Users/kshampur/gecko/tools/lint/perfdocs/generator.py", line 156, in _create_perfdocs
    framework_docs = self.build_perfdocs_from_tree()
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kshampur/gecko/tools/lint/perfdocs/generator.py", line 266, in generate_perfdocs
    perfdocs_tmpdir = self._create_perfdocs()
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kshampur/gecko/tools/lint/perfdocs/generator.py", line 93, in build_perfdocs_from_tree
    test_description = gatherer.build_test_description(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kshampur/gecko/tools/lint/perfdocs/framework_gatherers.py", line 248, in build_test_description
    raise Exception(
  File "/Users/kshampur/gecko/tools/lint/perfdocs/generator.py", line 156, in _create_perfdocs
    framework_docs = self.build_perfdocs_from_tree()
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kshampur/gecko/tools/lint/perfdocs/generator.py", line 93, in build_perfdocs_from_tree
    test_description = gatherer.build_test_description(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: No tests exist for the following name (obtained from config.yml): motionmark-1.3
  File "/Users/kshampur/gecko/tools/lint/perfdocs/framework_gatherers.py", line 248, in build_test_description
    raise Exception(
Exception: No tests exist for the following name (obtained from config.yml): motionmark-1.3
 0:32.58 /testing/raptor/raptor/perfdocs/config.yml  0  error<function TerminalColors.__init__.<locals>.apply_formatting at 0x1050760c0>  Could not find an existing test for motionmark-1.3 - bad test name?  Flawless performance docs. (perfdocs)<function TerminalColors.__init__.<locals>.apply_formatting at 0x1050760c0>
 0:32.58 /testing/raptor/raptor/tests/benchmarks/motionmark-1-3-mobile.toml  0  error<function TerminalColors.__init__.<locals>.apply_formatting at 0x1050760c0>  Could not find a test description for motionmark-1-3  Flawless performance docs. (perfdocs)<function TerminalColors.__init__.<locals>.apply_formatting at 0x1050760c0>
 0:32.58 /testing/raptor/raptor/tests/benchmarks/motionmark-1-3-mobile.toml  0  error<function TerminalColors.__init__.<locals>.apply_formatting at 0x1050760c0>  Could not find a test description for motionmark-htmlsuite-1-3  Flawless performance docs. (perfdocs)<function TerminalColors.__init__.<locals>.apply_formatting at 0x1050760c0>
/testing/raptor/raptor/perfdocs/config.yml
  0   error  Could not find an existing test for motionmark-1.3 - bad test name?  Flawless performance docs. (perfdocs)

/testing/raptor/raptor/tests/benchmarks/motionmark-1-3-mobile.toml
  0   error  Could not find a test description for motionmark-1-3            Flawless performance docs. (perfdocs)
  0   error  Could not find a test description for motionmark-htmlsuite-1-3  Flawless performance docs. (perfdocs)

A failure occurred in the perfdocs linter.
✖ 4 problems (3 errors, 0 warnings, 1 failure, 0 fixed)
Whiteboard: [fxp]

:kshampur, can you double-check the error you were getting? This error shows that the test name in the config doesn't match the test name found in the raptor test manifests which is expected if the test name was still motionmark-1-3.

Flags: needinfo?(kshampur)

Ah my mistake. that is in the toml file

it was [motionmark-1-3] rather than 1.3

I changed it to [motionmark-1.3] just now (just in desktop for the moment), but still get an error

 0:06.61 CRITICAL No tests exist for the following name (obtained from config.yml): motionmark-1.3
Traceback (most recent call last):
  File "/Users/kshampur/gecko/python/mozlint/mozlint/roller.py", line 72, in _run_worker
    res = func(paths, config, **lintargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kshampur/gecko/python/mozlint/mozlint/types.py", line 57, in __call__
    return self._lint(paths, config, **lintargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kshampur/gecko/python/mozlint/mozlint/types.py", line 200, in _lint
    func(files, config, logger, **lintargs)
  File "/Users/kshampur/gecko/tools/lint/perfdocs/__init__.py", line 13, in lint
    return perfdocs.run_perfdocs(config, logger=logger, paths=paths, generate=fix)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kshampur/gecko/tools/lint/perfdocs/perfdocs.py", line 95, in run_perfdocs
    generator.generate_perfdocs()
  File "/Users/kshampur/gecko/tools/lint/perfdocs/generator.py", line 266, in generate_perfdocs
    perfdocs_tmpdir = self._create_perfdocs()
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kshampur/gecko/tools/lint/perfdocs/generator.py", line 156, in _create_perfdocs
    framework_docs = self.build_perfdocs_from_tree()
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kshampur/gecko/tools/lint/perfdocs/generator.py", line 93, in build_perfdocs_from_tree
    test_description = gatherer.build_test_description(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kshampur/gecko/tools/lint/perfdocs/framework_gatherers.py", line 248, in build_test_description
    raise Exception(
Exception: No tests exist for the following name (obtained from config.yml): motionmark-1.3
 0:05.52 /testing/raptor/raptor/perfdocs/config.yml  0  error<function TerminalColors.__init__.<locals>.apply_formatting at 0x103eea0c0>  Could not find an existing test for motionmark-1.3 - bad test name?  Flawless performance docs. (perfdocs)<function TerminalColors.__init__.<locals>.apply_formatting at 0x103eea0c0>
 0:05.52 /testing/raptor/raptor/tests/benchmarks/motionmark-1-3-desktop.toml  0  error<function TerminalColors.__init__.<locals>.apply_formatting at 0x103eea0c0>  Could not find a test description for motionmark-1  Flawless performance docs. (perfdocs)<function TerminalColors.__init__.<locals>.apply_formatting at 0x103eea0c0>
 0:05.52 /testing/raptor/raptor/tests/benchmarks/motionmark-1-3-desktop.toml  0  error<function TerminalColors.__init__.<locals>.apply_formatting at 0x103eea0c0>  Could not find a test description for motionmark-htmlsuite-1  Flawless performance docs. (perfdocs)<function TerminalColors.__init__.<locals>.apply_formatting at 0x103eea0c0>
 0:05.52 /testing/raptor/raptor/tests/benchmarks/motionmark-1-3-mobile.toml  0  error<function TerminalColors.__init__.<locals>.apply_formatting at 0x103eea0c0>  Could not find a test description for motionmark-1-3  Flawless performance docs. (perfdocs)<function TerminalColors.__init__.<locals>.apply_formatting at 0x103eea0c0>
 0:05.52 /testing/raptor/raptor/tests/benchmarks/motionmark-1-3-mobile.toml  0  error<function TerminalColors.__init__.<locals>.apply_formatting at 0x103eea0c0>  Could not find a test description for motionmark-htmlsuite-1-3  Flawless performance docs. (perfdocs)<function TerminalColors.__init__.<locals>.apply_formatting at 0x103eea0c0>
/testing/raptor/raptor/perfdocs/config.yml
  0   error  Could not find an existing test for motionmark-1.3 - bad test name?  Flawless performance docs. (perfdocs)

/testing/raptor/raptor/tests/benchmarks/motionmark-1-3-desktop.toml
  0   error  Could not find a test description for motionmark-1            Flawless performance docs. (perfdocs)
  0   error  Could not find a test description for motionmark-htmlsuite-1  Flawless performance docs. (perfdocs)

/testing/raptor/raptor/tests/benchmarks/motionmark-1-3-mobile.toml
  0   error  Could not find a test description for motionmark-1-3            Flawless performance docs. (perfdocs)
  0   error  Could not find a test description for motionmark-htmlsuite-1-3  Flawless performance docs. (perfdocs)

notice how the desktop one only parses up to motionmark-1 rather than 1.3?

Flags: needinfo?(kshampur)

Ah I see, thanks for stack trace!

You need to log in before you can comment on or make changes to this bug.