Check for llvm-profdata in PGO builds

NEW
Unassigned

Status

enhancement
P3
normal
3 months ago
2 months ago

People

(Reporter: dmajor, Unassigned)

Tracking

Trunk

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

Yesterday I built my own trunk LLVM and was trying to minimize which pieces I compiled. I iterated until my PGO build's mach configure was happy. I then proceeded to mach build, but my build failed much later because I didn't have an llvm-profdata. It would have been nice to check for llvm-profdata during configure (at least when we're doing PGO).

Detecting problems earlier is better than detecting them later.

Pushed by nfroyd@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f593f5a5bfa3
require llvm-profdata when using pgo with clang{,-cl}; r=dmajor

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&revision=f593f5a5bfa3b1b32488eaeaefb6638c0c6c06e7&selectedJob=248983087

Failure log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=248983087&repo=autoland&lineNumber=964
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=248983029&repo=autoland&lineNumber=40626

Backout link: https://hg.mozilla.org/integration/autoland/rev/af3b6cb90e035ee66e0e1aefccb68f1db43236b5

[task 2019-05-29T16:09:51.611Z] 16:09:51 INFO - checking for llvm-profdata... not found
[task 2019-05-29T16:09:51.611Z] 16:09:51 INFO - DEBUG: llvm_profdata: Trying llvm-profdata
[task 2019-05-29T16:09:51.611Z] 16:09:51 INFO - ERROR: Cannot find llvm-profdata
[task 2019-05-29T16:09:51.659Z] 16:09:51 INFO - *** Fix above errors and then restart with
[task 2019-05-29T16:09:51.659Z] 16:09:51 INFO - "./mach build"
[task 2019-05-29T16:09:51.659Z] 16:09:51 INFO - client.mk:111: recipe for target 'configure' failed
[task 2019-05-29T16:09:51.659Z] 16:09:51 INFO - make: *** [configure] Error 1
[task 2019-05-29T16:09:51.766Z] 16:09:51 ERROR - Return code: 2
[task 2019-05-29T16:09:51.766Z] 16:09:51 WARNING - setting return code to 2
[task 2019-05-29T16:09:51.766Z] 16:09:51 FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2019-05-29T16:09:51.767Z] 16:09:51 FATAL - Running post_fatal callback...
[task 2019-05-29T16:09:51.767Z] 16:09:51 FATAL - Exiting -1
[task 2019-05-29T16:09:51.767Z] 16:09:51 INFO - [mozharness: 2019-05-29 16:09:51.767078Z] Finished build step (failed)

[task 2019-05-29T16:26:28.351Z] 16:26:28 WARNING - ../python/mozbuild/mozbuild/test/configure/test_checks_configure.py::TestChecksConfigure::test_pkg_check_modules TEST-UNEXPECTED-FAIL
[task 2019-05-29T16:26:28.351Z] 16:26:28 INFO - ../python/mozbuild/mozbuild/test/configure/test_checks_configure.py::TestChecksConfigure::test_simple_keyfile PASSED
[task 2019-05-29T16:26:28.351Z] 16:26:28 INFO - =================================== FAILURES ===================================
[task 2019-05-29T16:26:28.351Z] 16:26:28 INFO - __________________ TestChecksConfigure.test_pkg_check_modules __________________
[task 2019-05-29T16:26:28.351Z] 16:26:28 INFO - self = <test_checks_configure.TestChecksConfigure testMethod=test_pkg_check_modules>
[task 2019-05-29T16:26:28.351Z] 16:26:28 INFO - def test_pkg_check_modules(self):
[task 2019-05-29T16:26:28.351Z] 16:26:28 INFO - mock_pkg_config_version = '0.10.0'
[task 2019-05-29T16:26:28.351Z] 16:26:28 INFO - mock_pkg_config_path = mozpath.abspath('/usr/bin/pkg-config')
[task 2019-05-29T16:26:28.351Z] 16:26:28 INFO - def mock_pkg_config(_, args):
[task 2019-05-29T16:26:28.351Z] 16:26:28 INFO - if args[0:2] == ['--errors-to-stdout', '--print-errors']:
[task 2019-05-29T16:26:28.351Z] 16:26:28 INFO - assert len(args) == 3
[task 2019-05-29T16:26:28.351Z] 16:26:28 INFO - package = args[2]
[task 2019-05-29T16:26:28.351Z] 16:26:28 INFO - if package == 'unknown':
[task 2019-05-29T16:26:28.351Z] 16:26:28 INFO - return (1, "Package unknown was not found in the pkg-config search path.\n"
[task 2019-05-29T16:26:28.352Z] 16:26:28 INFO - "Perhaps you should add the directory containing unknown.pc'\n" [task 2019-05-29T16:26:28.352Z] 16:26:28 INFO - "to the PKG_CONFIG_PATH environment variable\n" [task 2019-05-29T16:26:28.352Z] 16:26:28 INFO - "No package 'unknown' found", '') [task 2019-05-29T16:26:28.352Z] 16:26:28 INFO - if package == 'valid': [task 2019-05-29T16:26:28.352Z] 16:26:28 INFO - return 0, '', '' [task 2019-05-29T16:26:28.352Z] 16:26:28 INFO - if package == 'new > 1.1': [task 2019-05-29T16:26:28.352Z] 16:26:28 INFO - return 1, "Requested 'new > 1.1' but version of new is 1.1", '' [task 2019-05-29T16:26:28.352Z] 16:26:28 INFO - if args[0] == '--cflags': [task 2019-05-29T16:26:28.352Z] 16:26:28 INFO - assert len(args) == 2 [task 2019-05-29T16:26:28.352Z] 16:26:28 INFO - return 0, '-I/usr/include/%s' % args[1], '' [task 2019-05-29T16:26:28.352Z] 16:26:28 INFO - if args[0] == '--libs': [task 2019-05-29T16:26:28.352Z] 16:26:28 INFO - assert len(args) == 2 [task 2019-05-29T16:26:28.352Z] 16:26:28 INFO - return 0, '-l%s' % args[1], '' [task 2019-05-29T16:26:28.352Z] 16:26:28 INFO - if args[0] == '--version': [task 2019-05-29T16:26:28.352Z] 16:26:28 INFO - return 0, mock_pkg_config_version, '' [task 2019-05-29T16:26:28.352Z] 16:26:28 INFO - self.fail("Unexpected arguments to mock_pkg_config: %s" % args) [task 2019-05-29T16:26:28.353Z] 16:26:28 INFO - def get_result(cmd, args=[], extra_paths=None): [task 2019-05-29T16:26:28.358Z] 16:26:28 INFO - return self.get_result(textwrap.dedent('''\ [task 2019-05-29T16:26:28.358Z] 16:26:28 INFO - option('--disable-compile-environment', help='compile env') [task 2019-05-29T16:26:28.359Z] 16:26:28 INFO - compile_environment = depends(when='--enable-compile-environment')(lambda: True) [task 2019-05-29T16:26:28.359Z] 16:26:28 INFO - toolchain_prefix = depends(when=True)(lambda: None) [task 2019-05-29T16:26:28.360Z] 16:26:28 INFO - include('%(topsrcdir)s/build/moz.configure/util.configure') [task 2019-05-29T16:26:28.361Z] 16:26:28 INFO - include('%(topsrcdir)s/build/moz.configure/checks.configure') [task 2019-05-29T16:26:28.361Z] 16:26:28 INFO - include('%(topsrcdir)s/build/moz.configure/pkg.configure') [task 2019-05-29T16:26:28.362Z] 16:26:28 INFO - ''' % {'topsrcdir': topsrcdir}) + cmd, args=args, extra_paths=extra_paths, [task 2019-05-29T16:26:28.362Z] 16:26:28 INFO - includes=()) [task 2019-05-29T16:26:28.363Z] 16:26:28 INFO - extra_paths = { [task 2019-05-29T16:26:28.363Z] 16:26:28 INFO - mock_pkg_config_path: mock_pkg_config, [task 2019-05-29T16:26:28.364Z] 16:26:28 INFO - } [task 2019-05-29T16:26:28.364Z] 16:26:28 INFO - config, output, status = get_result("pkg_check_modules('MOZ_VALID', 'valid')") [task 2019-05-29T16:26:28.365Z] 16:26:28 INFO - self.assertEqual(status, 1) [task 2019-05-29T16:26:28.365Z] 16:26:28 INFO - self.assertEqual(output, textwrap.dedent('''\ [task 2019-05-29T16:26:28.366Z] 16:26:28 INFO - checking for pkg_config... not found [task 2019-05-29T16:26:28.366Z] 16:26:28 INFO - ERROR: *** The pkg-config script could not be found. Make sure it is [task 2019-05-29T16:26:28.367Z] 16:26:28 INFO - *** in your path, or set the PKG_CONFIG environment variable [task 2019-05-29T16:26:28.367Z] 16:26:28 INFO - *** to the full path to pkg-config. [task 2019-05-29T16:26:28.368Z] 16:26:28 INFO - ''')) [task 2019-05-29T16:26:28.368Z] 16:26:28 INFO - config, output, status = get_result("pkg_check_modules('MOZ_VALID', 'valid')", [task 2019-05-29T16:26:28.369Z] 16:26:28 INFO - extra_paths=extra_paths) [task 2019-05-29T16:26:28.369Z] 16:26:28 INFO - self.assertEqual(status, 0) [task 2019-05-29T16:26:28.370Z] 16:26:28 INFO - self.assertEqual(output, textwrap.dedent('''\ [task 2019-05-29T16:26:28.370Z] 16:26:28 INFO - checking for pkg_config... %s [task 2019-05-29T16:26:28.371Z] 16:26:28 INFO - checking for pkg-config version... %s [task 2019-05-29T16:26:28.371Z] 16:26:28 INFO - checking for valid... yes [task 2019-05-29T16:26:28.372Z] 16:26:28 INFO - checking MOZ_VALID_CFLAGS... -I/usr/include/valid [task 2019-05-29T16:26:28.372Z] 16:26:28 INFO - checking MOZ_VALID_LIBS... -lvalid [task 2019-05-29T16:26:28.373Z] 16:26:28 INFO - ''' % (mock_pkg_config_path, mock_pkg_config_version))) [task 2019-05-29T16:26:28.373Z] 16:26:28 INFO - self.assertEqual(config, { [task 2019-05-29T16:26:28.374Z] 16:26:28 INFO - 'PKG_CONFIG': mock_pkg_config_path, [task 2019-05-29T16:26:28.375Z] 16:26:28 INFO - 'MOZ_VALID_CFLAGS': ('-I/usr/include/valid',), [task 2019-05-29T16:26:28.375Z] 16:26:28 INFO - 'MOZ_VALID_LIBS': ('-lvalid',), [task 2019-05-29T16:26:28.376Z] 16:26:28 INFO - }) [task 2019-05-29T16:26:28.377Z] 16:26:28 INFO - config, output, status = get_result("pkg_check_modules('MOZ_UKNOWN', 'unknown')", [task 2019-05-29T16:26:28.377Z] 16:26:28 INFO - extra_paths=extra_paths) [task 2019-05-29T16:26:28.378Z] 16:26:28 INFO - self.assertEqual(status, 1) [task 2019-05-29T16:26:28.378Z] 16:26:28 INFO - self.assertEqual(output, textwrap.dedent('''\ [task 2019-05-29T16:26:28.379Z] 16:26:28 INFO - checking for pkg_config... %s [task 2019-05-29T16:26:28.379Z] 16:26:28 INFO - checking for pkg-config version... %s [task 2019-05-29T16:26:28.379Z] 16:26:28 INFO - checking for unknown... no [task 2019-05-29T16:26:28.380Z] 16:26:28 INFO - ERROR: Package unknown was not found in the pkg-config search path. [task 2019-05-29T16:26:28.380Z] 16:26:28 INFO - ERROR: Perhaps you should add the directory containingunknown.pc'
[task 2019-05-29T16:26:28.381Z] 16:26:28 INFO - ERROR: to the PKG_CONFIG_PATH environment variable
[task 2019-05-29T16:26:28.381Z] 16:26:28 INFO - ERROR: No package 'unknown' found
[task 2019-05-29T16:26:28.381Z] 16:26:28 INFO - ''' % (mock_pkg_config_path, mock_pkg_config_version)))
[task 2019-05-29T16:26:28.382Z] 16:26:28 INFO - self.assertEqual(config, {
[task 2019-05-29T16:26:28.382Z] 16:26:28 INFO - 'PKG_CONFIG': mock_pkg_config_path,
[task 2019-05-29T16:26:28.383Z] 16:26:28 INFO - })
[task 2019-05-29T16:26:28.383Z] 16:26:28 INFO - config, output, status = get_result("pkg_check_modules('MOZ_NEW', 'new > 1.1')",
[task 2019-05-29T16:26:28.383Z] 16:26:28 INFO - extra_paths=extra_paths)
[task 2019-05-29T16:26:28.384Z] 16:26:28 INFO - self.assertEqual(status, 1)
[task 2019-05-29T16:26:28.384Z] 16:26:28 INFO - self.assertEqual(output, textwrap.dedent('''
[task 2019-05-29T16:26:28.384Z] 16:26:28 INFO - checking for pkg_config... %s
[task 2019-05-29T16:26:28.385Z] 16:26:28 INFO - checking for pkg-config version... %s
[task 2019-05-29T16:26:28.385Z] 16:26:28 INFO - checking for new > 1.1... no
[task 2019-05-29T16:26:28.385Z] 16:26:28 INFO - ERROR: Requested 'new > 1.1' but version of new is 1.1
[task 2019-05-29T16:26:28.385Z] 16:26:28 INFO - ''' % (mock_pkg_config_path, mock_pkg_config_version)))
[task 2019-05-29T16:26:28.385Z] 16:26:28 INFO - self.assertEqual(config, {
[task 2019-05-29T16:26:28.386Z] 16:26:28 INFO - 'PKG_CONFIG': mock_pkg_config_path,
[task 2019-05-29T16:26:28.386Z] 16:26:28 INFO - })
[task 2019-05-29T16:26:28.386Z] 16:26:28 INFO - # allow_missing makes missing packages non-fatal.
[task 2019-05-29T16:26:28.386Z] 16:26:28 INFO - cmd = textwrap.dedent('''
[task 2019-05-29T16:26:28.386Z] 16:26:28 INFO - have_new_module = pkg_check_modules('MOZ_NEW', 'new > 1.1', allow_missing=True)
[task 2019-05-29T16:26:28.386Z] 16:26:28 INFO - @depends(have_new_module)
[task 2019-05-29T16:26:28.386Z] 16:26:28 INFO - def log_new_module_error(mod):
[task 2019-05-29T16:26:28.387Z] 16:26:28 INFO - if mod is not True:
[task 2019-05-29T16:26:28.387Z] 16:26:28 INFO - log.info('Module not found.')
[task 2019-05-29T16:26:28.387Z] 16:26:28 INFO - ''')
[task 2019-05-29T16:26:28.387Z] 16:26:28 INFO - config, output, status = get_result(cmd, extra_paths=extra_paths)
[task 2019-05-29T16:26:28.387Z] 16:26:28 INFO - self.assertEqual(status, 0)
[task 2019-05-29T16:26:28.388Z] 16:26:28 INFO - self.assertEqual(output, textwrap.dedent('''
[task 2019-05-29T16:26:28.388Z] 16:26:28 INFO - checking for pkg_config... %s
[task 2019-05-29T16:26:28.388Z] 16:26:28 INFO - checking for pkg-config version... %s
[task 2019-05-29T16:26:28.388Z] 16:26:28 INFO - checking for new > 1.1... no
[task 2019-05-29T16:26:28.388Z] 16:26:28 INFO - WARNING: Requested 'new > 1.1' but version of new is 1.1
[task 2019-05-29T16:26:28.389Z] 16:26:28 INFO - Module not found.
[task 2019-05-29T16:26:28.389Z] 16:26:28 INFO - ''' % (mock_pkg_config_path, mock_pkg_config_version)))
[task 2019-05-29T16:26:28.389Z] 16:26:28 INFO - self.assertEqual(config, {
[task 2019-05-29T16:26:28.389Z] 16:26:28 INFO - 'PKG_CONFIG': mock_pkg_config_path,
[task 2019-05-29T16:26:28.390Z] 16:26:28 INFO - })
[task 2019-05-29T16:26:28.390Z] 16:26:28 INFO - config, output, status = get_result(cmd,
[task 2019-05-29T16:26:28.390Z] 16:26:28 INFO - args=['--disable-compile-environment'],
[task 2019-05-29T16:26:28.390Z] 16:26:28 INFO - extra_paths=extra_paths)
[task 2019-05-29T16:26:28.390Z] 16:26:28 INFO - self.assertEqual(status, 0)
[task 2019-05-29T16:26:28.391Z] 16:26:28 INFO - > self.assertEqual(output, 'Module not found.\n')
[task 2019-05-29T16:26:28.391Z] 16:26:28 INFO - E AssertionError: 'checking for pkg_config... not found\nModule not found.\n' != u'Module not found.\n'
[task 2019-05-29T16:26:28.391Z] 16:26:28 INFO - ../python/mozbuild/mozbuild/test/configure/test_checks_configure.py:752: AssertionError

Flags: needinfo?(nfroyd)

Bugbug thinks this bug is a enhancement, but please change it back in case of error.

Type: defect → enhancement
Flags: needinfo?(nfroyd)
Priority: -- → P3

There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:froydnj, could you have a look please?
For more information, please visit auto_nag documentation.

Flags: needinfo?(nfroyd)
Flags: needinfo?(nfroyd)
You need to log in before you can comment on or make changes to this bug.