1354791 - Unable to find moz.build metadata for testing/firefox-ui/tests/functional/security/test_safe_browsing_initial_download.py

Status: REOPENED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

When I try to find this file's metadata, it instead gives me an error:
"unable to obtain moz.build info"

I don't know why this is happening, as the closest moz.build file I find should cover it: https://hg.mozilla.org/mozilla-central/file/tip/testing/firefox-ui/moz.build

Am I missing something? Is there a syntax error somewhere or something?

I guess something gets logged somewhere: https://dxr.mozilla.org/hgcustom_version-control-tools/source/hgext/hgmo/__init__.py#301-305

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/json-mozbuildinfo?p=testing/firefox-ui/ fails with that "unable to obtain" error.


https://hg.mozilla.org/mozilla-central/json-mozbuildinfo?p=testing/ does not fail, though it doesn't list any metadata, I assume because testing/moz.build only has metadata for things matching various "with Files" conditions, which aren't there for the root "testing" folder.

Comment 2

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/json-mozbuildinfo?p=testing/xpcshell/ fails with the same error.

https://hg.mozilla.org/mozilla-central/json-mozbuildinfo?p=testing/talos/ loads just fine with suggestions.

Comment 3

[Gregory Szorc [:gps]]

It encounters an exception. But then it raises an exception when formatting that exception to a string. Derp.

$ /usr/bin/sudo /usr/local/bin/mozbuild-eval /repo/hg/mozilla/mozilla-central
{"repo": "/repo/hg/mozilla/mozilla-central", "node": "default", "paths": ["testing/firefox-ui/tests/functional/security/test_safe_browsing_initial_download.py"]}
** unknown exception encountered, please report by visiting
** https://mercurial-scm.org/wiki/BugTracker
** Python 2.7.12 (default, Apr 12 2017, 00:20:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]
** Mercurial Distributed SCM (version 4.1.2)
** Extensions loaded: hgmo
Traceback (most recent call last):
  File "/usr/bin/hg", line 45, in <module>
    mercurial.dispatch.run()
  File "/usr/lib/python2.7/site-packages/mercurial/dispatch.py", line 63, in run
    sys.exit((dispatch(request(pycompat.sysargv[1:])) or 0) & 255)
  File "/usr/lib/python2.7/site-packages/mercurial/dispatch.py", line 129, in dispatch
    ret = _runcatch(req)
  File "/usr/lib/python2.7/site-packages/mercurial/dispatch.py", line 219, in _runcatch
    return callcatch(ui, _runcatchfunc)
  File "/usr/lib/python2.7/site-packages/mercurial/dispatch.py", line 227, in callcatch
    return scmutil.callcatch(ui, func)
  File "/usr/lib/python2.7/site-packages/mercurial/scmutil.py", line 152, in callcatch
    return func()
  File "/usr/lib/python2.7/site-packages/mercurial/dispatch.py", line 208, in _runcatchfunc
    return _dispatch(req)
  File "/usr/lib/python2.7/site-packages/mercurial/dispatch.py", line 811, in _dispatch
    cmdpats, cmdoptions)
  File "/usr/lib/python2.7/site-packages/mercurial/dispatch.py", line 563, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/usr/lib/python2.7/site-packages/mercurial/dispatch.py", line 819, in _runcommand
    return cmdfunc()
  File "/usr/lib/python2.7/site-packages/mercurial/dispatch.py", line 808, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
  File "/usr/lib/python2.7/site-packages/mercurial/util.py", line 1051, in check
    return func(*args, **kwargs)
  File "/var/hg/version-control-tools/hgext/hgmo/__init__.py", line 604, in mozbuildinfocommand
    d = {'error': 'Exception reading moz.build info: %s' % str(e)}
  File "/usr/lib/python2.7/site-packages/mozbuild/frontend/reader.py", line 614, in __str__
    self._print_sandbox_error(s)
  File "/usr/lib/python2.7/site-packages/mozbuild/frontend/reader.py", line 644, in _print_sandbox_error
    frames = traceback.extract_tb(trace)
  File "/usr/lib/python2.7/traceback.py", line 101, in extract_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/usr/lib/python2.7/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib/python2.7/linecache.py", line 41, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib/python2.7/linecache.py", line 89, in updatecache
    if module_globals and '__loader__' in module_globals:
  File "/usr/lib/python2.7/site-packages/mozbuild/frontend/sandbox.py", line 297, in __len__
    raise NotImplementedError('Not supported')
NotImplementedError: Not supported

Comment 4

[Gregory Szorc [:gps]]

I can't reproduce this failure with `mach file-info` for this path.

I'm guessing this has to do with hg.mo's moz.build evaluation environment using an older version of the moz.build evaluation code. Whenever we introduce a new variable we need to upgrade that evaluation environment otherwise any moz.build references to unknown primitives result in failure.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/02be48e20435
docker: upgrade mozbuild package in chroot environment

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/f640bbe25218
docker: add futures to moz.build chroot

Comment 7

[Gregory Szorc [:gps]]

Had to back these out because I kept going down the rabbit hole of more missing imports. When I hit an import for "mach" I knew something was wrong. This will require some import refactoring in the mozbuild package to cut down on bloat.

Comment 8

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/764180b39556
Backed out changesets f640bbe25218 and 02be48e20435

Comment 9

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

We're hitting this more and more, lately, it seems (anecdotally, at least, I don't have hard numbers for it).

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

:gps, is this something you can pick up?  It would be a much cleaner fix to bug 1429030 than hacking up treeherder code that is on the edge of being hacky.

Comment 12

[Gregory Szorc [:gps]]

I doubt we'll ever fix moz.build reading on hg.mozilla.org :(

We seem to be slowly marching towards using moz.yaml files to express file-level metadata, such as bug components and suggested reviewers. We haven't formalized plans yet, but everyone involved seems to think that's what will happen. At that point, adding an API on hg.mo that parses YAML is much easier to maintain.

At this point, we may want to create a Taskcluster task that emits a file containing the bug component mapping. Then we can stand up a simple query service that consumes that. We may lose the ability to easily query an arbitrary VCS revision. But we could certainly index the heads of the major repos easily enough.