Closed Bug 1454613 Opened 6 years ago Closed 5 years ago

[BinAST] Build binsource when building SpiderMonkey

Categories

(Core :: JavaScript Engine, enhancement, P2)

Product:
Core
Component:
JavaScript Engine
Type:
enhancement

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla65
Tracking Flags:
Tracking Status
firefox65 --- fixed

People

(Reporter: jorendorff, Assigned: ted)

References

Details

Attachments

(1 file, 1 obsolete file)

bug 1454613 - build binsource when building SpiderMonkey. r?build
47 bytes, text/x-phabricator-request
Details | Review 
Rev 978eea13c274 fixes a trivial typo in binsource. It should be easy to add a build of binsource, even though we won't use the executable anywhere, to catch this kind of mistake.
Should this bug be part of another components, such as the build-system?
status-firefox61: --- → fix-optional
Flags: needinfo?(dteller)
Priority: -- → P2
I agree with jorendorff that it's in the right place.
Flags: needinfo?(dteller)
Attempting to add either RUST_PROGRAMS += ['binsource'] or HOST_RUST_PROGRAMS += ['binsource'] causes the following error:

KeyError: u'RUST_TARGET'

:/
Full error, during `./mach configure`:

 0:11.90 Creating config.status
 0:12.32 Reticulating splines...
 0:12.52 Traceback (most recent call last):
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/configure.py", line 127, in <module>
 0:12.52     sys.exit(main(sys.argv))
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/configure.py", line 34, in main
 0:12.52     return config_status(config)
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/configure.py", line 122, in config_status
 0:12.52     return config_status(args=[], **encode(sanitized_config, encoding))
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/python/mozbuild/mozbuild/config_status.py", line 144, in config_status
 0:12.52     definitions = list(definitions)
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/python/mozbuild/mozbuild/frontend/emitter.py", line 184, in emit
 0:12.52     objs = list(emitfn(out))
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/python/mozbuild/mozbuild/frontend/emitter.py", line 1148, in emit_from_context
 0:12.52     for obj in self._handle_linkables(context, passthru, generated_files):
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/python/mozbuild/mozbuild/frontend/emitter.py", line 619, in _handle_linkables
 0:12.52     self._binaries[program] = cls(context, program, cargo_file)
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/python/mozbuild/mozbuild/frontend/data.py", line 562, in __init__
 0:12.52     cargo_dir = cargo_output_directory(context, self.TARGET_SUBST_VAR)
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/python/mozbuild/mozbuild/frontend/data.py", line 543, in cargo_output_directory
 0:12.52     return mozpath.join(context.config.substs[target_var], rust_build_kind)
 0:12.52 KeyError: u'RUST_TARGET'
 0:12.58 *** Fix above errors and then restart with               "/Applications/Xcode.app/Contents/Developer/usr/bin/make -f client.mk build"
 0:12.58 make: *** [configure] Error 1

See patch https://bugzilla.mozilla.org/attachment.cgi?id=8970485

Ted, any suggestion?
Flags: needinfo?(ted)
Summary: Build binsource when building SpiderMonkey → [BinAST] Build binsource when building SpiderMonkey
I think this just needs the patch from bug 1444141 in order to work.
Flags: needinfo?(ted)
Now, with David's patch, mach configure gives me this:

...
 0:19.21 js/src> Creating config.status
 0:19.44 Creating config.status
 0:19.77 Reticulating splines...
 0:19.87 Traceback (most recent call last):
 0:19.87   File "/Users/jorendorff/work/gecko/configure.py", line 123, in <module>
 0:19.87     sys.exit(main(sys.argv))
 0:19.87   File "/Users/jorendorff/work/gecko/configure.py", line 34, in main
 0:19.87     return config_status(config)
 0:19.87   File "/Users/jorendorff/work/gecko/configure.py", line 118, in config_status
 0:19.87     return config_status(args=[], **encode(sanitized_config, encoding))
 0:19.87   File "/Users/jorendorff/work/gecko/python/mozbuild/mozbuild/config_status.py", line 143, in config_status
 0:19.87     definitions = list(definitions)
 0:19.87   File "/Users/jorendorff/work/gecko/python/mozbuild/mozbuild/frontend/emitter.py", line 184, in emit
 0:19.87     objs = list(emitfn(out))
 0:19.87   File "/Users/jorendorff/work/gecko/python/mozbuild/mozbuild/frontend/emitter.py", line 1196, in emit_from_context
 0:19.88     for obj in self._handle_linkables(context, passthru, generated_files):
 0:19.88   File "/Users/jorendorff/work/gecko/python/mozbuild/mozbuild/frontend/emitter.py", line 971, in _handle_linkables
 0:19.88     linkable.sources[suffix] += srcs
 0:19.88 AttributeError: 'RustProgram' object has no attribute 'sources'
 0:19.92 *** Fix above errors and then restart with\
 0:19.92                "./mach build"
 0:19.92 gmake: *** [client.mk:115: configure] Error 1
Flags: needinfo?(ted)
Ah, you're going to need a separate moz.build file for binsource. Unfortunately our build system has never supported building multiple independent targets per directory, so it's getting confused and trying to associate the SOURCES from js/src with the binsource binary.
Flags: needinfo?(ted)
That sounds kind of hopeful, though. Thanks.

David, can you take this?
Flags: needinfo?(dteller)
I can take it, but can I put it very far on my backlog?
Flags: needinfo?(dteller)
Sure. This isn't drop-everything urgent; it's just one more hard requirement (among many) if we ever want to ship this thing.

Quoting bug 1497378 comment 3:

> Friendly reminder: BinAST isn't going to ship without at least a test
> that ensures we can always build the `-auto` generated files from their
> sources. David and I agreed to this months ago, but the build system is
> still enough of an obstacle that it hasn't gotten done; see bug 1454613.
> 
> The test approach is what we do for the HTML parser. It's also
> autogenerated, and also a pain to build when cross-compiling, so the
> generated files are in-tree, and we have a test that makes sure they can
> be regenerated and that the generated code is up-to-date.
> 
> Just so no one is surprised when it comes time to flip the pref, or turn
> on JS_BUILD_BINAST outside nightly: it's a hard requirement.
Attachment #8970485 - Attachment is obsolete: true
Assignee: nobody → ted
With the patch I just submitted it seems to build OK locally:
lrwxrwxrwx 1 luser luser 72 Nov  6 10:45 /build/debug-mozilla-central/dist/bin/binsource -> ../../js/src/frontend/binsource/x86_64-unknown-linux-gnu/debug/binsource

luser@eye7:/build/mozilla-central$ /build/debug-mozilla-central/dist/bin/binsource 
error: The following required arguments were not provided:
    <INPUT.webidl>
    <INPUT.yaml>
    --out-class <OUT_HEADER_CLASS_FILE>
    --out-impl <OUT_IMPL_FILE>
    --out-token <OUT_TOKEN_FILE>

USAGE:
    binsource <INPUT.webidl> <INPUT.yaml> --out-class <OUT_HEADER_CLASS_FILE> --out-impl <OUT_IMPL_FILE> --out-token <OUT_TOKEN_FILE>

For more information try --help
Yoric: should we land this patch? It'll make us build this binary, but doesn't add any checks for the generated code.
Flags: needinfo?(dteller)
(In reply to Ted Mielczarek [:ted] [:ted.mielczarek] from comment #14)
> Yoric: should we land this patch? It'll make us build this binary, but
> doesn't add any checks for the generated code.

Let's do it.
Flags: needinfo?(dteller)
I'll give it a run past try to make sure it isn't going to break in any strange environments. Also: I just realized that this probably needs to go in `HOST_RUST_PROGRAMS` since it's going to be used as part of the build eventually.
OK, that looks good.
Pushed by tmielczarek@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d3468dd807eb
build binsource when building SpiderMonkey. r=firefox-build-system-reviewers,chmanchester
Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&fromchange=d3468dd807eb970c794257cde9f48650bbf7d7bd&tochange=4dcf88b1fabcfb7fb0e4c87aec107f09e81a200b&searchStr=btup&selectedJob=212295179

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=212295179&repo=autoland&lineNumber=1663

Backout link: https://hg.mozilla.org/integration/autoland/rev/4dcf88b1fabcfb7fb0e4c87aec107f09e81a200b

[task 2018-11-16T20:22:40.193Z] 20:22:40     INFO -  Reticulating splines...
[task 2018-11-16T20:22:42.036Z] 20:22:42     INFO -   0:01.90 File already read. Skipping: /builds/worker/workspace/build/src/gfx/angle/targets/angle_common/moz.build
[task 2018-11-16T20:22:51.755Z] 20:22:51     INFO -   0:11.62 /builds/worker/workspace/build/src/rustc/bin/cargo build --build-plan -Z unstable-options --release --frozen --manifest-path /builds/worker/workspace/build/src/js/src/rust/Cargo.toml --lib --target=x86_64-unknown-linux-gnu
[task 2018-11-16T20:22:52.180Z] 20:22:52    ERROR -  Traceback (most recent call last):
[task 2018-11-16T20:22:52.180Z] 20:22:52     INFO -    File "/builds/worker/workspace/build/src/configure.py", line 123, in <module>
[task 2018-11-16T20:22:52.180Z] 20:22:52     INFO -      sys.exit(main(sys.argv))
[task 2018-11-16T20:22:52.180Z] 20:22:52     INFO -    File "/builds/worker/workspace/build/src/configure.py", line 34, in main
[task 2018-11-16T20:22:52.180Z] 20:22:52     INFO -      return config_status(config)
[task 2018-11-16T20:22:52.180Z] 20:22:52     INFO -    File "/builds/worker/workspace/build/src/configure.py", line 118, in config_status
[task 2018-11-16T20:22:52.180Z] 20:22:52     INFO -      return config_status(args=[], **encode(sanitized_config, encoding))
[task 2018-11-16T20:22:52.180Z] 20:22:52     INFO -    File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/config_status.py", line 146, in config_status
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -      the_backend.consume(definitions)
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -    File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/backend/base.py", line 128, in consume
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -      if (not self.consume_object(obj) and
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -    File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/backend/tup.py", line 664, in consume_object
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -      self._gen_rust_rules(obj, backend_file)
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -    File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/backend/tup.py", line 1027, in _gen_rust_rules
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -      cargo_plan = json.loads(''.join(output_lines))
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -    File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -      return _default_decoder.decode(s)
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -    File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -      obj, end = self.raw_decode(s, idx=_w(s, 0).end())
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -    File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -      raise ValueError("No JSON object could be decoded")
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -  ValueError: No JSON object could be decoded
[task 2018-11-16T20:22:52.764Z] 20:22:52     INFO -  *** Fix above errors and then restart with\
[task 2018-11-16T20:22:52.764Z] 20:22:52     INFO -                 "./mach build"
[task 2018-11-16T20:22:52.764Z] 20:22:52     INFO -  client.mk:111: recipe for target 'configure' failed
[task 2018-11-16T20:22:52.764Z] 20:22:52     INFO -  make: *** [configure] Error 1
[task 2018-11-16T20:22:52.856Z] 20:22:52    ERROR - Return code: 2
[task 2018-11-16T20:22:52.856Z] 20:22:52  WARNING - setting return code to 2
[task 2018-11-16T20:22:52.856Z] 20:22:52    FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2018-11-16T20:22:52.856Z] 20:22:52    FATAL - Running post_fatal callback...
[task 2018-11-16T20:22:52.857Z] 20:22:52    FATAL - Exiting -1
Flags: needinfo?(ted)
I dug into this, I filed bug 1508344. I can work around that for now by fixing the Cargo.toml to not cause cargo to emit warnings.
Flags: needinfo?(ted)
Pushed by tmielczarek@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/43c0e876bc6f
build binsource when building SpiderMonkey. r=firefox-build-system-reviewers,chmanchester
https://hg.mozilla.org/mozilla-central/rev/43c0e876bc6f
Status: NEW → RESOLVED
Closed: 5 years ago
status-firefox65: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla65
Blocks: 1576079
You need to log in before you can comment on or make changes to this bug.