1454613 - [BinAST] Build binsource when building SpiderMonkey

Status: RESOLVED FIXED

(Core :: JavaScript Engine, enhancement, P2)

Description

[Jason Orendorff [:jorendorff]]

Rev 978eea13c274 fixes a trivial typo in binsource. It should be easy to add a build of binsource, even though we won't use the executable anywhere, to catch this kind of mistake.

Comment 1

[Nicolas B. Pierron [:nbp]]

Should this bug be part of another components, such as the build-system?

Comment 2

[David Teller [:Yoric] - still alive but not very active]

I agree with jorendorff that it's in the right place.

Comment 3

[David Teller [:Yoric] - still alive but not very active]

Attempting to add either RUST_PROGRAMS += ['binsource'] or HOST_RUST_PROGRAMS += ['binsource'] causes the following error:

KeyError: u'RUST_TARGET'

:/

Comment 4

[David Teller [:Yoric] - still alive but not very active]

Attachment: Bug 1454613 - Build binsource when building SpiderMonkey

Review commit: https://reviewboard.mozilla.org/r/239252/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/239252/

Comment 5

[David Teller [:Yoric] - still alive but not very active]

Full error, during `./mach configure`:

 0:11.90 Creating config.status
 0:12.32 Reticulating splines...
 0:12.52 Traceback (most recent call last):
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/configure.py", line 127, in <module>
 0:12.52     sys.exit(main(sys.argv))
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/configure.py", line 34, in main
 0:12.52     return config_status(config)
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/configure.py", line 122, in config_status
 0:12.52     return config_status(args=[], **encode(sanitized_config, encoding))
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/python/mozbuild/mozbuild/config_status.py", line 144, in config_status
 0:12.52     definitions = list(definitions)
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/python/mozbuild/mozbuild/frontend/emitter.py", line 184, in emit
 0:12.52     objs = list(emitfn(out))
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/python/mozbuild/mozbuild/frontend/emitter.py", line 1148, in emit_from_context
 0:12.52     for obj in self._handle_linkables(context, passthru, generated_files):
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/python/mozbuild/mozbuild/frontend/emitter.py", line 619, in _handle_linkables
 0:12.52     self._binaries[program] = cls(context, program, cargo_file)
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/python/mozbuild/mozbuild/frontend/data.py", line 562, in __init__
 0:12.52     cargo_dir = cargo_output_directory(context, self.TARGET_SUBST_VAR)
 0:12.52   File "/Volumes/Code/dteller/mc-binjs/python/mozbuild/mozbuild/frontend/data.py", line 543, in cargo_output_directory
 0:12.52     return mozpath.join(context.config.substs[target_var], rust_build_kind)
 0:12.52 KeyError: u'RUST_TARGET'
 0:12.58 *** Fix above errors and then restart with               "/Applications/Xcode.app/Contents/Developer/usr/bin/make -f client.mk build"
 0:12.58 make: *** [configure] Error 1

See patch https://bugzilla.mozilla.org/attachment.cgi?id=8970485

Ted, any suggestion?

Comment 6

[(not currently active) Ted Mielczarek]

I think this just needs the patch from bug 1444141 in order to work.

Comment 7

[Jason Orendorff [:jorendorff]]

Now, with David's patch, mach configure gives me this:

...
 0:19.21 js/src> Creating config.status
 0:19.44 Creating config.status
 0:19.77 Reticulating splines...
 0:19.87 Traceback (most recent call last):
 0:19.87   File "/Users/jorendorff/work/gecko/configure.py", line 123, in <module>
 0:19.87     sys.exit(main(sys.argv))
 0:19.87   File "/Users/jorendorff/work/gecko/configure.py", line 34, in main
 0:19.87     return config_status(config)
 0:19.87   File "/Users/jorendorff/work/gecko/configure.py", line 118, in config_status
 0:19.87     return config_status(args=[], **encode(sanitized_config, encoding))
 0:19.87   File "/Users/jorendorff/work/gecko/python/mozbuild/mozbuild/config_status.py", line 143, in config_status
 0:19.87     definitions = list(definitions)
 0:19.87   File "/Users/jorendorff/work/gecko/python/mozbuild/mozbuild/frontend/emitter.py", line 184, in emit
 0:19.87     objs = list(emitfn(out))
 0:19.87   File "/Users/jorendorff/work/gecko/python/mozbuild/mozbuild/frontend/emitter.py", line 1196, in emit_from_context
 0:19.88     for obj in self._handle_linkables(context, passthru, generated_files):
 0:19.88   File "/Users/jorendorff/work/gecko/python/mozbuild/mozbuild/frontend/emitter.py", line 971, in _handle_linkables
 0:19.88     linkable.sources[suffix] += srcs
 0:19.88 AttributeError: 'RustProgram' object has no attribute 'sources'
 0:19.92 *** Fix above errors and then restart with\
 0:19.92                "./mach build"
 0:19.92 gmake: *** [client.mk:115: configure] Error 1

Comment 8

[(not currently active) Ted Mielczarek]

Ah, you're going to need a separate moz.build file for binsource. Unfortunately our build system has never supported building multiple independent targets per directory, so it's getting confused and trying to associate the SOURCES from js/src with the binsource binary.

Comment 9

[Jason Orendorff [:jorendorff]]

That sounds kind of hopeful, though. Thanks.

David, can you take this?

Comment 10

[David Teller [:Yoric] - still alive but not very active]

I can take it, but can I put it very far on my backlog?

Comment 11

[Jason Orendorff [:jorendorff]]

Sure. This isn't drop-everything urgent; it's just one more hard requirement (among many) if we ever want to ship this thing.

Quoting bug 1497378 comment 3:

> Friendly reminder: BinAST isn't going to ship without at least a test
> that ensures we can always build the `-auto` generated files from their
> sources. David and I agreed to this months ago, but the build system is
> still enough of an obstacle that it hasn't gotten done; see bug 1454613.
> 
> The test approach is what we do for the HTML parser. It's also
> autogenerated, and also a pain to build when cross-compiling, so the
> generated files are in-tree, and we have a test that makes sure they can
> be regenerated and that the generated code is up-to-date.
> 
> Just so no one is surprised when it comes time to flip the pref, or turn
> on JS_BUILD_BINAST outside nightly: it's a hard requirement.

Comment 12

[(not currently active) Ted Mielczarek]

Attachment: bug 1454613 - build binsource when building SpiderMonkey. r?build

Comment 13

[(not currently active) Ted Mielczarek]

With the patch I just submitted it seems to build OK locally:
lrwxrwxrwx 1 luser luser 72 Nov  6 10:45 /build/debug-mozilla-central/dist/bin/binsource -> ../../js/src/frontend/binsource/x86_64-unknown-linux-gnu/debug/binsource

luser@eye7:/build/mozilla-central$ /build/debug-mozilla-central/dist/bin/binsource 
error: The following required arguments were not provided:
    <INPUT.webidl>
    <INPUT.yaml>
    --out-class <OUT_HEADER_CLASS_FILE>
    --out-impl <OUT_IMPL_FILE>
    --out-token <OUT_TOKEN_FILE>

USAGE:
    binsource <INPUT.webidl> <INPUT.yaml> --out-class <OUT_HEADER_CLASS_FILE> --out-impl <OUT_IMPL_FILE> --out-token <OUT_TOKEN_FILE>

For more information try --help

Comment 14

[(not currently active) Ted Mielczarek]

Yoric: should we land this patch? It'll make us build this binary, but doesn't add any checks for the generated code.

Comment 15

[David Teller [:Yoric] - still alive but not very active]

(In reply to Ted Mielczarek [:ted] [:ted.mielczarek] from comment #14)
> Yoric: should we land this patch? It'll make us build this binary, but
> doesn't add any checks for the generated code.

Let's do it.

Comment 16

[(not currently active) Ted Mielczarek]

I'll give it a run past try to make sure it isn't going to break in any strange environments. Also: I just realized that this probably needs to go in `HOST_RUST_PROGRAMS` since it's going to be used as part of the build eventually.

Comment 17

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e7148bd1af1eb8efe098cd4da65b06813fb52c0e

Comment 18

[(not currently active) Ted Mielczarek]

OK, that looks good.

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d3468dd807eb
build binsource when building SpiderMonkey. r=firefox-build-system-reviewers,chmanchester

Comment 20

[Cosmin Sabou [:CosminS]]

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&fromchange=d3468dd807eb970c794257cde9f48650bbf7d7bd&tochange=4dcf88b1fabcfb7fb0e4c87aec107f09e81a200b&searchStr=btup&selectedJob=212295179

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=212295179&repo=autoland&lineNumber=1663

Backout link: https://hg.mozilla.org/integration/autoland/rev/4dcf88b1fabcfb7fb0e4c87aec107f09e81a200b

[task 2018-11-16T20:22:40.193Z] 20:22:40     INFO -  Reticulating splines...
[task 2018-11-16T20:22:42.036Z] 20:22:42     INFO -   0:01.90 File already read. Skipping: /builds/worker/workspace/build/src/gfx/angle/targets/angle_common/moz.build
[task 2018-11-16T20:22:51.755Z] 20:22:51     INFO -   0:11.62 /builds/worker/workspace/build/src/rustc/bin/cargo build --build-plan -Z unstable-options --release --frozen --manifest-path /builds/worker/workspace/build/src/js/src/rust/Cargo.toml --lib --target=x86_64-unknown-linux-gnu
[task 2018-11-16T20:22:52.180Z] 20:22:52    ERROR -  Traceback (most recent call last):
[task 2018-11-16T20:22:52.180Z] 20:22:52     INFO -    File "/builds/worker/workspace/build/src/configure.py", line 123, in <module>
[task 2018-11-16T20:22:52.180Z] 20:22:52     INFO -      sys.exit(main(sys.argv))
[task 2018-11-16T20:22:52.180Z] 20:22:52     INFO -    File "/builds/worker/workspace/build/src/configure.py", line 34, in main
[task 2018-11-16T20:22:52.180Z] 20:22:52     INFO -      return config_status(config)
[task 2018-11-16T20:22:52.180Z] 20:22:52     INFO -    File "/builds/worker/workspace/build/src/configure.py", line 118, in config_status
[task 2018-11-16T20:22:52.180Z] 20:22:52     INFO -      return config_status(args=[], **encode(sanitized_config, encoding))
[task 2018-11-16T20:22:52.180Z] 20:22:52     INFO -    File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/config_status.py", line 146, in config_status
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -      the_backend.consume(definitions)
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -    File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/backend/base.py", line 128, in consume
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -      if (not self.consume_object(obj) and
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -    File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/backend/tup.py", line 664, in consume_object
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -      self._gen_rust_rules(obj, backend_file)
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -    File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/backend/tup.py", line 1027, in _gen_rust_rules
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -      cargo_plan = json.loads(''.join(output_lines))
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -    File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -      return _default_decoder.decode(s)
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -    File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -      obj, end = self.raw_decode(s, idx=_w(s, 0).end())
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -    File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -      raise ValueError("No JSON object could be decoded")
[task 2018-11-16T20:22:52.181Z] 20:22:52     INFO -  ValueError: No JSON object could be decoded
[task 2018-11-16T20:22:52.764Z] 20:22:52     INFO -  *** Fix above errors and then restart with\
[task 2018-11-16T20:22:52.764Z] 20:22:52     INFO -                 "./mach build"
[task 2018-11-16T20:22:52.764Z] 20:22:52     INFO -  client.mk:111: recipe for target 'configure' failed
[task 2018-11-16T20:22:52.764Z] 20:22:52     INFO -  make: *** [configure] Error 1
[task 2018-11-16T20:22:52.856Z] 20:22:52    ERROR - Return code: 2
[task 2018-11-16T20:22:52.856Z] 20:22:52  WARNING - setting return code to 2
[task 2018-11-16T20:22:52.856Z] 20:22:52    FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2018-11-16T20:22:52.856Z] 20:22:52    FATAL - Running post_fatal callback...
[task 2018-11-16T20:22:52.857Z] 20:22:52    FATAL - Exiting -1

Comment 21

[(not currently active) Ted Mielczarek]

I dug into this, I filed bug 1508344. I can work around that for now by fixing the Cargo.toml to not cause cargo to emit warnings.

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/43c0e876bc6f
build binsource when building SpiderMonkey. r=firefox-build-system-reviewers,chmanchester

Comment 23

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/43c0e876bc6f