Closed Bug 1469815 Opened 2 years ago Closed Last year

Intermittent 'mach build -v' did not run successfully. Please check log for errors.

Categories

(Firefox Build System :: General, defect, P5)

3 Branch
defect

Tracking

(firefox62 fixed)

RESOLVED FIXED
mozilla62
Tracking Status
firefox62 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: chmanchester)

References

Details

Attachments

(1 file)

Filed by: aciure [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=183943943&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/YeLc-29mR7y_dvDOuRaJtA/runs/0/artifacts/public/logs/live_backing.log

[task 2018-06-20T09:41:31.845Z] 09:41:31     INFO -   *** tup: 1 job failed.
[task 2018-06-20T09:41:31.906Z] 09:41:31     INFO -  140 compiler warnings present.
[task 2018-06-20T09:41:31.982Z] 09:41:31    ERROR - Return code: 1
[task 2018-06-20T09:41:31.982Z] 09:41:31  WARNING - setting return code to 2
[task 2018-06-20T09:41:31.982Z] 09:41:31    FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2018-06-20T09:41:31.983Z] 09:41:31    FATAL - Running post_fatal callback...
[task 2018-06-20T09:41:31.983Z] 09:41:31    FATAL - Exiting -1
[task 2018-06-20T09:41:31.983Z] 09:41:31     INFO - [mozharness: 2018-06-20 09:41:31.983194Z] Finished build step (failed)
[task 2018-06-20T09:41:31.983Z] 09:41:31     INFO - Running post-run listener: _summarize
[task 2018-06-20T09:41:31.983Z] 09:41:31    ERROR - # TBPL FAILURE #
[task 2018-06-20T09:41:31.983Z] 09:41:31     INFO - [mozharness: 2018-06-20 09:41:31.983446Z] FxDesktopBuild summary:
[task 2018-06-20T09:41:31.983Z] 09:41:31    ERROR - # TBPL FAILURE #
[task 2018-06-20T09:41:31.983Z] 09:41:31     INFO - Running post-run listener: copy_logs_to_upload_dir
[task 2018-06-20T09:41:31.983Z] 09:41:31     INFO - Copying logs to upload dir...
[task 2018-06-20T09:41:31.983Z] 09:41:31     INFO - mkdir: /builds/worker/workspace/build/upload/logs
[task 2018-06-20T09:41:31.983Z] 09:41:31    DEBUG - Copying /builds/worker/logs/localconfig.json to /builds/worker/workspace/build/upload/logs/localconfig.json
[task 2018-06-20T09:41:31.984Z] 09:41:31    DEBUG - mkdir_p: /builds/worker/workspace/build/upload/logs Already exists.
[task 2018-06-20T09:41:31.984Z] 09:41:31    DEBUG - Copying /builds/worker/logs/log_info.log to /builds/worker/workspace/build/upload/logs/log_info.log
[task 2018-06-20T09:41:31.984Z] 09:41:31    DEBUG - mkdir_p: /builds/worker/workspace/build/upload/logs Already exists.
[task 2018-06-20T09:41:31.984Z] 09:41:31    DEBUG - Copying /builds/worker/logs/log_raw.log to /builds/worker/workspace/build/upload/logs/log_raw.log
[task 2018-06-20T09:41:31.985Z] 09:41:31    DEBUG - mkdir_p: /builds/worker/workspace/build/upload/logs Already exists.
[task 2018-06-20T09:41:31.985Z] 09:41:31    DEBUG - Copying /builds/worker/logs/log_warning.log to /builds/worker/workspace/build/upload/logs/log_warning.log
[task 2018-06-20T09:41:31.985Z] 09:41:31    DEBUG - mkdir_p: /builds/worker/workspace/build/upload/logs Already exists.
[task 2018-06-20T09:41:31.985Z] 09:41:31    DEBUG - Copying /builds/worker/logs/log_critical.log to /builds/worker/workspace/build/upload/logs/log_critical.log
[task 2018-06-20T09:41:31.985Z] 09:41:31    DEBUG - mkdir_p: /builds/worker/workspace/build/upload/logs Already exists.
[task 2018-06-20T09:41:31.985Z] 09:41:31    DEBUG - Copying /builds/worker/logs/log_error.log to /builds/worker/workspace/build/upload/logs/log_error.log
[task 2018-06-20T09:41:31.985Z] 09:41:31    DEBUG - mkdir_p: /builds/worker/workspace/build/upload/logs Already exists.
[task 2018-06-20T09:41:31.985Z] 09:41:31    DEBUG - Copying /builds/worker/logs/log_debug.log to /builds/worker/workspace/build/upload/logs/log_debug.log
[task 2018-06-20T09:41:31.986Z] 09:41:31    DEBUG - mkdir_p: /builds/worker/workspace/build/upload/logs Already exists.
[task 2018-06-20T09:41:31.986Z] 09:41:31    DEBUG - Copying /builds/worker/logs/log_fatal.log to /builds/worker/workspace/build/upload/logs/log_fatal.log
The tup error lines are:

[task 2018-06-20T09:40:50.902Z] 09:40:50     INFO -  tup error: Expected node 'shorthands' to be in directory 'obj-firefox/toolkit/library/x86_64-unknown-linux-gnu/release/build/style-04158a1f19297f56/out', but it is not there.
[task 2018-06-20T09:40:50.902Z] 09:40:50     INFO -  tup error: File '/builds/worker/workspace/build/src/obj-firefox/toolkit/library/x86_64-unknown-linux-gnu/release/build/style-04158a1f19297f56/out/shorthands/margin.rs' was written to, but is not in .tup/db. You probably should specify it as an output
[task 2018-06-20T09:40:50.902Z] 09:40:50     INFO -   *** Command failed due to errors processing the output dependencies.

This is not intermittent. We might need to update python/mozbuild/mozbuild/backend/cargo_build_defs.py
Assignee: nobody → cmanchester
Comment on attachment 8986486 [details]
Bug 1469815 - Add new output files from style crate build script to cargo_build_defs.py for the Tup build.

https://reviewboard.mozilla.org/r/251820/#review258260

LGTM! It's unfortunate that this is now a fairly large list - seems like it might be the kind of thing that changes (and breaks the tup backend) more frequently. Perhaps we need to come up with the proposal to annotate these in the crates themselves sooner rather than later. Though I'm not sure of a good way to verify that when a crate is updated without actually having tup build it.
Attachment #8986486 - Flags: review?(mshal) → review+
Pushed by cmanchester@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3b66a441d832
Add new output files from style crate build script to cargo_build_defs.py for the Tup build. r=mshal
Perhaps we could have a simple check like "if a crate has a build script, it must have an entry in this list"? That ought to let us do a simple toml parse of all the Cargo.toml files for in-tree crates to check for build scripts. That could be in the build frontend or a standalone Python test.

There might be some build scripts that do work that doesn't require providing additional info to tup, which is OK, they'd just have to get an empty entry in the list. That would mean that at least someone had to manually look at it.

That still won't fix the case where a crate gets updated to a newer version and the build script does something new that's not in the list. Obviously the only sustainable long-term option is to get this info into Cargo.toml or something like that.
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #5)
> That still won't fix the case where a crate gets updated to a newer version
> and the build script does something new that's not in the list. Obviously
> the only sustainable long-term option is to get this info into Cargo.toml or
> something like that.

I guess that's what I'm wondering - if we get this in Cargo.toml or somewhere, would cargo/rust be able to check the output list against what the build.rs script actually does before we run it in the tup backend? Eg: by having cargo wrap open() somehow when running them, or via cargo check or whatever.
https://hg.mozilla.org/mozilla-central/rev/3b66a441d832
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
(In reply to Michael Shal [:mshal] from comment #6)
> I guess that's what I'm wondering - if we get this in Cargo.toml or
> somewhere, would cargo/rust be able to check the output list against what
> the build.rs script actually does before we run it in the tup backend? Eg:
> by having cargo wrap open() somehow when running them, or via cargo check or
> whatever.

It's a good question. I guess you could imagine implementing the execution of build scripts with some sort of sandboxing such that they can only touch the files they've declared they will, but I don't know how practical that'd be in reality.
Recent failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=187829637&repo=autoland&lineNumber=5287

 ERROR - Return code: 1
[task 2018-07-12T13:32:21.863Z] 13:32:21  WARNING - setting return code to 2
[task 2018-07-12T13:32:21.863Z] 13:32:21    FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2018-07-12T13:32:21.863Z] 13:32:21    FATAL - Running post_fatal callback...
[task 2018-07-12T13:32:21.863Z] 13:32:21    FATAL - Exiting -1
[task 2018-07-12T13:32:21.863Z] 13:32:21     INFO - [mozharness: 2018-07-12 13:32:21.863444Z] Finished build step (failed)
[task 2018-07-12T13:32:21.863Z] 13:32:21     INFO - Running post-run listener: _summarize
[task 2018-07-12T13:32:21.863Z] 13:32:21    ERROR - # TBPL FAILURE #
[task 2018-07-12T13:32:21.863Z] 13:32:21     INFO - [mozharness: 2018-07-12 13:32:21.863659Z] FxDesktopBuild summary:
[task 2018-07-12T13:32:21.863Z] 13:32:21    ERROR - # TBPL FAILURE #
[task 2018-07-12T13:32:21.863Z] 13:32:21     INFO - Running post-run listener: copy_logs_to_upload_dir
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Eliza Balazs [:ebalazs_] from comment #11)

A new bug has been created for that issue and it's being tracked here: https://bugzilla.mozilla.org/show_bug.cgi?id=1475278
Status: REOPENED → RESOLVED
Closed: 2 years agoLast year
Resolution: --- → FIXED
Version: Version 3 → 3 Branch
You need to log in before you can comment on or make changes to this bug.