1632776 - autocfg tests don't work properly on Windows PGO builds

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: archaeopteryx [at] coole-files.de
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=299010711&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LMUl2YVwRpWiOAHFlwkBcg/runs/0/artifacts/public/logs/live_backing.log

---

This message is part of all successful Windows instrumentation tasks and Treeherder picks it up as failure line because it's logged with ERROR level. Can this be logged with a different level if it's expected and not fatal? Thank you.

[task 2020-04-23T12:55:07.892Z] 12:55:07 INFO - [hashbrown 0.7.1] cargo:rerun-if-changed=build.rs
[task 2020-04-23T12:55:07.892Z] 12:55:07 INFO - [hashbrown 0.7.1] error: Profile-guided optimization does not yet work in conjunction with -Cpanic=unwind on Windows when targeting MSVC. See https://github.com/rust-lang/rust/issues/61002 for details.
[task 2020-04-23T12:55:07.892Z] 12:55:07 INFO - [hashbrown 0.7.1]
[task 2020-04-23T12:55:07.892Z] 12:55:07 INFO - Compiling ffi-support v0.4.0
[task 2020-04-23T12:55:07.894Z] 12:55:07 INFO - Running CARGO_PKG_HOMEPAGE= CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION=0.4.0 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_REPOSITORY='https://github.com/mozilla/application-services' CARGO_PKG_VERSION_PRE= CARGO_PKG_NAME=ffi-support CARGO_PKG_AUTHORS='Thom Chiovoloni <[e-mail]>' CARGO=/builds/worker/fetches/rustc/bin/cargo CARGO_MANIFEST_DIR=/builds/worker/checkouts/gecko/third_party/rust/ffi-support CARGO_PKG_VERSION_MINOR=4 LD_LIBRARY_PATH='/builds/worker/workspace/obj-build/release/deps:/builds/worker/fetches/rustc/lib' CARGO_PKG_DESCRIPTION='A crate to help expose Rust functions over the FFI.' /builds/worker/fetches/rustc/bin/rustc --edition=2018 --crate-name ffi_support /builds/worker/checkouts/gecko/third_party/rust/ffi-support/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=2 -C panic=abort --cfg 'feature="default"' -C metadata=684000eb2e253f67 -C extra-filename=-684000eb2e253f67 --out-dir /builds/worker/workspace/obj-build/x86_64-pc-windows-msvc/release/deps --target x86_64-pc-windows-msvc -C linker=/builds/worker/checkouts/gecko/build/cargo-linker -L dependency=/builds/worker/workspace/obj-build/x86_64-pc-windows-msvc/release/deps -L dependency=/builds/worker/workspace/obj-build/release/deps --extern lazy_static=/builds/worker/workspace/obj-build/x86_64-pc-windows-msvc/release/deps/liblazy_static-bd6dd9879d308357.rmeta --extern log=/builds/worker/workspace/obj-build/x86_64-pc-windows-msvc/release/deps/liblog-5c89d94462311ce4.rmeta --cap-lints warn -C opt-level=2 -C debuginfo=2 -C force-frame-pointers=yes -Dwarnings -C profile-generate=/builds/worker/workspace/obj-build -C codegen-units=1
[task 2020-04-23T12:55:07.894Z] 12:55:07 ERROR - [hashbrown 0.7.1] error: aborting due to previous error
[task 2020-04-23T12:55:07.894Z] 12:55:07 INFO - [hashbrown 0.7.1]
[task 2020-04-23T12:55:07.894Z] 12:55:07 INFO - [hashbrown 0.7.1] error: Profile-guided optimization does not yet work in conjunction with -Cpanic=unwind on Windows when targeting MSVC. See https://github.com/rust-lang/rust/issues/61002 for details.
[task 2020-04-23T12:55:07.894Z] 12:55:07 INFO - [hashbrown 0.7.1]
[task 2020-04-23T12:55:07.894Z] 12:55:07 ERROR - [hashbrown 0.7.1] error: aborting due to previous error
[task 2020-04-23T12:55:07.894Z] 12:55:07 INFO - [hashbrown 0.7.1]
[task 2020-04-23T12:55:07.894Z] 12:55:07 INFO - [hashbrown 0.7.1] warning: autocfg could not probe for std

Comment 1

[Simon Sapin (:SimonSapin)]

Sorry, I don’t know much about PGO or why bug 1631721 would affect it. Redirecting per https://wiki.mozilla.org/Modules/Core#Build_Config

Comment 2

[Emilio Cobos Álvarez [:emilio]]

Huh, but we should not be building with panic=unwind...

Comment 3

[Simon Sapin (:SimonSapin)]

Ah, based on looking at the log with some context it appears that this message comes from the execution of hashbrown’s build script which uses the autocfg crate which in turns shells out to rustc to query its version number. It looks like that invocation of rustc is failing. Presumably that’s what prints the error message about PGO.

I don’t know what component turns this into an ERROR, though, since compilation of hashbrown and downstream crates seem to succeed in that log.

Comment 4

[Mike Hommey [:glandium]]

That hashbrown thing is a red herring.

Comment 5

[Mike Hommey [:glandium]]

Oh, missed the part where you're saying it's on green builds. Yeah, I've seen this today.

Comment 6

[Mike Hommey [:glandium]]

Presumably, autocfg is passing too many flags to rustc, one of which is the one for PGO, and thanks to the rust issue linked in the error message, rustc emits an error in that case.

Comment 7

[Mike Hommey [:glandium]]

And actually reading https://github.com/cuviper/autocfg/blob/1.0.0/src/version.rs#L27-L30 it seems it doesn't pass any. So this must come from RUSTC_FLAGS in the environment, that rustc doesn't ignore when passed --version.

Comment 8

[Simon Sapin (:SimonSapin)]

The RUSTFLAGS environment variable is read (and turned into explicit arguments) by Cargo, not rustc:

However, the log also contains a autocfg could not probe for `std` message which indicates that what fails is autocfg’s "sanity check" which does read RUSTFLAGS.

In that code we see that autocfg prints "warning: autocfg could not probe for `std`\n" to the build script’s stderr. Normally Cargo does not print stderr from build script to its own stdrr. But in the TC log that line is not only present, but prefixed by [task 2020-04-23T12:55:07.894Z] 12:55:07 INFO - [hashbrown 0.7.1]. What’s responsible for this? I something in Gecko’s build system actively parsing the $CARGO_TARGET_DIR/release/build/hashbrown-*/output file and looking for lines with "error" in it?

The first and last log lines quoted here are from the first and last meaningful source lines in https://github.com/rust-lang/hashbrown/blob/v0.7.1/build.rs The ffi-support bit is probably something unrelated running concurrently.

[task 2020-04-23T12:55:07.892Z] 12:55:07     INFO -  [hashbrown 0.7.1] cargo:rerun-if-changed=build.rs
[task 2020-04-23T12:55:07.892Z] 12:55:07     INFO -  [hashbrown 0.7.1] error: Profile-guided optimization does not yet work in conjunction with `-Cpanic=unwind` on Windows when targeting MSVC. See https://github.com/rust-lang/rust/issues/61002 for details.
[task 2020-04-23T12:55:07.892Z] 12:55:07     INFO -  [hashbrown 0.7.1]
[task 2020-04-23T12:55:07.892Z] 12:55:07     INFO -     Compiling ffi-support v0.4.0
[task 2020-04-23T12:55:07.894Z] 12:55:07     INFO -       Running `CARGO_PKG_HOMEPAGE= CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION=0.4.0 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_REPOSITORY='https://github.com/mozilla/application-services' CARGO_PKG_VERSION_PRE= CARGO_PKG_NAME=ffi-support CARGO_PKG_AUTHORS='Thom Chiovoloni <[e-mail]>' CARGO=/builds/worker/fetches/rustc/bin/cargo CARGO_MANIFEST_DIR=/builds/worker/checkouts/gecko/third_party/rust/ffi-support CARGO_PKG_VERSION_MINOR=4 LD_LIBRARY_PATH='/builds/worker/workspace/obj-build/release/deps:/builds/worker/fetches/rustc/lib' CARGO_PKG_DESCRIPTION='A crate to help expose Rust functions over the FFI.' /builds/worker/fetches/rustc/bin/rustc --edition=2018 --crate-name ffi_support /builds/worker/checkouts/gecko/third_party/rust/ffi-support/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=2 -C panic=abort --cfg 'feature="default"' -C metadata=684000eb2e253f67 -C extra-filename=-684000eb2e253f67 --out-dir /builds/worker/workspace/obj-build/x86_64-pc-windows-msvc/release/deps --target x86_64-pc-windows-msvc -C linker=/builds/worker/checkouts/gecko/build/cargo-linker -L dependency=/builds/worker/workspace/obj-build/x86_64-pc-windows-msvc/release/deps -L dependency=/builds/worker/workspace/obj-build/release/deps --extern lazy_static=/builds/worker/workspace/obj-build/x86_64-pc-windows-msvc/release/deps/liblazy_static-bd6dd9879d308357.rmeta --extern log=/builds/worker/workspace/obj-build/x86_64-pc-windows-msvc/release/deps/liblog-5c89d94462311ce4.rmeta --cap-lints warn -C opt-level=2 -C debuginfo=2 -C force-frame-pointers=yes -Dwarnings -C profile-generate=/builds/worker/workspace/obj-build -C codegen-units=1`
[task 2020-04-23T12:55:07.894Z] 12:55:07    ERROR -  [hashbrown 0.7.1] error: aborting due to previous error
[task 2020-04-23T12:55:07.894Z] 12:55:07     INFO -  [hashbrown 0.7.1]
[task 2020-04-23T12:55:07.894Z] 12:55:07     INFO -  [hashbrown 0.7.1] error: Profile-guided optimization does not yet work in conjunction with `-Cpanic=unwind` on Windows when targeting MSVC. See https://github.com/rust-lang/rust/issues/61002 for details.
[task 2020-04-23T12:55:07.894Z] 12:55:07     INFO -  [hashbrown 0.7.1]
[task 2020-04-23T12:55:07.894Z] 12:55:07    ERROR -  [hashbrown 0.7.1] error: aborting due to previous error
[task 2020-04-23T12:55:07.894Z] 12:55:07     INFO -  [hashbrown 0.7.1]
[task 2020-04-23T12:55:07.894Z] 12:55:07     INFO -  [hashbrown 0.7.1] warning: autocfg could not probe for `std`
[task 2020-04-23T12:55:07.894Z] 12:55:07     INFO -  [hashbrown 0.7.1] cargo:rustc-cfg=has_extern_crate_alloc

Comment 9

[Mike Hommey [:glandium]]

cargo does print build script output with -v.

Comment 10

[Simon Sapin (:SimonSapin)]

TIL. When I try it I see lines such as:

[hashbrown 0.7.1] cargo:rustc-cfg=has_extern_crate_alloc

What component is responsible for classifying lines as INFO v.s. ERROR? The bug description seems to be about that:

Can this be logged with a different level if it's expected and not fatal?

Comment 11

[Mike Hommey [:glandium]]

https://searchfox.org/mozilla-central/rev/b8fbb6ead517720daf0b0211115f407b4b951c74/testing/mozharness/mozharness/base/errors.py#89

But that can't be changed, because that is how actual errors show up too.

Comment 12

[Mike Hommey [:glandium]]

Very clearly, there's a bug in autocfg that should be fixed. Also, hashbrown could switch to rustc_version instead.

Comment 13

[Intermittent Failures Robot]

2 failures in 5582 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• windows2012-64-shippable: 1
• windows2012-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1632776&startday=2020-04-20&endday=2020-04-26&tree=all

Comment 14

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1631721

Comment 15

[Intermittent Failures Robot]

1 failures in 5423 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• windows2012-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1632776&startday=2020-04-27&endday=2020-05-03&tree=all

Comment 16

[Intermittent Failures Robot]

1 failures in 5276 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• windows2012-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1632776&startday=2020-05-04&endday=2020-05-10&tree=all

Comment 17

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 19

[Mike Hommey [:glandium]]

Attachment: Bug 1632776 - Work around autocfg tests failing on Windows PGO builds.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f1c0d2ec9876
Work around autocfg tests failing on Windows PGO builds. r=firefox-build-system-reviewers,sheehan,mhentges

Comment 21

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/f1c0d2ec9876