Perma Infra WR(wrench-deps) - failed to download from `https://crates.io/api/v1/crates
Categories
(Firefox Build System :: Toolchains, defect)
Tracking
(firefox-esr68 fixed, firefox75 fixed)
People
(Reporter: NarcisB, Assigned: emilio)
Details
Attachments
(1 file)
Deps are perma failing on on b-linux / gecko-3
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=292102914&repo=autoland&lineNumber=413
Log snippet:
[task 2020-03-07T11:36:04.978Z] warning: spurious network error (2 tries remaining): [28] Timeout was reached (failed to download any data for block v0.1.6 within 30s)
[task 2020-03-07T11:36:34.978Z] warning: spurious network error (2 tries remaining): [28] Timeout was reached (download of lazy_static v0.2.11 failed to transfer more than 10 bytes in 30s)
[task 2020-03-07T11:36:34.979Z] warning: spurious network error (2 tries remaining): [28] Timeout was reached (failed to download any data for x11-dl v2.18.3 within 30s)
[task 2020-03-07T11:37:04.991Z] error: failed to sync
[task 2020-03-07T11:37:04.991Z]
[task 2020-03-07T11:37:04.991Z] Caused by:
[task 2020-03-07T11:37:04.991Z] failed to download from https://crates.io/api/v1/crates/kernel32-sys/0.2.2/download
[task 2020-03-07T11:37:04.991Z]
[task 2020-03-07T11:37:04.991Z] Caused by:
[task 2020-03-07T11:37:04.991Z] [55] Failed sending data to the peer (SSL_write() error: error:1409E10F:SSL routines:ssl3_write_bytes:bad length)
[fetches 2020-03-07T11:37:04.992Z] removing /builds/worker/fetches
[fetches 2020-03-07T11:37:05.096Z] finished
[taskcluster 2020-03-07 11:37:05.480Z] === Task Finished ===
[taskcluster 2020-03-07 11:37:05.598Z] Artifact "public/build" not found at "/builds/worker/artifacts/"
[taskcluster 2020-03-07 11:37:06.183Z] Unsuccessful task run with exit code: 101 completed in 1102.424 seconds
|
Reporter | |
Updated•5 years ago
|
|
||
Comment 1•5 years ago
|
||
I think Kats was looking at this.
|
||
Comment 2•5 years ago
|
||
Worth mentioning that autoland is closed since 11:44:11 UTC because of these bustages.
|
Assignee | |
Comment 3•5 years ago
|
||
I can repro this on my machine... it seems like cargo-vendor tries to download 212 crates at the same time and it fails awfully. Retrying it works.
|
|
Assignee | |
Comment 4•5 years ago
|
||
This is a bug in cargo-vendor or crates.io. It tries to download everything at the same time and crates.io is choking at that.
Regular cargo vendor (no cargo-vendor vendor) works, and we should probably use it, but it doesn't support the --relative-path flag.
|
|
Assignee | |
Comment 5•5 years ago
|
||
I don't know why that'd be needed though.
|
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Comment 6•5 years ago
|
||
We were using the old cargo-vendor, which tries to fetch dependencies in a way
that crates.io dislikes, leading to spurious timeouts.
Use the built-in command instead. We no longer need the --relative-path flag
because the built-in one uses a relative path by default.
|
|
Assignee | |
Comment 7•5 years ago
|
||
|
|
Assignee | |
Comment 8•5 years ago
|
||
I talked with the sheriffs, and I'm going to ask for forgiveness rather than permission for this one :)
|
|
||
Comment 10•5 years ago
|
||
Patch holds, re-opened autoland: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=success%2Cpending%2Crunning%2Ctestfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=d2ac41047c108daa9a8933fa53af59d695063d6f&searchStr=toolchains&selectedJob=292123260
Thanks Emilio.
|
||
Comment 11•5 years ago
|
||
| bugherder | ||
|
||
Comment 12•5 years ago
|
||
Sorry I'm so late to the party here! Thanks Emilio for figuring this out and landing a fix. Patch looks good to me. Also /cc jnicol who was seeing this problem in his try pushes on Friday.
|
||
Comment 13•5 years ago
|
||
The reason this started happening is because of the release of a new curl-sys release using a new version of curl, see https://github.com/rust-lang/cargo/issues/7974
So the reason the fix worked is that it stopped building the third party cargo vendor, so cargo vendor ends up not using that new version of curl-sys.
This is at least the second time we're hit by some jobs that build rust not having their own Cargo.lock...
|
|
||
Comment 14•5 years ago
|
||
(In reply to Mike Hommey [:glandium] from comment #13)
This is at least the second time we're hit by some jobs that build rust not having their own Cargo.lock...
Just to be clear, the thing missing the Cargo.lock file is the cargo-vendor crate published to crates.io. If that had a lock file that froze the version of curl-sys being used this wouldn't have been a problem. And the cargo-vendor source repository has a Cargo.lock file checked in, so I assume it's the cargo publish process that skips uploading the Cargo.lock file. So this would be a bug in the crate publishing infrastructure that results in cargo install producing different results at different times, even with a pinned version.
|
||
Comment 15•5 years ago
|
||
| bugherder uplift | ||
https://hg.mozilla.org/releases/mozilla-esr68/rev/00a7aedc07c3
FIREFOX_ESR_68_6_X_RELBRANCH: https://hg.mozilla.org/releases/mozilla-esr68/rev/298575e503f944f0ba7f8f4975dcca215682b445
THUNDERBIRD_68_VERBRANCH: https://hg.mozilla.org/releases/mozilla-esr68/rev/2a6f2ff09c34eb40fdf77cec2af1176d5c96a467
|
|
||
Comment 16•5 years ago
•
|
||
| uplift | ||
Fixed merge/uplift conflict:
https://hg.mozilla.org/releases/mozilla-esr68/rev/249678c4a9d464c0663ac783f210f8e01289b431
FIREFOX_ESR_68_6_X_RELBRANCH: https://hg.mozilla.org/releases/mozilla-esr68/rev/198e5bfb6b8786438cd2b598200aa0adec86a94e
THUNDERBIRD_68_VERBRANCH: https://hg.mozilla.org/releases/mozilla-esr68/rev/85f60df596c611284330d787556e35361363a0ca
|
||
Updated•5 years ago
|
Description
•