1407687 - Intermittent timeouts in linux rusttests tasks

Status: RESOLVED WORKSFORME

(Firefox Build System :: General, defect)

Description

[Geoff Brown [:gbrown]]

In bug 1204281 there are reports of timeouts in linux32-rusttests and linux64-rusttests.

https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=135749181&lineNumber=32774

[task 2017-10-09T18:48:30.672Z] 18:48:30     INFO - note: library: m
[task 2017-10-09T18:48:30.672Z] 18:48:30     INFO - note: library: rt
[task 2017-10-09T18:48:30.672Z] 18:48:30     INFO - note: library: pthread
[task 2017-10-09T18:48:30.672Z] 18:48:30     INFO - note: library: util
[task 2017-10-09T18:48:32.190Z] 18:48:32     INFO -      Running `/builds/worker/workspace/build/src/rustc/bin/rustc --crate-name stylo_tests /builds/worker/workspace/build/src/servo/tests/unit/stylo/lib.rs --emit=dep-info,link -C opt-level=3 --test -C metadata=c9082d76cfaf8a76 -C extra-filename=-c9082d76cfaf8a76 --out-dir /builds/worker/workspace/build/src/obj-firefox/toolkit/library/i686-unknown-linux-gnu/release/deps --target i686-unknown-linux-gnu -C linker=/builds/worker/workspace/build/src/build/cargo-linker -L dependency=/builds/worker/workspace/build/src/obj-firefox/toolkit/library/i686-unknown-linux-gnu/release/deps -L dependency=/builds/worker/workspace/build/src/obj-firefox/toolkit/library/release/deps --extern env_logger=/builds/worker/workspace/build/src/obj-firefox/toolkit/library/i686-unknown-linux-gnu/release/deps/libenv_logger-40b33462646310da.rlib --extern euclid=/builds/worker/workspace/build/src/obj-firefox/toolkit/library/i686-unknown-linux-gnu/release/deps/libeuclid-53520cc4344e88fc.rlib --extern cssparser=/builds/worker/workspace/build/src/obj-firefox/toolkit/library/i686-unknown-linux-gnu/release/deps/libcssparser-049e8fc1533dc8df.rlib --extern size_of_test=/builds/worker/workspace/build/src/obj-firefox/toolkit/library/i686-unknown-linux-gnu/release/deps/libsize_of_test-53dc78ce558d81b4.rlib --extern malloc_size_of=/builds/worker/workspace/build/src/obj-firefox/toolkit/library/i686-unknown-linux-gnu/release/deps/libmalloc_size_of-0ae69d75e21b4bfb.rlib --extern geckoservo=/builds/worker/workspace/build/src/obj-firefox/toolkit/library/i686-unknown-linux-gnu/release/deps/libgeckoservo-f7ee3e90a6761274.rlib --extern style_traits=/builds/worker/workspace/build/src/obj-firefox/toolkit/library/i686-unknown-linux-gnu/release/deps/libstyle_traits-c5400bc0cc2f4476.rlib --extern log=/builds/worker/workspace/build/src/obj-firefox/toolkit/library/i686-unknown-linux-gnu/release/deps/liblog-fd2fff032e50a631.rlib --extern selectors=/builds/worker/workspace/build/src/obj-firefox/toolkit/library/i686-unknown-linux-gnu/release/deps/libselectors-ade130cf9af8dd02.rlib --extern smallvec=/builds/worker/workspace/build/src/obj-firefox/toolkit/library/i686-unknown-linux-gnu/release/deps/libsmallvec-9896d5f32c80d5c2.rlib --extern libc=/builds/worker/workspace/build/src/obj-firefox/toolkit/library/i686-unknown-linux-gnu/release/deps/liblibc-9a1c8ca7327d4e94.rlib --extern atomic_refcell=/builds/worker/workspace/build/src/obj-firefox/toolkit/library/i686-unknown-linux-gnu/release/deps/libatomic_refcell-b0459433c291d7e6.rlib --extern style=/builds/worker/workspace/build/src/obj-firefox/toolkit/library/i686-unknown-linux-gnu/release/deps/libstyle-e7eb6d1272252479.rlib -C debuginfo=2`

[taskcluster:error] Task timeout after 3600 seconds. Force killing container.
[taskcluster 2017-10-09 18:48:44.699Z] === Task Finished ===
[taskcluster 2017-10-09 18:48:44.700Z] Unsuccessful task run with exit code: -1 completed in 3603.589 seconds

Comment 1

[Geoff Brown [:gbrown]]

Other examples:

https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=135794039&lineNumber=33655
https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=135897400&lineNumber=32844
https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=136041990&lineNumber=33113
https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=136118806&lineNumber=31239

...and more in bug 1204281.

Comment 2

[Geoff Brown [:gbrown]]

If these tasks just need more time to run, that's easy to modify:

https://dxr.mozilla.org/mozilla-central/rev/a0488ecc201c04f2617e7b02f039344e8fbf0d9a/taskcluster/ci/build/linux.yml#317

but I'm not sure if that's appropriate. It looks like these tasks often run in ~30 minutes.

Nathan -- Can you have a look?

Comment 3

[Nathan Froyd [:froydnj]]

We could bump up the amount of time they need to run...but I'm unsure what would cause the execution time to double.  Oh, maybe if sccache isn't hitting in the cache and we wind up having to compile *all* the C++ code (or at least a significant fraction of it) + the Rust code for libxul + the Rust code for the tests...that would definitely make things run more slowly.

This also indicates that it'd be really nice if we were able to run the normal builds, and then package up the objdirs and ship them somewhere so we could just run the Rust compilation for the tests as a separate job.

Bumping the timeout here to 5400 seconds or even 7200 should be sufficient.  Do you want to do that or shall I?

Comment 4

[(not currently active) Ted Mielczarek]

If we made the rusttest builds dependent on the normal builds, would that help us get sccache cache hits? I assume most of the crates we wind up building will be the same between both builds, and it's just the actual crates we're testing that will get built in the test configuration.

Comment 5

[Nathan Froyd [:froydnj]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #4)
> If we made the rusttest builds dependent on the normal builds, would that
> help us get sccache cache hits? I assume most of the crates we wind up
> building will be the same between both builds, and it's just the actual
> crates we're testing that will get built in the test configuration.

Yes, that would help a lot.  How would we accomplish that?  Would that even ensure that the objdir from the normal build gets reused for the rusttests build?

Comment 6

[Geoff Brown [:gbrown]]

I'll bump the max run time to 5400, at least for short-term help. Thanks for looking at the bigger issue.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8d668d63bfe0
Increase max run time for linux rusttests; r=me,test-only

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/8d668d63bfe0

Comment 9

[(not currently active) Ted Mielczarek]

(In reply to Nathan Froyd [:froydnj] from comment #5)
> Yes, that would help a lot.  How would we accomplish that?  Would that even
> ensure that the objdir from the normal build gets reused for the rusttests
> build?

You'd have to fiddle with the task definitions so that the rusttest tasks depend on the build tasks. It would not allow us to reuse objdirs AFAIK, but it should allow you to get sccache hits for everything that was built during the build.

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:kmoir, maybe it's time to close this bug?

Comment 11

[Kim Moir [:kmoir] ET]

:gbrown is this bug still an issue or can it be closed?  It currently has the leave-open keyword so it isn't being closed.

Comment 12

[Geoff Brown [:gbrown]]

I think this is okay now. I don't see recent failures on mozilla-central.