thread 'webdriver dispatcher' panicked at 'attempted to leave type `linked_hash_map::Node<value::Value, value::Value>` uninitialized, which is invalid' when geckodriver built with Rust 1.49
Categories
(Testing :: geckodriver, defect, P2)
Tracking
(firefox86 fixed)
Tracking | Status | |
---|---|---|
firefox86 | --- | fixed |
People
(Reporter: whimboo, Assigned: whimboo)
References
Details
(Keywords: crash)
Attachments
(2 files)
Via bug 1680461 the version of Rust used in CI to build geckodriver has been reverted from 1.48 to 1.47 given that there were issues with the connection on Android.
https://treeherder.mozilla.org/logviewer?job_id=323400025&repo=fenix&lineNumber=1938-1941
[task 2020-12-03T05:53:30.919Z] 05:25:19 INFO - raptor-browsertime Info: firefox failed to start, trying 1 more time(s): ECONNREFUSED connect ECONNREFUSED 127.0.0.1:44402
We should make sure to use the latest available version of Rust to build geckodriver.
Assignee | ||
Comment 1•4 years ago
|
||
Updated•4 years ago
|
Assignee | ||
Comment 2•4 years ago
|
||
An update for Rust 1.49 just landed. So we will directly upgrade to the same version too.
Updated•4 years ago
|
Assignee | ||
Updated•4 years ago
|
Comment 4•4 years ago
•
|
||
Backed out changeset 931f753c12c0 (bug 1682219) for Android btime failures. CLOSED TREE
Log:
https://treeherder.mozilla.org/logviewer?job_id=326059767&repo=autoland&lineNumber=1949
Log:
https://treeherder.mozilla.org/logviewer?job_id=326059898&repo=autoland&lineNumber=1904
Push that introduced failures:
https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&revision=931f753c12c019a760ded711608d78388c20f304
Backout:
https://hg.mozilla.org/integration/autoland/rev/56c634741809de0e033f9a7f36b502808de2426a
Assignee | ||
Comment 5•4 years ago
|
||
Greg, how can we enable Marionette/geckodriver trace logs for BrowserTime jobs? Is there a way to pass -vv
to geckodriver, or set marionette.log.level=Trace
?
Assignee | ||
Comment 6•4 years ago
|
||
Also needinfo Andrew given that he also does a lot with BrowserTime.
Comment 7•4 years ago
|
||
Henrik, I believe that if you add the arguments, -vvvvv
here to browsertime, you will then get geckodriver logs:
I don't know if this is possible via the try command.
Assignee | ||
Comment 8•4 years ago
|
||
In line 250 I can see:
if self.verbose:
browsertime_options.append("-vvv")
I wonder what's the difference between that one and -vvvvv
.
Anyway here is the try build:
https://treeherder.mozilla.org/jobs?repo=try&revision=0ad35cce305c44e6fec735564cb347a9f04c0105
Assignee | ||
Comment 9•4 years ago
|
||
Ok, that was helpful. There is actually a crash in the Android handler:
https://treeherder.mozilla.org/logviewer?job_id=326428160&repo=try&lineNumber=2041-2078
thread 'webdriver dispatcher' panicked at 'attempted to leave type `linked_hash_map::Node<serde_yaml::Value, serde_yaml::Value>` uninitialized, which is invalid', /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/core/src/mem/mod.rs:659:9
stack backtrace:
0: 0x6684e0 - std::backtrace_rs::backtrace::libunwind::trace::hca287ea17d9489a6
at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/../../backtrace/src/backtrace/libunwind.rs:100:5
[..]
15: 0x6a7bad - core::panicking::panic::he533e234650a7fdb
at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/core/src/panicking.rs:50:5
16: 0x493011 - linked_hash_map::LinkedHashMap<K,V,S>::insert::h89d33b5337ef845e
17: 0x42db9d - geckodriver::android::AndroidHandler::prepare::haafa9a81713f08d8
18: 0x4d051c - geckodriver::marionette::MarionetteHandler::create_connection::hb5fdd326bb5f22e9
19: 0x4d3daa - <geckodriver::marionette::MarionetteHandler as webdriver::server::WebDriverHandler<geckodriver::command::GeckoExtensionRoute>>::handle_command::hd31696178a2c910d
So some call to LinkedHashMap.insert()
fails here under AndroidHandler::prepare()
. The only insertion that we do is in generate_config_file()
for the environment variables:
And the Browsertime jobs pass in "env":{"MOZ_WEBRENDER":"0"}
.
Assignee | ||
Comment 10•4 years ago
|
||
The problem here is with serde-yaml and it's Mapping class. The in-tree version is 0.8.9, while the latest is 0.8.15. Building geckodriver with that recent release fixes the problem.
This is a problem only happening in-tree, but not when users building geckodriver via the Github repository. So I will go ahead and try to vendor the Rust modules to get more up-to-date releases.
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Comment 11•4 years ago
|
||
Updating serde_yaml
results in:
cargo update -p serde_yaml
Updating crates.io index
Updating linked-hash-map v0.5.1 -> v0.5.4
Updating serde_yaml v0.8.9 -> v0.8.15
Updating yaml-rust v0.4.2 -> v0.4.5
The problem was actually in linked-hash-map (https://github.com/contain-rs/linked-hash-map/pull/96). As such I'm going to only update that particular crate.
Assignee | ||
Comment 12•4 years ago
|
||
Greg, I wonder why we didn't get the Rust traceback for the crash in the log output given that this is basically independent from the geckodriver log level. It would have been much easier to investigate and fix the problem.
Comment 13•4 years ago
|
||
I think that would be because of this which filters out lines that don't raptor
in them: https://searchfox.org/mozilla-central/source/testing/raptor/raptor/outputhandler.py#38
I've filed a bug about this (bug 1686230).
Assignee | ||
Comment 14•4 years ago
|
||
Comment 15•4 years ago
|
||
Comment 16•4 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/02913f00f2ef
https://hg.mozilla.org/mozilla-central/rev/7e1ed116d07f
Description
•