Closed Bug 1682219 Opened 4 years ago Closed 4 years ago

thread 'webdriver dispatcher' panicked at 'attempted to leave type `linked_hash_map::Node<value::Value, value::Value>` uninitialized, which is invalid' when geckodriver built with Rust 1.49

Categories

(Testing :: geckodriver, defect, P2)

Default
defect

Tracking

(firefox86 fixed)

RESOLVED FIXED
86 Branch
Tracking Status
firefox86 --- fixed

People

(Reporter: whimboo, Assigned: whimboo)

References

Details

(Keywords: crash)

Attachments

(2 files)

Via bug 1680461 the version of Rust used in CI to build geckodriver has been reverted from 1.48 to 1.47 given that there were issues with the connection on Android.

https://treeherder.mozilla.org/logviewer?job_id=323400025&repo=fenix&lineNumber=1938-1941

[task 2020-12-03T05:53:30.919Z] 05:25:19 INFO - raptor-browsertime Info: firefox failed to start, trying 1 more time(s): ECONNREFUSED connect ECONNREFUSED 127.0.0.1:44402

We should make sure to use the latest available version of Rust to build geckodriver.

Assignee: nobody → hskupin
Status: NEW → ASSIGNED

An update for Rust 1.49 just landed. So we will directly upgrade to the same version too.

Summary: Build geckodriver with latest Rust 1.48 → Build geckodriver with latest Rust 1.49
Attachment #9195573 - Attachment description: Bug 1682219 - [geckodriver] Build with Rust 1.48 for Linux. → Bug 1682219 - [geckodriver] Build with Rust 1.49 for Linux.
Blocks: 1668243
Severity: -- → S3
Priority: -- → P3
Pushed by hskupin@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/931f753c12c0 [geckodriver] Build with Rust 1.49 for Linux. r=glandium

Greg, how can we enable Marionette/geckodriver trace logs for BrowserTime jobs? Is there a way to pass -vv to geckodriver, or set marionette.log.level=Trace?

Flags: needinfo?(hskupin) → needinfo?(gmierz2)

Also needinfo Andrew given that he also does a lot with BrowserTime.

Flags: needinfo?(acreskey)

Henrik, I believe that if you add the arguments, -vvvvv here to browsertime, you will then get geckodriver logs:

https://searchfox.org/mozilla-central/rev/31ddf859c57e812878a5f817e4097efb06de4d97/testing/raptor/raptor/browsertime/base.py#225

I don't know if this is possible via the try command.

Flags: needinfo?(acreskey)

In line 250 I can see:

        if self.verbose:
            browsertime_options.append("-vvv")

I wonder what's the difference between that one and -vvvvv.

Anyway here is the try build:
https://treeherder.mozilla.org/jobs?repo=try&revision=0ad35cce305c44e6fec735564cb347a9f04c0105

Flags: needinfo?(gmierz2)

Ok, that was helpful. There is actually a crash in the Android handler:

https://treeherder.mozilla.org/logviewer?job_id=326428160&repo=try&lineNumber=2041-2078

thread 'webdriver dispatcher' panicked at 'attempted to leave type `linked_hash_map::Node<serde_yaml::Value, serde_yaml::Value>` uninitialized, which is invalid', /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/core/src/mem/mod.rs:659:9
stack backtrace:
   0:           0x6684e0 - std::backtrace_rs::backtrace::libunwind::trace::hca287ea17d9489a6
                               at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/../../backtrace/src/backtrace/libunwind.rs:100:5
  [..]
  15:           0x6a7bad - core::panicking::panic::he533e234650a7fdb
                               at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/core/src/panicking.rs:50:5
  16:           0x493011 - linked_hash_map::LinkedHashMap<K,V,S>::insert::h89d33b5337ef845e
  17:           0x42db9d - geckodriver::android::AndroidHandler::prepare::haafa9a81713f08d8
  18:           0x4d051c - geckodriver::marionette::MarionetteHandler::create_connection::hb5fdd326bb5f22e9
  19:           0x4d3daa - <geckodriver::marionette::MarionetteHandler as webdriver::server::WebDriverHandler<geckodriver::command::GeckoExtensionRoute>>::handle_command::hd31696178a2c910d

So some call to LinkedHashMap.insert() fails here under AndroidHandler::prepare(). The only insertion that we do is in generate_config_file() for the environment variables:

https://searchfox.org/mozilla-central/rev/31ddf859c57e812878a5f817e4097efb06de4d97/testing/geckodriver/src/android.rs#264-282

And the Browsertime jobs pass in "env":{"MOZ_WEBRENDER":"0"}.

The problem here is with serde-yaml and it's Mapping class. The in-tree version is 0.8.9, while the latest is 0.8.15. Building geckodriver with that recent release fixes the problem.

This is a problem only happening in-tree, but not when users building geckodriver via the Github repository. So I will go ahead and try to vendor the Rust modules to get more up-to-date releases.

Priority: P3 → P2
Type: task → defect
Keywords: crash
Summary: Build geckodriver with latest Rust 1.49 → thread 'webdriver dispatcher' panicked at 'attempted to leave type `linked_hash_map::Node<value::Value, value::Value>` uninitialized, which is invalid' when geckodriver built with Rust 1.49

Updating serde_yaml results in:

cargo update -p serde_yaml
    Updating crates.io index
    Updating linked-hash-map v0.5.1 -> v0.5.4
    Updating serde_yaml v0.8.9 -> v0.8.15
    Updating yaml-rust v0.4.2 -> v0.4.5

The problem was actually in linked-hash-map (https://github.com/contain-rs/linked-hash-map/pull/96). As such I'm going to only update that particular crate.

Greg, I wonder why we didn't get the Rust traceback for the crash in the log output given that this is basically independent from the geckodriver log level. It would have been much easier to investigate and fix the problem.

Flags: needinfo?(gmierz2)

I think that would be because of this which filters out lines that don't raptor in them: https://searchfox.org/mozilla-central/source/testing/raptor/raptor/outputhandler.py#38

I've filed a bug about this (bug 1686230).

Flags: needinfo?(gmierz2)
Pushed by hskupin@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/02913f00f2ef [third-party] Upgrade linked-hash-map crate from 0.5.1 to 0.5.4. r=jgraham https://hg.mozilla.org/integration/autoland/rev/7e1ed116d07f [geckodriver] Build with Rust 1.49 for Linux. r=glandium
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → 86 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: