1557161 - enable rkv consumers to migrate LMDB environments across architecture changes

Status: RESOLVED FIXED

(Toolkit :: Storage, enhancement)

Description

[Myk Melez [:myk] [@mykmelez]]

rkv just gained support for migrating LMDB environments across architecture changes (specifically: changes to the bit-depth of the build that is accessing the LMDB environment). rkv consumers in Firefox should take advantage of this feature to migrate their environments when a user changes architectures.

Comment 1

[Myk Melez [:myk] [@mykmelez]]

Attachment: enable rkv consumers to migrate LMDB environments across architecture changes

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ce509bb0895a
enable rkv consumers to migrate LMDB environments across architecture changes r=lina,keeler,froydnj

Comment 3

[Cosmin Sabou [:CosminS]]

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=busted&tochange=4a1ccc981b6ef5644acad094f08c98e679263ca1&fromchange=ce509bb0895aac04378f959a5c6627fb10b384d8&selectedJob=250978005

Failure log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=250978005&repo=autoland&lineNumber=16161
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=250986259&repo=autoland&lineNumber=1104

Backout link: https://hg.mozilla.org/integration/autoland/rev/4a1ccc981b6ef5644acad094f08c98e679263ca1

18:38:26 INFO - Compiling mozilla-central-workspace-hack v0.1.0 (Z:\build\build\src\build\workspace-hack)
18:38:26 INFO - Running set CARGO_PKG_HOMEPAGE=&& set CARGO='\\?\Z:\task_1560186311\build\src\rustc\bin\cargo.exe'&& set CARGO_MANIFEST_DIR='Z:\build\build\src\build\workspace-hack'&& set CARGO_PKG_VERSION_MINOR=1&& set CARGO_PKG_VERSION_PRE=&& set CARGO_PKG_NAME=mozilla-central-workspace-hack&& set PATH='z:/build/build/src/obj-firefox\release\deps;Z:\task_1560186311\build\src\rustc\bin;z:\build\build\src\clang\bin;z:\build\build\src\clang\bin;c:\Program Files\Mercurial;c:\mozilla-build\bin;c:\mozilla-build\kdiff3;c:\mozilla-build\moztools-x64\bin;c:\mozilla-build\mozmake;C:\mozilla-build\msys\bin;C:\mozilla-build\msys\local\bin;c:\mozilla-build\nsis-3.01;c:\mozilla-build\python;c:\mozilla-build\python\Scripts;c:\mozilla-build\python3;c:\Windows\system32;c:\Windows;c:\Windows\System32\Wbem;c:\Windows\System32\WindowsPowerShell\v1.0\;c:\Program Files\Amazon\cfn-bootstrap\;c:\Program Files (x86)\GNU\GnuPG\pub;c:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;c:\mozilla-build\python\lib\site-packages\pywin32_system32;c:\mozilla-build\python\lib\site-packages\pywin32_system32;c:\mozilla-build\python\lib\site-packages\pywin32_system32;z:/build/build/src/vs2017_15.9.6/VC/bin/HostX64/arm64;C:\Users\task_1560186311\.cargo\bin;z:\build\.mozbuild\clang\bin;z:\build\.mozbuild\cbindgen;z:\build\.mozbuild\nasm;z:/build/build/src/vs2017_15.9.6/VC/bin/HostX64/x64'&& set CARGO_PKG_AUTHORS=&& set CARGO_PKG_VERSION_PATCH=0&& set CARGO_PKG_VERSION=0.1.0&& set CARGO_PKG_REPOSITORY=&& set CARGO_PKG_VERSION_MAJOR=0&& set CARGO_PKG_DESCRIPTION=&& z:/build/build/src/sccache/sccache.exe 'z:/build/build/src/rustc/bin/rustc.exe' --crate-name mozilla_central_workspace_hack 'build\workspace-hack\src\lib.rs' --color never --crate-type lib --emit=dep-info,link -C opt-level=2 -C panic=abort -C codegen-units=1 -C metadata=97f4bf9cbdc26846 -C extra-filename=-97f4bf9cbdc26846 --out-dir 'z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\release\deps' --target aarch64-pc-windows-msvc -C 'linker=z:/build/build/src/build/cargo-linker.bat' -L 'dependency=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\release\deps' -L 'dependency=z:/build/build/src/obj-firefox\release\deps' --extern 'winapi=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\release\deps\libwinapi-3b6a9ca92c5837bd.rlib' -C opt-level=2 -C debuginfo=2 -Dwarnings
18:38:26 ERROR - error[E0425]: cannot find value STRUCT_SIZE in this scope
18:38:26 INFO - --> Z:\build\build\src\third_party\rust\remove_dir_all\src\fs.rs:165:54
18:38:26 INFO - |
18:38:26 INFO - 165 | let mut data: Vec<u16> = iter::repeat(0u16).take(STRUCT_SIZE/2)
18:38:26 INFO - | ^^^^^^^^^^^ not found in this scope
18:38:26 ERROR - error[E0425]: cannot find value STRUCT_SIZE in this scope
18:38:26 INFO - --> Z:\build\build\src\third_party\rust\remove_dir_all\src\fs.rs:179:42
18:38:26 INFO - |
18:38:26 INFO - 179 | (*info).FileNameLength = (size - STRUCT_SIZE) as DWORD;
18:38:26 INFO - | ^^^^^^^^^^^ not found in this scope
18:38:26 INFO - warning: ignoring -C extra-filename flag due to -o flag

19:14:48 INFO - TEST-START | toolkit/components/kvstore/test/xpcshell/test_kvstore.js
19:14:54 WARNING - TEST-UNEXPECTED-FAIL | toolkit/components/kvstore/test/xpcshell/test_kvstore.js | xpcshell return code: 0
19:14:54 INFO - TEST-INFO took 5690ms
19:14:54 INFO - >>>>>>>
19:14:54 INFO - PID 592 | Unable to load \untrusted-startup-test-dll.dll; LoadLibraryW failed: 126Couldn't convert chrome URL: chrome://branding/locale/brand.properties
19:14:54 INFO - PID 592 | [592, Main Thread] WARNING: Failed to get directory to cache.: file z:/build/build/src/security/sandbox/win/src/sandboxbroker/sandboxBroker.cpp, line 82
19:14:54 INFO - PID 592 | [592, Main Thread] WARNING: Failed to get directory to cache.: file z:/build/build/src/security/sandbox/win/src/sandboxbroker/sandboxBroker.cpp, line 82
19:14:54 INFO - PID 592 | [592, Main Thread] WARNING: Failed to get directory to cache.: file z:/build/build/src/security/sandbox/win/src/sandboxbroker/sandboxBroker.cpp, line 82
19:14:54 INFO - PID 592 | [592, Main Thread] WARNING: Failed to get directory to cache.: file z:/build/build/src/security/sandbox/win/src/sandboxbroker/sandboxBroker.cpp, line 82
19:14:54 INFO - PID 592 | [592, Main Thread] WARNING: Failed to get directory to cache.: file z:/build/build/src/security/sandbox/win/src/sandboxbroker/sandboxBroker.cpp, line 82
19:14:54 INFO - PID 592 | [592, Main Thread] WARNING: Couldn't get the user appdata directory. Crash events may not be produced.: file z:/build/build/src/toolkit/crashreporter/nsExceptionHandler.cpp, line 2514
19:14:54 INFO - (xpcshell/head.js) | test MAIN run_test pending (1)
19:14:54 INFO - PID 592 | [592, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80520012: file z:/build/build/src/extensions/permissions/nsPermissionManager.cpp, line 2903
19:14:54 INFO - (xpcshell/head.js) | test run_next_test 0 pending (2)
19:14:54 INFO - (xpcshell/head.js) | test MAIN run_test finished (2)
19:14:54 INFO - running event loop
19:14:54 INFO - PID 592 | [592, Main Thread] WARNING: Could not get the program name for a cubeb stream.: 'NS_SUCCEEDED(rv)', file z:/build/build/src/dom/media/CubebUtils.cpp, line 388
19:14:54 INFO - toolkit/components/kvstore/test/xpcshell/test_kvstore.js | Starting getService
19:14:54 INFO - (xpcshell/head.js) | test getService pending (2)
19:14:54 INFO - TEST-PASS | toolkit/components/kvstore/test/xpcshell/test_kvstore.js

Comment 4

[Myk Melez [:myk] [@mykmelez]]

The failure is due to https://github.com/XAMPPRocky/remove_dir_all/issues/10. I'm looking into fixing it upstream.

Comment 5

[Myk Melez [:myk] [@mykmelez]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=77d8feb3fc34f56f5cafce29468d46e491b56b11

Comment 6

[Myk Melez [:myk] [@mykmelez]]

There's a second test failure due to the way the patch specifies support-files for the xpcshell tests, which works on macOS/Linux but not on Windows. I have a fix that works on all three OSes locally. Here's a tryserver run for confirmation:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c6a52e9624d9c7bf9ad8e233872e6463548f7895

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/595c5084ff82
enable rkv consumers to migrate LMDB environments across architecture changes r=lina,keeler,froydnj

Comment 8

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/595c5084ff82