1608759 - QM: Implement and use verifyStorage() function in all upgrade tests

Status: RESOLVED FIXED

(Core :: Storage: Quota Manager, task, P2)

Description

[Jan Varga [:janv]]

There are some things that need to be slightly refactored or cleaned up before we start ignoring unknown directories in repositories in bug 1594075.

Comment 1

[Jan Varga [:janv]]

Attachment: Bug 1608759 - installPackage() shouldn't allow file overwrites by default; r=ttung

Comment 2

[Jan Varga [:janv]]

Attachment: Bug 1608759 - Implement and use verifyStorage() function in all upgrade tests; r=ttung

The new verifyStorage() function takes current storage structure on disk and compares it with the expected structure. The expected structure is defined in JSON and consists of a per test package definition and a shared package definition. The shared package definition contains unknown files and directories which need to be ignored in all upgrade methods.
The new infrastructure for checking storage structure will be used later in other tests to verify handling of unknown or obsolete stuff for example during temporary storage initialization.

Comment 3

[Jan Varga [:janv]]

I plan to add more sub directories in dom/quota/test/unit:
quota-loading
reporting
data-clearing

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f2a08319ac10
installPackage() shouldn't allow file overwrites by default; r=dom-workers-and-storage-reviewers,ttung
https://hg.mozilla.org/integration/autoland/rev/cb706e608d58
Implement and use verifyStorage() function in all upgrade tests; r=dom-workers-and-storage-reviewers,ttung

Comment 5

[Daniel Varga [:dvarga]]

Backed out 2 changesets (Bug 1608759) for causing lint failure

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=287938918&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=cb706e608d58d42644958b7f79e8a420904bdd3a

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=287938918&repo=autoland&lineNumber=303

Backout link: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=287939862&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=f7fcd987559aa44518344c6ab921cf952fff53d3

[task 2020-02-07T15:07:38.146Z] /builds/worker/checkouts/gecko/media
[task 2020-02-07T15:07:38.169Z] 15:07:38.168 eslint (103) | Command: /usr/local/bin/node /builds/worker/checkouts/gecko/node_modules/eslint/bin/eslint.js --ext [js,jsm,jsx,xul,html,xhtml] --format json --no-error-on-unmatched-pattern --quiet --ignore-pattern media/libmkv --ignore-pattern media/mtransport/third_party --ignore-pattern devtools/shared/heapsnapshot/CoreDump.pb.cc --ignore-pattern extensions/spellcheck/hunspell/src --ignore-pattern devtools/shared/qrcode/encoder --ignore-pattern mfbt/double-conversion/double-conversion --ignore-pattern media/kiss_fft --ignore-pattern devtools/shared/node-properties --ignore-pattern media/libsoundtouch --ignore-pattern devtools/client/shared/sourceeditor/test/cm_mode_ruby.js --ignore-pattern devtools/client/shared/source-map --ignore-pattern media/libaom --ignore-pattern devtools/shared/acorn --ignore-pattern media/webrtc/signaling/src/sdp/sipcc --ignore-pattern media/libpng --ignore-pattern devtools/shared/heapsnapshot/CoreDump.pb.h --ignore-pattern browser/components/translation/cld2 --ignore-pattern media/ffvpx --ignore-pattern media/libspeex_resampler --ignore-pattern browser/components/pocket/content/panels/js/vendor --ignore-pattern media/openmax_il --ignore-pattern media/libwebp --ignore-pattern devtools/client/shared/vendor --ignore-pattern media/webrtc/trunk --ignore-pattern media/libtremor --ignore-pattern devtools/client/shared/demangle.js --ignore-pattern browser/components/newtab/vendor --ignore-pattern media/libvpx --ignore-pattern devtools/client/jsonview/lib/require.js --ignore-pattern browser/extensions/screenshots/build/raven.js --ignore-pattern media/libyuv --ignore-pattern media/libogg --ignore-pattern media/libvorbis --ignore-pattern parser/expat --ignore-pattern devtools/client/debugger/flow-typed/npm --ignore-pattern media/libnestegg --ignore-pattern browser/extensions/formautofill/content/third-party --ignore-pattern media/libjpeg --ignore-pattern devtools/client/shared/build/babel.js --ignore-pattern media/libdav1d --ignore-pattern devtools/client/shared/sourceeditor/codemirror --ignore-pattern devtools/client/shared/sourceeditor/test/codemirror --ignore-pattern devtools/shared/storage/vendor --ignore-pattern media/openmax_dl --ignore-pattern devtools/shared/jsbeautify --ignore-pattern devtools/shared/sprintfjs --ignore-pattern media/libtheora --ignore-pattern mobile/android/geckoview/src/thirdparty --ignore-pattern media/libcubeb --ignore-pattern devtools/shared/qrcode/decoder --ignore-pattern media/libopus /builds/worker/checkouts/gecko/mfbt /builds/worker/checkouts/gecko/hal /builds/worker/checkouts/gecko/extensions /builds/worker/checkouts/gecko/mobile /builds/worker/checkouts/gecko/browser /builds/worker/checkouts/gecko/devtools /builds/worker/checkouts/gecko/parser /builds/worker/checkouts/gecko/media
[task 2020-02-07T15:07:38.227Z] 15:07:38.227 eslint (102) | Passing the following paths:
[task 2020-02-07T15:07:38.228Z] /builds/worker/checkouts/gecko/dom  
[task 2020-02-07T15:07:38.228Z] /builds/worker/checkouts/gecko/docs  
[task 2020-02-07T15:07:38.228Z] /builds/worker/checkouts/gecko/servo  
[task 2020-02-07T15:07:38.228Z] /builds/worker/checkouts/gecko/chrome  
[task 2020-02-07T15:07:38.228Z] /builds/worker/checkouts/gecko/testing  
[task 2020-02-07T15:07:38.228Z] /builds/worker/checkouts/gecko/widget  
[task 2020-02-07T15:07:38.228Z] /builds/worker/checkouts/gecko/ipc  
[task 2020-02-07T15:07:38.228Z] /builds/worker/checkouts/gecko/toolkit
[task 2020-02-07T15:07:38.239Z] 15:07:38.236 eslint (102) | Command: /usr/local/bin/node /builds/worker/checkouts/gecko/node_modules/eslint/bin/eslint.js --ext [js,jsm,jsx,xul,html,xhtml] --format json --no-error-on-unmatched-pattern --quiet --ignore-pattern testing/modules/sinon-7.2.7.js --ignore-pattern dom/media/platforms/ffmpeg/ffmpeg58 --ignore-pattern testing/talos/talos/tests/dromaeo --ignore-pattern dom/u2f/tests/pkijs --ignore-pattern testing/mochitest/pywebsocket --ignore-pattern dom/media/gmp/rlz --ignore-pattern dom/media/webspeech/recognition/energy_endpointer.h --ignore-pattern dom/tests/mochitest/dom-level2-html --ignore-pattern toolkit/crashreporter/breakpad-client --ignore-pattern testing/xpcshell/node-ip --ignore-pattern dom/webauthn/tests/pkijs --ignore-pattern testing/xpcshell/node-http2 --ignore-pattern dom/media/platforms/ffmpeg/libav54 --ignore-pattern testing/gtest/gtest --ignore-pattern dom/tests/mochitest/dom-level1-core --ignore-pattern testing/web-platform/tests/resources/webidl2 --ignore-pattern dom/media/webaudio/test/blink --ignore-pattern dom/media/platforms/ffmpeg/libav55 --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module_ext.h --ignore-pattern dom/media/webvtt/vtt.jsm --ignore-pattern toolkit/components/certviewer/content/vendor --ignore-pattern testing/talos/talos/tests/kraken --ignore-pattern testing/mochitest/MochiKit --ignore-pattern toolkit/components/url-classifier/chromium --ignore-pattern testing/mozbase/mozproxy/mozproxy/backends/mitm/scripts/catapult --ignore-pattern dom/media/platforms/ffmpeg/ffmpeg57 --ignore-pattern toolkit/crashreporter/google-breakpad --ignore-pattern dom/webauthn/cbor-cpp --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module_proxy.h --ignore-pattern dom/tests/mochitest/ajax --ignore-pattern testing/web-platform/tests/tools/third_party --ignore-pattern toolkit/components/protobuf --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module_export.h --ignore-pattern dom/tests/mochitest/dom-level2-core --ignore-pattern dom/media/webspeech/recognition/energy_endpointer.cc --ignore-pattern dom/media/webspeech/recognition/energy_endpointer_params.h --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module.h --ignore-pattern testing/gtest/gmock --ignore-pattern testing/mochitest/tests/MochiKit-1.4.2 --ignore-pattern dom/media/webspeech/recognition/endpointer.cc --ignore-pattern ipc/chromium/src/third_party --ignore-pattern toolkit/components/jsoncpp --ignore-pattern dom/media/webspeech/recognition/energy_endpointer_params.cc --ignore-pattern testing/talos/talos/tests/v8_7 --ignore-pattern dom/media/platforms/ffmpeg/libav53 --ignore-pattern dom/imptests --ignore-pattern dom/media/webspeech/recognition/endpointer.h --ignore-pattern toolkit/components/normandy/vendor --ignore-pattern testing/xpcshell/dns-packet --ignore-pattern testing/modules/ajv-4.1.1.js --ignore-pattern dom/canvas/test/webgl-conf/checkout --ignore-pattern toolkit/components/utils/mozjexl.js /builds/worker/checkouts/gecko/dom /builds/worker/checkouts/gecko/docs /builds/worker/checkouts/gecko/servo /builds/worker/checkouts/gecko/chrome /builds/worker/checkouts/gecko/testing /builds/worker/checkouts/gecko/widget /builds/worker/checkouts/gecko/ipc /builds/worker/checkouts/gecko/toolkit
[task 2020-02-07T15:07:42.510Z] 15:07:42.509 eslint (101) | Finished in 4.49 seconds
[task 2020-02-07T15:08:23.701Z] 15:08:23.700 eslint (104) | Finished in 45.68 seconds
[task 2020-02-07T15:08:24.824Z] 15:08:24.823 eslint (105) | Finished in 46.79 seconds
[task 2020-02-07T15:08:26.997Z] 15:08:26.996 eslint (100) | Finished in 48.98 seconds
[task 2020-02-07T15:08:30.476Z] 15:08:30.476 eslint (99) | Finished in 52.46 seconds
[task 2020-02-07T15:08:34.279Z] 15:08:34.279 eslint (106) | Finished in 56.25 seconds
[task 2020-02-07T15:14:35.386Z] 15:14:35.386 eslint (102) | Finished in 417.37 seconds
[task 2020-02-07T15:15:04.047Z] 15:15:04.47 eslint (103) | Finished in 446.03 seconds
[task 2020-02-07T15:15:04.054Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/dom/quota/test/unit/head-shared.js:535:44 | Replace `⏎····packageDefinitionRelativePath,⏎····key⏎··` with `packageDefinitionRelativePath,·key` (prettier/prettier)
[task 2020-02-07T15:15:04.054Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/dom/quota/test/unit/head-shared.js:537:5 | 'key' is already declared in the upper scope. (no-shadow)
[task 2020-02-07T15:15:04.054Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/dom/quota/test/unit/head-shared.js:585:12 | 'name' is already declared in the upper scope. (no-shadow)
[task 2020-02-07T15:15:04.054Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/dom/quota/test/unit/head-shared.js:627:12 | 'name' is already declared in the upper scope. (no-shadow)
[taskcluster 2020-02-07 15:15:04.396Z] === Task Finished ===
[taskcluster 2020-02-07 15:15:04.718Z] Unsuccessful task run with exit code: 1 completed in 638.332 seconds

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2b0d755fe468
installPackage() shouldn't allow file overwrites by default; r=dom-workers-and-storage-reviewers,ttung

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ecb837a257f4
Implement and use verifyStorage() function in all upgrade tests; r=dom-workers-and-storage-reviewers,ttung

Comment 8

[Jan Varga [:janv]]

Fixed lint errors.

Comment 9

[Jan Varga [:janv]]

Attachment: Bug 1608759 - Add "beforeInstall" key to package definitions; r=ttung

This is a preparation for "afterInitTemporaryStorage" key which requires initialized storage.

Comment 10

[Jan Varga [:janv]]

Attachment: Bug 1608759 - Enhance SimpleDB to support other persistence types; r=ttung

Comment 11

[Jan Varga [:janv]]

Attachment: Bug 1608759 - Forbid leaf common entries; r=ttung

Comment 12

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/2b0d755fe468

Comment 13

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/ecb837a257f4

Comment 14

[Phabricator Automation]

Comment on attachment 9125102 [details]
Bug 1608759 - Enhance SimpleDB to support other persistence types; r=ttung

Revision D62058 was moved to bug 1615552. Setting attachment 9125102 [details] to obsolete.

Comment 15

[Jan Varga [:janv]]

The original idea was to just clean up origin parser related things, but it has evolved into much bigger cleanup, refactoring and enhancements so I'm going to split this into separate bugs.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e2de6ed6fddd
Add "beforeInstall" key to package definitions; r=ttung,dom-workers-and-storage-reviewers
https://hg.mozilla.org/integration/autoland/rev/26df3a77634a
Forbid leaf common entries; r=ttung,dom-workers-and-storage-reviewers

Comment 17

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/e2de6ed6fddd
https://hg.mozilla.org/mozilla-central/rev/26df3a77634a

Comment 18

[Phabricator Automation]

Comment on attachment 9125102 [details]
Bug 1608759 - Enhance SimpleDB to support other persistence types; r=ttung

Revision D62058 was moved to bug 1615552. Setting attachment 9125102 [details] to obsolete.