Closed Bug 1608759 Opened 2 years ago Closed 2 years ago

QM: Implement and use verifyStorage() function in all upgrade tests

Categories

(Core :: Storage: Quota Manager, task, P2)

task

Tracking

()

RESOLVED FIXED
mozilla75
Tracking Status
firefox75 --- fixed

People

(Reporter: janv, Assigned: janv)

References

(Blocks 1 open bug)

Details

Attachments

(4 files, 1 obsolete file)

There are some things that need to be slightly refactored or cleaned up before we start ignoring unknown directories in repositories in bug 1594075.

Depends on: 1608025
Blocks: 1593365
Summary: QM: Cleanup OriginParser related things → QM: Clean up OriginParser related things

The new verifyStorage() function takes current storage structure on disk and compares it with the expected structure. The expected structure is defined in JSON and consists of a per test package definition and a shared package definition. The shared package definition contains unknown files and directories which need to be ignored in all upgrade methods.
The new infrastructure for checking storage structure will be used later in other tests to verify handling of unknown or obsolete stuff for example during temporary storage initialization.

Keywords: leave-open

I plan to add more sub directories in dom/quota/test/unit:
quota-loading
reporting
data-clearing

Attachment #9124036 - Attachment description: Bug 1608759 - installPackage() shouldn't allow file overwrites by default; r=asuth → Bug 1608759 - installPackage() shouldn't allow file overwrites by default; r=ttung
Attachment #9124037 - Attachment description: Bug 1608759 - Implement and use verifyStorage() function in all upgrade tests; r=asuth → Bug 1608759 - Implement and use verifyStorage() function in all upgrade tests; r=ttung
Pushed by jvarga@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f2a08319ac10
installPackage() shouldn't allow file overwrites by default; r=dom-workers-and-storage-reviewers,ttung
https://hg.mozilla.org/integration/autoland/rev/cb706e608d58
Implement and use verifyStorage() function in all upgrade tests; r=dom-workers-and-storage-reviewers,ttung

Backed out 2 changesets (Bug 1608759) for causing lint failure

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=287938918&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=cb706e608d58d42644958b7f79e8a420904bdd3a

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=287938918&repo=autoland&lineNumber=303

Backout link: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=287939862&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=f7fcd987559aa44518344c6ab921cf952fff53d3

[task 2020-02-07T15:07:38.146Z] /builds/worker/checkouts/gecko/media
[task 2020-02-07T15:07:38.169Z] 15:07:38.168 eslint (103) | Command: /usr/local/bin/node /builds/worker/checkouts/gecko/node_modules/eslint/bin/eslint.js --ext [js,jsm,jsx,xul,html,xhtml] --format json --no-error-on-unmatched-pattern --quiet --ignore-pattern media/libmkv --ignore-pattern media/mtransport/third_party --ignore-pattern devtools/shared/heapsnapshot/CoreDump.pb.cc --ignore-pattern extensions/spellcheck/hunspell/src --ignore-pattern devtools/shared/qrcode/encoder --ignore-pattern mfbt/double-conversion/double-conversion --ignore-pattern media/kiss_fft --ignore-pattern devtools/shared/node-properties --ignore-pattern media/libsoundtouch --ignore-pattern devtools/client/shared/sourceeditor/test/cm_mode_ruby.js --ignore-pattern devtools/client/shared/source-map --ignore-pattern media/libaom --ignore-pattern devtools/shared/acorn --ignore-pattern media/webrtc/signaling/src/sdp/sipcc --ignore-pattern media/libpng --ignore-pattern devtools/shared/heapsnapshot/CoreDump.pb.h --ignore-pattern browser/components/translation/cld2 --ignore-pattern media/ffvpx --ignore-pattern media/libspeex_resampler --ignore-pattern browser/components/pocket/content/panels/js/vendor --ignore-pattern media/openmax_il --ignore-pattern media/libwebp --ignore-pattern devtools/client/shared/vendor --ignore-pattern media/webrtc/trunk --ignore-pattern media/libtremor --ignore-pattern devtools/client/shared/demangle.js --ignore-pattern browser/components/newtab/vendor --ignore-pattern media/libvpx --ignore-pattern devtools/client/jsonview/lib/require.js --ignore-pattern browser/extensions/screenshots/build/raven.js --ignore-pattern media/libyuv --ignore-pattern media/libogg --ignore-pattern media/libvorbis --ignore-pattern parser/expat --ignore-pattern devtools/client/debugger/flow-typed/npm --ignore-pattern media/libnestegg --ignore-pattern browser/extensions/formautofill/content/third-party --ignore-pattern media/libjpeg --ignore-pattern devtools/client/shared/build/babel.js --ignore-pattern media/libdav1d --ignore-pattern devtools/client/shared/sourceeditor/codemirror --ignore-pattern devtools/client/shared/sourceeditor/test/codemirror --ignore-pattern devtools/shared/storage/vendor --ignore-pattern media/openmax_dl --ignore-pattern devtools/shared/jsbeautify --ignore-pattern devtools/shared/sprintfjs --ignore-pattern media/libtheora --ignore-pattern mobile/android/geckoview/src/thirdparty --ignore-pattern media/libcubeb --ignore-pattern devtools/shared/qrcode/decoder --ignore-pattern media/libopus /builds/worker/checkouts/gecko/mfbt /builds/worker/checkouts/gecko/hal /builds/worker/checkouts/gecko/extensions /builds/worker/checkouts/gecko/mobile /builds/worker/checkouts/gecko/browser /builds/worker/checkouts/gecko/devtools /builds/worker/checkouts/gecko/parser /builds/worker/checkouts/gecko/media
[task 2020-02-07T15:07:38.227Z] 15:07:38.227 eslint (102) | Passing the following paths:
[task 2020-02-07T15:07:38.228Z] /builds/worker/checkouts/gecko/dom  
[task 2020-02-07T15:07:38.228Z] /builds/worker/checkouts/gecko/docs  
[task 2020-02-07T15:07:38.228Z] /builds/worker/checkouts/gecko/servo  
[task 2020-02-07T15:07:38.228Z] /builds/worker/checkouts/gecko/chrome  
[task 2020-02-07T15:07:38.228Z] /builds/worker/checkouts/gecko/testing  
[task 2020-02-07T15:07:38.228Z] /builds/worker/checkouts/gecko/widget  
[task 2020-02-07T15:07:38.228Z] /builds/worker/checkouts/gecko/ipc  
[task 2020-02-07T15:07:38.228Z] /builds/worker/checkouts/gecko/toolkit
[task 2020-02-07T15:07:38.239Z] 15:07:38.236 eslint (102) | Command: /usr/local/bin/node /builds/worker/checkouts/gecko/node_modules/eslint/bin/eslint.js --ext [js,jsm,jsx,xul,html,xhtml] --format json --no-error-on-unmatched-pattern --quiet --ignore-pattern testing/modules/sinon-7.2.7.js --ignore-pattern dom/media/platforms/ffmpeg/ffmpeg58 --ignore-pattern testing/talos/talos/tests/dromaeo --ignore-pattern dom/u2f/tests/pkijs --ignore-pattern testing/mochitest/pywebsocket --ignore-pattern dom/media/gmp/rlz --ignore-pattern dom/media/webspeech/recognition/energy_endpointer.h --ignore-pattern dom/tests/mochitest/dom-level2-html --ignore-pattern toolkit/crashreporter/breakpad-client --ignore-pattern testing/xpcshell/node-ip --ignore-pattern dom/webauthn/tests/pkijs --ignore-pattern testing/xpcshell/node-http2 --ignore-pattern dom/media/platforms/ffmpeg/libav54 --ignore-pattern testing/gtest/gtest --ignore-pattern dom/tests/mochitest/dom-level1-core --ignore-pattern testing/web-platform/tests/resources/webidl2 --ignore-pattern dom/media/webaudio/test/blink --ignore-pattern dom/media/platforms/ffmpeg/libav55 --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module_ext.h --ignore-pattern dom/media/webvtt/vtt.jsm --ignore-pattern toolkit/components/certviewer/content/vendor --ignore-pattern testing/talos/talos/tests/kraken --ignore-pattern testing/mochitest/MochiKit --ignore-pattern toolkit/components/url-classifier/chromium --ignore-pattern testing/mozbase/mozproxy/mozproxy/backends/mitm/scripts/catapult --ignore-pattern dom/media/platforms/ffmpeg/ffmpeg57 --ignore-pattern toolkit/crashreporter/google-breakpad --ignore-pattern dom/webauthn/cbor-cpp --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module_proxy.h --ignore-pattern dom/tests/mochitest/ajax --ignore-pattern testing/web-platform/tests/tools/third_party --ignore-pattern toolkit/components/protobuf --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module_export.h --ignore-pattern dom/tests/mochitest/dom-level2-core --ignore-pattern dom/media/webspeech/recognition/energy_endpointer.cc --ignore-pattern dom/media/webspeech/recognition/energy_endpointer_params.h --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module.h --ignore-pattern testing/gtest/gmock --ignore-pattern testing/mochitest/tests/MochiKit-1.4.2 --ignore-pattern dom/media/webspeech/recognition/endpointer.cc --ignore-pattern ipc/chromium/src/third_party --ignore-pattern toolkit/components/jsoncpp --ignore-pattern dom/media/webspeech/recognition/energy_endpointer_params.cc --ignore-pattern testing/talos/talos/tests/v8_7 --ignore-pattern dom/media/platforms/ffmpeg/libav53 --ignore-pattern dom/imptests --ignore-pattern dom/media/webspeech/recognition/endpointer.h --ignore-pattern toolkit/components/normandy/vendor --ignore-pattern testing/xpcshell/dns-packet --ignore-pattern testing/modules/ajv-4.1.1.js --ignore-pattern dom/canvas/test/webgl-conf/checkout --ignore-pattern toolkit/components/utils/mozjexl.js /builds/worker/checkouts/gecko/dom /builds/worker/checkouts/gecko/docs /builds/worker/checkouts/gecko/servo /builds/worker/checkouts/gecko/chrome /builds/worker/checkouts/gecko/testing /builds/worker/checkouts/gecko/widget /builds/worker/checkouts/gecko/ipc /builds/worker/checkouts/gecko/toolkit
[task 2020-02-07T15:07:42.510Z] 15:07:42.509 eslint (101) | Finished in 4.49 seconds
[task 2020-02-07T15:08:23.701Z] 15:08:23.700 eslint (104) | Finished in 45.68 seconds
[task 2020-02-07T15:08:24.824Z] 15:08:24.823 eslint (105) | Finished in 46.79 seconds
[task 2020-02-07T15:08:26.997Z] 15:08:26.996 eslint (100) | Finished in 48.98 seconds
[task 2020-02-07T15:08:30.476Z] 15:08:30.476 eslint (99) | Finished in 52.46 seconds
[task 2020-02-07T15:08:34.279Z] 15:08:34.279 eslint (106) | Finished in 56.25 seconds
[task 2020-02-07T15:14:35.386Z] 15:14:35.386 eslint (102) | Finished in 417.37 seconds
[task 2020-02-07T15:15:04.047Z] 15:15:04.47 eslint (103) | Finished in 446.03 seconds
[task 2020-02-07T15:15:04.054Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/dom/quota/test/unit/head-shared.js:535:44 | Replace `⏎····packageDefinitionRelativePath,⏎····key⏎··` with `packageDefinitionRelativePath,·key` (prettier/prettier)
[task 2020-02-07T15:15:04.054Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/dom/quota/test/unit/head-shared.js:537:5 | 'key' is already declared in the upper scope. (no-shadow)
[task 2020-02-07T15:15:04.054Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/dom/quota/test/unit/head-shared.js:585:12 | 'name' is already declared in the upper scope. (no-shadow)
[task 2020-02-07T15:15:04.054Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/dom/quota/test/unit/head-shared.js:627:12 | 'name' is already declared in the upper scope. (no-shadow)
[taskcluster 2020-02-07 15:15:04.396Z] === Task Finished ===
[taskcluster 2020-02-07 15:15:04.718Z] Unsuccessful task run with exit code: 1 completed in 638.332 seconds
Flags: needinfo?(jvarga)
Pushed by jvarga@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2b0d755fe468
installPackage() shouldn't allow file overwrites by default; r=dom-workers-and-storage-reviewers,ttung
Pushed by jvarga@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ecb837a257f4
Implement and use verifyStorage() function in all upgrade tests; r=dom-workers-and-storage-reviewers,ttung

Fixed lint errors.

Flags: needinfo?(jvarga)

This is a preparation for "afterInitTemporaryStorage" key which requires initialized storage.

Comment on attachment 9125102 [details]
Bug 1608759 - Enhance SimpleDB to support other persistence types; r=ttung

Revision D62058 was moved to bug 1615552. Setting attachment 9125102 [details] to obsolete.

Attachment #9125102 - Attachment is obsolete: true

The original idea was to just clean up origin parser related things, but it has evolved into much bigger cleanup, refactoring and enhancements so I'm going to split this into separate bugs.

No longer blocks: 1594075
Summary: QM: Clean up OriginParser related things → QM: Implement and use verifyStorage() function in all upgrade tests
Keywords: leave-open
Pushed by jvarga@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e2de6ed6fddd
Add "beforeInstall" key to package definitions; r=ttung,dom-workers-and-storage-reviewers
https://hg.mozilla.org/integration/autoland/rev/26df3a77634a
Forbid leaf common entries; r=ttung,dom-workers-and-storage-reviewers
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla75
Attachment #9125102 - Attachment is obsolete: false

Comment on attachment 9125102 [details]
Bug 1608759 - Enhance SimpleDB to support other persistence types; r=ttung

Revision D62058 was moved to bug 1615552. Setting attachment 9125102 [details] to obsolete.

Attachment #9125102 - Attachment is obsolete: true
No longer blocks: 1593365
Blocks: 1619891
Blocks: 1620277
No longer blocks: 1619891
No longer depends on: 1608025
You need to log in before you can comment on or make changes to this bug.