1897498 - Build a single-file archive export method on BackupService

Status: RESOLVED FIXED

(Firefox :: Profile Backup, task, P3)

Description

[Mike Conley (:mconley) (:⚙️)]

This method should:

1. Take in a URL or handle to a local ZIP file. It might be a file URI, or maybe an nsIInputStream or ReadableStream.
2. Have the method accept some items from an ArchiveEncryptionState if encryption is enabled. Those things are the public key, the salt, the backup auth key, and the wrapped static secrets. Make sure that the worker knows if encryption is expected to be enabled so that it can fail out if any of these things are not present or cannot be worked with.
3. Have the method use some template URI for the single-file archive contents, as well as the backup metadata.
4. Have the method fill in the template with items from the metadata, and then create a document comment block after the closing </html> tag, like this: <!-- Begin inline MIME --
5. Create a MIME message header on the next line containing the content type data (Content-Type: multipart/mixed; boundary="<some generated bounday>")
6. Create a application/json MIME section header, and then write a JSON structure. For now, it'll just be the metadata. In the future, it'll be the metadata alongside some cryptographic information.
7. Create a new binary/octet-stream MIME section header, including the content length.
8. Pump the base64 representation of the ZIP file into this region. Since the BackupService will be the only application that we intend to support reading this block, we don't need to break it up with newlines.
9. On a newline, close the comment block with -- End inline MIME -->, followed by a newline.

We won't worry about doing the cryptography bit just yet - we'll just do the raw base64 encoding right now. We also won't worry about hooking this up to the createBackup mechanism just yet.

Comment 1

[Mike Conley (:mconley) (:⚙️)]

Initially, I envisioned this being done in a PromiseWorker, but I'm going to opt for simplicity to start before trying to factor out a worker (which might not even be necessary in the long run).

Comment 2

[Mike Conley (:mconley) (:⚙️)]

I'm waffling a bit here. I think I'm going back to the Worker model, mainly because I realized I can use IOUtils in a privileged ChromeWorker, which is nice - and also because base64 encoding large chunks of data on the main thread is never going to pass muster, performance-wise.

Comment 3

[Mike Conley (:mconley) (:⚙️)]

Attachment: Bug 1897498 - Add a mechanism for exporting backup snapshots to a single file archive. r?#backup-reviewers!,Gijs!,valentin!

There are a number of interesting things going on this patch that I think are worth highlighting
here for my reviewers:

1. The single-file archive format is an HTML file that uses an inlined multipart/mixed MIME
   message within a HTML document comment in order to embed the backup data into the archive.
2. We use the multipart/mixed nsIStreamConverter to extract the JSON and binary data from
   the MIME block.
3. We use a Archive Worker to do the archive creation, allowing us to do the work of construction
   off of the main thread.
4. The Archive Worker is only parsing the header and getting the byte offset of the MIME block.
   Extraction is happening in the parent process. This is mainly for simplicity for now, since
   the Archive Worker cannot invoke an nsIStreamConverter. Down the line, if we determine that
   we'd prefer the Archive Worker do the base64 decoding off of the main thread, we may need
   to use a Message Channel to send the byte sfrom the nsIStreamConverter to it, and add
   stream-writing support to IOUtils so that the Archive Worker can take care of sending the
   decoded bytes to disk.
5. The patch doesn't expose the extraction mechanism in any way except through the debug
   interface right now. That will come down the line. In the meantime, this mechanism
   can be manually tested in the debug interface by creating a backup, which should also
   create an "archive.html" file in the backups folder. Using the "Extract from archive"
   button in the debug tool will let you select that HTML file and extract the ZIP as
   a file in the backups folder called "extraction.zip".
6. The test template contains Unicode characters because certain locales might involve
   us writing Unicode characters in the HTML template when generating the archive. The
   fun part about that is calculating where the byte offset is for the MIME block! See
   the comment in the Archive.worker.mjs script for how that works.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d88e9ba4a866
Add a mechanism for exporting backup snapshots to a single file archive. r=Gijs,valentin,backup-reviewers,frontend-codestyle-reviewers,Standard8,kpatenio

Comment 5

[pstanciu]

Backed out for causing xpcshell failures in test_BackupService_archive.js

• Backout link
• Push with failures
• Failure Log

Comment 6

[Iulian Moraru]

Pulse bot did not comment on the bug when this got pushed again to autoland because of bugzilla hiccup.
Push link: https://hg.mozilla.org/integration/autoland/rev/5f31ec57317e668a647da5299e4dee4809fc8ccf

Comment 7

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/5f31ec57317e