1852082 - Avoid fallible `dup` calls during DataPipe serialization

Status: RESOLVED FIXED

(Core :: IPC, enhancement)

Description

[Nika Layzell [:nika] (ni? for response)]

Currently when serializing a DataPipe over IPC, the underlying file descriptor for the pipe will be duplicated. The underlying dup or DuplicateHandle call can fail (e.g. due to file handle exhaustion), which is currently handled by crashing.

To avoid crashing in this scenario, we should consider pre-dup-ing the handle during the fallible constructor, such that serialization can instead transfer ownership of the handle into the message, rather than duplicating it.

In the future we'd probably want to do this using the RawShmem APIs. However, until bug 1797039 is implemented, those APIs are just implemented on top of SharedMemoryBasic, so it makes sense to continue using that type in DataType.

Comment 1

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1852082 - Part 1: Allow stealing ownership of the shared memory handle from SharedMemoryBasic, r=#ipc-reviewers

This is not the ideal form of this API, but avoids unnecessary handle
cloning and the corresponding fallible calls in some situations. In the
future we still want to do something more like bug 1797039, which will
provide a proper separation of handles from mappings for shared memory
regions.

Comment 2

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1852082 - Part 2: Avoid fallible operations during DataPipe serialization, r=#ipc-reviewers

Before this change, the shared memory region handle was stored in a
shared object, meaning that it needed to be cloned when serializing to
create an owned copy of the handle to serialize over IPC. As cloning a
file descriptor or HANDLE is fallible, this meant that serializing a
DataPipe could crash if file descriptors were exhausted.

This change pre-clones the file descriptors and closes the original
descriptor at creation, removing that fallible operation from DataPipe
serialization.

Depends on D187682

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3bba8dcd16a0
Part 1: Allow stealing ownership of the shared memory handle from SharedMemoryBasic, r=ipc-reviewers,mccr8
https://hg.mozilla.org/integration/autoland/rev/8b2f3c598351
Part 2: Avoid fallible operations during DataPipe serialization, r=ipc-reviewers,mccr8

Comment 4

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/3bba8dcd16a0
https://hg.mozilla.org/mozilla-central/rev/8b2f3c598351