1769593 - Fail when serializing, rather than deserializing, file handles in excess of `MAX_DESCRIPTORS_PER_MESSAGE`

Status: RESOLVED FIXED

(Core :: IPC, task)

Description

[Nika Layzell [:nika] (ni? for response)]

Currently the code for serializing simply logs an error to the console when serializing handles in excess of MAX_DESCRIPTORS_PER_MESSAGE, which ends up leading to the receiving side crashing when deserialization fails, potentially producing confusing errors which don't mention handles at all (such as bug 1769504). We should fail earlier on the sending side in this case, like we do for too-large messages, rather than failing when trying to read a message which we sent despite knowing it'll fail to deserialize.

We should also report the actual cause of the error better when both serializing and deserializing.

Comment 1

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1769593 - Part 1: Fail when serializing file handles in excess of MAX_DESCRIPTORS_PER_MESSAGE, r=#ipc-reviewers

Comment 2

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1769593 - Part 2: Improve reporting of fatal DataPipe (de)serialization errors, r=#ipc-reviewers

Depends on D146501

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/16811ba73030
Part 1: Fail when serializing file handles in excess of MAX_DESCRIPTORS_PER_MESSAGE, r=jld
https://hg.mozilla.org/integration/autoland/rev/353b922a3293
Part 2: Improve reporting of fatal DataPipe (de)serialization errors, r=jld

Comment 4

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/16811ba73030
https://hg.mozilla.org/mozilla-central/rev/353b922a3293