984651 - nsIScriptableInputStream needs a readDelimitedBytes method

Status: ASSIGNED

(Core :: Networking, defect, P5)

Description

[Jim Blandy :jimb]

nsIScriptableInputStream should have the ability to read from the stream up to the first occurrence of a delimiter character, or up to a maximum number of bytes.

For the developer tools' remote protocol, our packets have one of two forms:

COUNT ":" JSONBODY - a byte count in ASCII, followed by a colon, followed by that many bytes of UTF-8 encoded JSON.

"bulk " ACTOR " " TYPE " " COUNT ":" BULKBODY - a bulk packet containing arbitrary binary data, where COUNT is the length of BULKBODY in bytes.

Note that, in both cases, a colon character marks the beginning of the fixed-length packet body. The packet header is not allowed to contain a colon.

The second format (currently being implemented) is meant to support transferring large amounts of binary data: downloading apps to a phone; uploading megabytes of profile data or memory dumps; and in general, any case where converting the data to JSON first would greatly increase memory consumption.

When receiving bulk packets, our goal is to simply hand over the connection's nsIInputStream to a function that reads the bytes of BULKBODY directly off the connection, thereby avoiding the need to convert it to some JavaScript object or string first.

However, in order to be sure that our stream is at the right position, we cannot simply grab as many bytes as |nsIScriptableInputStream::Available| promises off the stream; we will then read past the colon into the BULKBODY. Certainly, we could pass our overshoot data to the bulk data handler, but that's an ugly interface.

Or, we could read one character at a time from the nsIScriptableInputStream; but that seems slow.

This bug argues that we should extend nsIScriptableInputStream with the method described below, which would allow us to parse our packet headers in a completely straightforward way.


    /**
     * Read data from the stream, including NULL bytes, up to either a
     * maximum number of bytes, or the first occurrence of a delimiter
     * byte, or EOF, whichever comes first. If the delimiter byte occurs
     * within the given number of bytes, the stream is left immediately
     * after that delimiter byte.
     *
     * The caller can always determine why reading stopped:
     *
     * - If the returned string ends with the delimiter character, then
     *   that is why we stopped reading. Note that the string may still
     *   have the given maximum length, and the stream may be at EOF.
     *
     * - Otherwise, if the string has the given maximum length, then that
     *   is why we stopped reading. Note that the stream may be at EOF.
     *
     * - Otherwise, the string must be is shorter than the maximum, and
     *   must not end with the delimiter character. Thus, we were stopped
     *   by EOF.
     *
     * This interface is different from readBytes, which simply drops the
     * string on EOF. However, this interface allows the caller to handle
     * all possible streams without losing data.
     *
     * @param aCount the maximum number of bytes to read.
     *
     * @param aDelimiter the delimiter byte to look for. (When calling from
     *        JavaScript, this parameter can be a one-character string, if
     *        the character's code fits in a single byte.)
     *
     * @return the data from the stream, including the delimiter byte, if
     *        one was found.
     *
     * @throws NS_BASE_STREAM_WOULD_BLOCK if reading from the input stream
     *        would block the calling thread (non-blocking mode only). Note
     *        that the returned string is truncated in this case, and data
     *        is lost. However, if the caller ensures that the |aCount|
     *        parameter is not greater than the stream's |available| byte
     *        count, this condition should never occur.
     */

Comment 1

[Jim Blandy :jimb]

Attachment: Implement nsScriptableInputStream::ReadDelimitedBytes.

Draft implementation. Not tested; just for feedback.

Comment 2

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258