1092196 - How to properly receive bulk data sent over RDP

Status: RESOLVED INVALID

(DevTools :: General, defect)

Description

[Jan Honza Odvarko [:Honza] (always need-info? me)]

I am working on bug 977443 as well as on an example addon that shows how to send and receive bulk data over RDP.

Here is the addon example:
https://github.com/firebug/extension-examples/tree/master/BulkActor

Here is the client side (send) part:
https://github.com/firebug/extension-examples/blob/master/BulkActor/lib/bulkActor.js#L143
(this seems to be working)

And here is the server side (receiving):
https://github.com/firebug/extension-examples/blob/master/BulkActor/lib/bulkActor.js#L112

I don't know how to implement it.

The question is how should I properly ready the stream using the helper copyTo and convert it to a string? (I am actually sending a big string using bulk request to avoid base64 conversion)

I have been looking at this test
http://mxr.mozilla.org/mozilla-central/source/toolkit/devtools/transport/tests/unit/test_client_server_bulk.js

but it’s using temporary files (I'd like to do in-memory only operations)

Honza

Comment 1

[Panos Astithas (he/him) [:past] (please ni?)]

Ryan is actually the person who implemented the bulk transport functionality.

Comment 2

[Nick Fitzgerald [:fitzgen] [⏰PST; UTC-8]]

What Panos said. I haven't used or touched the bulk data stuff.

Comment 3

[Jan Honza Odvarko [:Honza] (always need-info? me)]

> Ryan is actually the person who implemented the bulk transport functionality.
Thanks Panos!

Honza

Comment 4

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

I believe the main issue you are seeing is that protocol.js has not yet been updated to work with bulk data (bug 949595), so you're having to do more work manually today.  It's possible also that there are changes needed to make it work at all.  I haven't tried it in a while myself.  It looks like protocol.js's actor delivery path expects a |conn| object that it can immediately send a reply out through[3], which is not likely to work with bulk data, since it is more asynchronous.  The single user of bulk data at the moment (installing apps on FxOS) is an old "before protocol.js" style actor.

Another thing to be aware of is that bulk data doesn't currently have a path across processes for e10s.  It's currently implemented only for local transports and socket transports (TCP, USB).

With bulk data, there are now 4 "conversation styles" that are possible:

1. JSON message to actor, JSON reply from actor (the typical one)
2. Bulk message to actor, JSON reply from actor (this is what bulk app install does)
3. JSON message to actor, bulk reply from actor
4. Bulk message to actor, bulk reply from actor

For a non-test example, you can look at the bulk app install actor[1].

Your current example code also appears to use style #2, just like the app install actor.

Your actor should be passed a single object with the properties described in the server's |onBulkPacket| method[2].

The |copyTo| helper (which you can get from the single object, or destructure it in the method signature like the app install actor does) is the simplest way to put the received data somewhere, but it expects an instance of |nsIAsyncOutputStream|.  Using these streaming objects is best, since only a small amount of memory is needed to move data around in chunks.

However, if you know you just want to pull all the received data into a string (which could be a lot of memory depending on what's being done), the actor does also get access to the incoming input stream (via |stream|), which you could provide to something like |NetUtil.readInputStreamToString|.  If this becomes a common thing, we could add a bulk data helper to do this for you.

It might be hard to be very efficient in an example add-on, since it's probably generic code.  If there is a specific task you want to achieve, there may be a more memory-efficient approach than just reading everything into a string.  If you aren't as worried about memory and just want it to be fast, then reading into a string via bulk data should at least still be faster than the base 64 conversion you mentioned.

[1]: http://dxr.mozilla.org/mozilla-central/source/toolkit/devtools/server/actors/webapps.js#169
[2]: http://dxr.mozilla.org/mozilla-central/source/toolkit/devtools/server/main.js#1428-1457
[3]: http://dxr.mozilla.org/mozilla-central/source/toolkit/devtools/server/protocol.js#994

Comment 5

[Jim Blandy :jimb]

Clearing needinfo, as jryans has hopefully answered the question.

Comment 6

[Jan Honza Odvarko [:Honza] (always need-info? me)]

The original requirement for this bug report is no longer relevant. Closing

Honza