Closed Bug 1566755 Opened 5 years ago Closed 5 years ago

Support for WAMP protocol

Categories

(DevTools :: Netmonitor, enhancement, P3)

Product:
DevTools
Component:
Netmonitor
Type:
enhancement

Tracking

(firefox74 fixed)

Status:
RESOLVED FIXED
Milestone:
Firefox 74
Tracking Flags:
Tracking Status
firefox74 --- fixed

People

(Reporter: Honza, Assigned: elad, NeedInfo)

References

(Blocks 2 open bugs)

Details

Attachments

(3 files, 4 obsolete files)

get-ws-subprotocol.diff
6.62 KB, patch
Details | Diff | Splinter Review
WAMP message parsing
125.52 KB, image/png
Details
Bug 1566755 - Adding WAMP support. Supporting JSON, MsgPack and CBOR. Using response Sec-WebSocket-Protocol header.
47 bytes, text/x-phabricator-request
Details | Review

The WebSocket payload preview (in a side panel) should be able to parse WAMP protocol and offer nicely formatted data to the user.

Honza

Hey,

that would be awesome! Being able to directly look at WAMP calls or events - incl app payload - could be super useful for app developers.

FWIW, I left some thoughts on the issue on Autobahn I created to track this and possibly contribute (I am original author of WAMP) in some way https://github.com/crossbario/autobahn-js/issues/481 ..

Cheers,
/Tobias

Tobias, this sounds great!

Some pointers to Firefox Networks panels code base:

  1. WS payload parsing happens here:
    https://searchfox.org/mozilla-central/rev/9df2ccc256fe54c314b34e52920471a0d04b283e/devtools/client/netmonitor/src/components/websockets/FramePayload.js#93-122

  2. Individual supported parsers are located here:
    https://searchfox.org/mozilla-central/source/devtools/client/netmonitor/src/components/websockets/parsers

So, the goal here is to include necessary files that are needed to parse WAMP payload in Firefox (needs license review) and update the parsePayload method mentioned in #1.

Honza

Thanks for the pointers and explanation! This is very helpful.

One problem I see is this: WAMP is able to use different serializers (JSON, CBOR, MessagePack, UBJSON), and the serializer to use on a specific WAMP-WebSocket connection is negotiated in the WebSocket opening handshake (via WebSocket subprotocol identifiers, as in eg wamp.2.cbor vs wamp.2.json.batched - where batched is another serialization related knob that effects the payload format spoken).

However, I don't see how I would keep state, and access the WebSocket subprotocol in use within a new parseWAMPPayload method .. I would need sth like this:

class FramePayload extends Component {

  initConnection(subprotocol) {
      if (subprotocol.substring(0, 4) == "wamp") {
        this.wampSerializer = WampSerializer(subprotocol);
      }
  }

  parsePayload(payload) {
      if (this.wampSerializer) {
          // WAMP processing
          const wampPayload = this.wampSerializer.parse(payload);
      } else {
          // existing code ..
      }
  }
}

Is sth like that possible? What would you recommend?

If I understand correctly, you need access to Sec-WebSocket-Protocol HTTP header in the parsePayload method. Please see the attached patch showing how it can be done.

Honza

Assignee: nobody → odvarko

^ awesome! yes, exactly, that provides what I was looking for;) cool, so I guess we have all the pieces needed for adding basis WAMP message parsing support.

"basic" as in: I could imagine that users would prefer to see eg an (incoming) CALL-RESULT message right next to the original (outgoing) CALL message .. at least in some mode. eg attach any reply messages next to the message rendered for the request message on the timeline of (outgoing) messages in the inspector view. not sure .. but a plain timestamp ordering of both incoming and outgoing WAMP messages might make it hard to look at, at the WAMP interaction action level (subscribe, publish, events, register, calls, invocations).

Ah, I assigned this to myself by accident.

@Tobias, should I assign this to you? Do you have time to put together all the individual pieces?

Honza

Assignee: odvarko → nobody
Flags: needinfo?(tobias.oberstein)

should I assign this to you? Do you have time to put together all the individual pieces?

yes, pls do so. I probably won't hack on the PR myself, but we can organize coding hands - hopefully. I think I have enough input (thanks!) to guide on design on sketch out code ..

Flags: needinfo?(tobias.oberstein)

Awesome!

Honza

Assignee: nobody → tobias.oberstein
Status: NEW → ASSIGNED

Hey Honza and Tobias. I wrote the WAMP parsing code a few years ago (see this issue). Anyway, I transpiled it to ES2015 and you can find the new code here.

The usage is parseWampMessage(JSON.parse(message)), where message is the text message you received from the WebSocket. You can also add support for msgpack parsing by parsing the msgpack using an external library and then passing the array to parseWampMessage. I might try to create a patch that integrates this to the inspector, but currently I'm having a hard time setting up a Firefox development environment on my computer.

Elad

Hi Elad, thanks for the update!

You might follow this page with instructions about how to get started.
https://docs.firefox-dev.tools/getting-started/

It would be awesome to have WAMP supported!

Honza

Hi Honza,

Is there a shortcut to this build process? I cloned the mozilla-central repository (it took me more than 3 hours yesterday) and I'm having trouble building this on my computer. It might have to do with the fact I'm using Windows. Is there a way to somehow just change the Javascript files without building this entire thing? I read about build artifacts but these also fail on my machine. The instructions also say

Getting the code
This will take a long time, because the repository is B I G (Firefox is more than 10 years old and there are lots of commits every day!). So please be patient.
You will also need a lot of disk space (about ~40Gb free).

And while I understand this, it seems to require a lot of effort for contributing a small patch. Here are the local changes I made. I would be happy if there is any way to view them without having to work so hard.

Elad

Cloning of the repo takes time and there is no workaround.
But, building can be significantly faster when using artifact build

In your .mozconfig file:
ac_add_options --enable-artifact-builds

I am on Windows and it works for me.
https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Windows_Prerequisites

Honza

Attached file Bug 1566755 - initial work (obsolete) —
Attached image WAMP message parsing —

It took me some effort (I can't believe I had to install PHP on my machine for this!), but here's an initial implementation of WAMP message support.

Excellent!

I'll take a look at the patch.

Thanks,
Honza

Attached file Bug 1566755: fixing eslint errors (obsolete) —

Depends on D57439

I fixed the eslint errors (I think).

I'll use the Sec-WebSocket-Protocol header in a future revision where I will add message pack support.

Elad

Attachment #9117014 - Attachment description: Bug Bug 1566755 - removing redundant space for eslint → Bug 1566755 - removing redundant space for eslint

Hi Honza,
Just one small thing. I want to implement MsgPack support, and for that I need to use an npm package (msgpack5). What is the required procedure to add a reference to such library? I guess it involves specifying a package.json file with the dependency. Is there anything else that needs to be done? Can you please link me to another place where they refer to an external library?

Thanks
Elad

ni Honza for comment above.

Flags: needinfo?(odvarko)

Might be also an option to land this first and follow up with MspPack support in a new bug. See bug 1589068 and license bug 1595738 for an example that adds a new library.

(In reply to :Harald Kirschner :digitarald from comment #21)

Might be also an option to land this first and follow up with MspPack support in a new bug. See bug 1589068 and license bug 1595738 for an example that adds a new library.

Yes, I agree.

Elad, can add the support for MsgPack in a follow up bug? Can you please file it?

Thanks for the help!
Honza

Flags: needinfo?(odvarko)
Attachment #9117014 - Attachment is obsolete: true
Attachment #9116973 - Attachment is obsolete: true
Attachment #9116332 - Attachment is obsolete: true

Sorry about all these revisions, but I think this one should be the final one (up to review comments). I use the Sec-WebSocket-Protocol response header and create an appropriate WAMP serializer according to the specified string. Works well with JSON, MessagePack and CBOR.

Attachment #9117206 - Attachment is obsolete: true
Pushed by jodvarko@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/82396fd59d58 Adding WAMP support. Supporting JSON, MsgPack and CBOR. Using response Sec-WebSocket-Protocol header. r=Honza,mhoye

Backed out changeset 82396fd59d58 (Bug 1566755) for ES lint failure on wamp/serializers.js

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&fromchange=82396fd59d5881e604eefd9561829b3e1082eb77&tochange=15dfb02919db9c2b212a2f72eaafca0a19eb4660&selectedJob=284351698

Backout link: https://hg.mozilla.org/integration/autoland/rev/15dfb02919db9c2b212a2f72eaafca0a19eb4660

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=284351698&repo=autoland&lineNumber=236

[task 2020-01-10T09:52:15.747Z] Installing setuptools, pip, wheel...
[task 2020-01-10T09:52:17.890Z] done.
[task 2020-01-10T09:52:19.248Z] b"running build_ext\nbuilding 'psutil._psutil_linux' extension\ncreating build\ncreating build/temp.linux-x86_64-3.6\ncreating build/temp.linux-x86_64-3.6/psutil\nx86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=563 -DPSUTIL_LINUX=1 -I/usr/include/python3.6m -I/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init_py3/include/python3.6m -c psutil/_psutil_common.c -o build/temp.linux-x86_64-3.6/psutil/_psutil_common.o\nx86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=563 -DPSUTIL_LINUX=1 -I/usr/include/python3.6m -I/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init_py3/include/python3.6m -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-3.6/psutil/_psutil_posix.o\nx86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=563 -DPSUTIL_LINUX=1 -I/usr/include/python3.6m -I/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init_py3/include/python3.6m -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-3.6/psutil/_psutil_linux.o\ncreating build/lib.linux-x86_64-3.6\ncreating build/lib.linux-x86_64-3.6/psutil\nx86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/psutil/_psutil_common.o build/temp.linux-x86_64-3.6/psutil/_psutil_posix.o build/temp.linux-x86_64-3.6/psutil/_psutil_linux.o -o build/lib.linux-x86_64-3.6/psutil/_psutil_linux.cpython-36m-x86_64-linux-gnu.so\nbuilding 'psutil._psutil_posix' extension\nx86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=563 -DPSUTIL_LINUX=1 -I/usr/include/python3.6m -I/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init_py3/include/python3.6m -c psutil/_psutil_common.c -o build/temp.linux-x86_64-3.6/psutil/_psutil_common.o\nx86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=563 -DPSUTIL_LINUX=1 -I/usr/include/python3.6m -I/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init_py3/include/python3.6m -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-3.6/psutil/_psutil_posix.o\nx86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/psutil/_psutil_common.o build/temp.linux-x86_64-3.6/psutil/_psutil_posix.o -o build/lib.linux-x86_64-3.6/psutil/_psutil_posix.cpython-36m-x86_64-linux-gnu.so\ncopying build/lib.linux-x86_64-3.6/psutil/_psutil_linux.cpython-36m-x86_64-linux-gnu.so -> psutil\ncopying build/lib.linux-x86_64-3.6/psutil/_psutil_posix.cpython-36m-x86_64-linux-gnu.so -> psutil\n"
[task 2020-01-10T09:52:19.248Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2020-01-10T09:52:19.606Z] 09:52:19.605 eslint (99) | Passing the following paths:
[task 2020-01-10T09:52:19.606Z] /builds/worker/checkouts/gecko
[task 2020-01-10T09:52:19.612Z] 09:52:19.612 eslint (99) | Command: /usr/local/bin/node /builds/worker/checkouts/gecko/node_modules/eslint/bin/eslint.js --ext [js,jsm,jsx,xul,html,xhtml] --format json --quiet --ignore-pattern node_modules --ignore-pattern dom/tests/mochitest/ajax --ignore-pattern browser/extensions/formautofill/content/third-party --ignore-pattern devtools/client/shared/build/babel.js --ignore-pattern dom/media/platforms/ffmpeg/ffmpeg58 --ignore-pattern media/openmax_dl --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module.h --ignore-pattern dom/media/webspeech/recognition/energy_endpointer_params.h --ignore-pattern media/libvpx --ignore-pattern media/libpng --ignore-pattern tools/fuzzing/libfuzzer --ignore-pattern js/src/zydis --ignore-pattern mobile/android/geckoview/src/thirdparty --ignore-pattern ipc/chromium/src/third_party --ignore-pattern media/kiss_fft --ignore-pattern media/libopus --ignore-pattern toolkit/components/normandy/vendor --ignore-pattern gfx/qcms --ignore-pattern media/webrtc/signaling/src/sdp/sipcc --ignore-pattern media/libvorbis --ignore-pattern dom/media/webspeech/recognition/endpointer.h --ignore-pattern testing/xpcshell/node-ip --ignore-pattern mfbt/double-conversion/double-conversion --ignore-pattern testing/talos/talos/tests/kraken --ignore-pattern gfx/harfbuzz --ignore-pattern dom/u2f/tests/pkijs --ignore-pattern gfx/graphite2 --ignore-pattern dom/canvas/test/webgl-conf/checkout --ignore-pattern toolkit/components/jsoncpp --ignore-pattern dom/media/platforms/ffmpeg/ffmpeg57 --ignore-pattern testing/talos/talos/tests/dromaeo --ignore-pattern devtools/shared/heapsnapshot/CoreDump.pb.cc --ignore-pattern js/src/vtune/disable_warnings.h --ignore-pattern gfx/wr --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module_proxy.h --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module_ext.h --ignore-pattern xpcom/io/crc32c.c --ignore-pattern dom/media/platforms/ffmpeg/libav53 --ignore-pattern netwerk/dns/nsIDNKitInterface.h --ignore-pattern testing/gtest/gmock --ignore-pattern tools/lint/test/files --ignore-pattern media/libaom --ignore-pattern browser/components/pocket/content/panels/js/vendor --ignore-pattern devtools/shared/jsbeautify --ignore-pattern modules/brotli --ignore-pattern dom/media/webspeech/recognition/endpointer.cc --ignore-pattern media/libsoundtouch --ignore-pattern dom/webauthn/tests/pkijs --ignore-pattern media/libspeex_resampler --ignore-pattern dom/webauthn/cbor-cpp --ignore-pattern browser/components/newtab/vendor --ignore-pattern toolkit/crashreporter/google-breakpad --ignore-pattern gfx/vr/service/openvr --ignore-pattern media/libnestegg --ignore-pattern js/src/vtune/jitprofiling.h --ignore-pattern security/nss --ignore-pattern gfx/sfntly --ignore-pattern obj-x86_64-pc-linux-gnu --ignore-pattern media/libwebp --ignore-pattern js/src/dtoa.c --ignore-pattern js/src/vtune/ittnotify_static.c --ignore-pattern toolkit/components/protobuf --ignore-pattern dom/tests/mochitest/dom-level2-html --ignore-pattern gfx/skia --ignore-pattern js/src/vtune/ittnotify.h --ignore-pattern js/src/octane --ignore-pattern extensions/spellcheck/hunspell/src --ignore-pattern testing/xpcshell/dns-packet --ignore-pattern js/src/jit/arm64/vixl --ignore-pattern devtools/shared/heapsnapshot/CoreDump.pb.h --ignore-pattern devtools/client/debugger/flow-typed/npm --ignore-pattern dom/tests/mochitest/dom-level1-core --ignore-pattern security/sandbox/chromium --ignore-pattern editor/libeditor/tests/browserscope/lib --ignore-pattern modules/woff2 --ignore-pattern media/libdav1d --ignore-pattern toolkit/crashreporter/breakpad-client --ignore-pattern gfx/ots --ignore-pattern xpcom/build/mach_override.h --ignore-pattern media/libjpeg --ignore-pattern parser/expat --ignore-pattern gfx/ycbcr --ignore-pattern mozglue/misc/decimal --ignore-pattern devtools/shared/acorn --ignore-pattern dom/media/webspeech/recognition/energy_endpointer_params.cc --ignore-pattern security/sandbox/chromium-shim --ignore-pattern media/libmkv --ignore-pattern testing/talos/talos/tests/v8_7 --ignore-pattern testing/mochitest/pywebsocket --ignore-pattern modules/xz-embedded --ignore-pattern dom/media/webspeech/recognition/energy_endpointer.cc --ignore-pattern testing/gtest/gtest --ignore-pattern media/libcubeb --ignore-pattern build/pymake --ignore-pattern js/src/vtune/ittnotify_types.h --ignore-pattern tools/profiler/core/vtune --ignore-pattern third_party --ignore-pattern testing/web-platform/tests/tools/third_party --ignore-pattern js/src/editline --ignore-pattern js/src/vtune/legacy --ignore-pattern js/src/ctypes/libffi --ignore-pattern browser/components/translation/cld2 --ignore-pattern media/openmax_il --ignore-pattern xpcom/build/mach_override.c --ignore-pattern modules/zlib --ignore-pattern other-licenses --ignore-pattern testing/web-platform/tests/resources/webidl2 --ignore-pattern dom/media/gmp/rlz --ignore-pattern netwerk/srtp/src --ignore-pattern media/mtransport/third_party --ignore-pattern js/src/vtune/ittnotify_static.h --ignore-pattern modules/fdlibm --ignore-pattern browser/extensions/screenshots/build/raven.js --ignore-pattern gfx/angle/checkout --ignore-pattern modules/freetype2 --ignore-pattern toolkit/components/certviewer/content/vendor --ignore-pattern toolkit/components/utils/mozjexl.js --ignore-pattern media/libtheora --ignore-pattern media/ffvpx --ignore-pattern devtools/client/shared/vendor --ignore-pattern netwerk/sctp/src --ignore-pattern gfx/cairo --ignore-pattern dom/media/platforms/ffmpeg/libav55 --ignore-pattern devtools/client/shared/sourceeditor/codemirror --ignore-pattern media/webrtc/trunk --ignore-pattern media/libogg --ignore-pattern media/libtremor --ignore-pattern dom/media/gmp/widevine-adapter/content_decryption_module_export.h --ignore-pattern nsprpub --ignore-pattern intl/icu --ignore-pattern js/src/vtune/ittnotify_config.h --ignore-pattern toolkit/components/url-classifier/chromium --ignore-pattern dom/tests/mochitest/dom-level2-core --ignore-pattern media/libyuv --ignore-pattern dom/media/webspeech/recognition/energy_endpointer.h --ignore-pattern js/src/vtune/jitprofiling.c --ignore-pattern dom/media/platforms/ffmpeg/libav54 /builds/worker/checkouts/gecko
[task 2020-01-10T10:06:08.921Z] 10:06:08.921 eslint (99) | Finished in 829.42 seconds
[task 2020-01-10T10:06:08.925Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/devtools/client/netmonitor/src/components/websockets/parsers/wamp/serializers.js:7:28 | relative paths are not allowed with require() (mozilla/reject-relative-requires)
[taskcluster 2020-01-10 10:06:09.472Z] === Task Finished ===
[taskcluster 2020-01-10 10:06:10.135Z] Unsuccessful task run with exit code: 1 completed in 1025.303 seconds

Flags: needinfo?(tobias.oberstein)
Pushed by jodvarko@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/3af87b72a2b4 Adding WAMP support. Supporting JSON, MsgPack and CBOR. Using response Sec-WebSocket-Protocol header. r=Honza,mhoye

https://hg.mozilla.org/mozilla-central/rev/3af87b72a2b4

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
status-firefox74: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 74
Assignee: tobias.oberstein → elad
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: