Open Bug 836897 (webmidi) Opened 12 years ago Updated 5 months ago

[meta] Implement the WebMIDI API

Categories

(Core :: DOM: Device Interfaces, enhancement, P3)

Product:
Core
Component:
DOM: Device Interfaces
Type:
enhancement

Tracking

()

Milestone:
99 Branch
Project Flags:
Webcompat Priority P3
Tracking Flags:
Tracking Status
relnote-firefox --- -
firefox98 --- disabled
firefox99 --- disabled

People

(Reporter: cwilso, Unassigned)

References

(Depends on 5 open bugs,
URL
)

Details

(Keywords: dev-doc-complete, meta)

Attachments

(1 file)

Output of `aconnect -l`
3.14 KB, text/plain
Details
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.28 (KHTML, like Gecko) Chrome/26.0.1396.1 Safari/537.28 Expected results: Firefox should implement the Web MIDI API (http://webaudio.github.com/web-midi-api/). This W3C specification-in-progress enables the web platform to gain access to MIDI input controllers, such as keyboard, drum and DJ controllers - it also enables access to external and internal synthesizers, such as the built-in software synthesizers on Windows and Mac OS. MIDI connections are extremely common in musical applications, and have become increasingly popular on mobile and tablet devices as well (MIDI is supported by iOS via CoreMIDI, and the iOS App Store has >400 MIDI applications. This has become so popolar that many MIDI keyboard controllers are available with iPad slots.) Support for Web MIDI, along with Web Audio, enables many music production scenarios, from notation entry and playback a la Noteflight.com to live synthesizers (e.g. http://webaudiodemos.appspot.com/midi-synth/index.html, as well as numerous iOS synthesizers from names like Roland, Yamaha and Korg), as well as providing programming interfaces to hardware synthesizers (E.g Roland's JP Synth Editor for iPad, used with their Jupiter-50 and Jupiter-80 synthesizers).
URL: http://webaudio.github.com/web-midi-api/
OS: Mac OS X → All
Hardware: x86 → All
Component: Untriaged → DOM: Other
Product: Firefox → Core
Component: DOM: Other → Video/Audio
This is much lower priority than Web Audio (and many other things) IMHO.
Chris, thanks for your suggestion. Marking this as a possible future enhancement.
Severity: normal → enhancement
Status: UNCONFIRMED → NEW
Ever confirmed: true
To me this sounds like a perfect target for maybe an intern to take on? Either way, I'm obviously in favor of this. I can also see the benefit of this API in things such as low-end mobile games (when you pay for transfer, MIDI starts sounding surprisingly good).
I'd personally prefer the intern to work on Web Audio first. It is of much higher priority.
I just wanted to vocalize official support regarding how important this feature is. The ability to add MIDI capability to web applications would open up massive new avenues for online music creation, collaboration, entertainment and education. As great as web audio is, the inability to flexibly manipulate it is a big drawback. We've already started to build support for our hardware (the gTar) in web midi through the use of the poly-fill / Jazz Soft midi plug in. The ability to build/deploy music related applications along with the coupling of Web MIDI is extremely exciting. If needed we would be willing to provide support however possible and within our means! Thanks, Idan
(In reply to comment #5) > If needed we would be willing to provide support however possible and within > our means! Are you interested in submitting patches to implement Web MIDI in Gecko?
I'd definitely agree that this API should be implemented in a not too distant future. While it's great that Web Audio is coming along, having this API in parallell would make a world of difference in terms of professional audio on the web. Having gone the long route implementing a musical "keyboard" on the computer keyboard, I can honestly say I wish this was around when we made Jam With Chrome, and I'd love to use it in a lot of coming projects too.
This API is really important, as MIDI is the universal standard for real-time I/O controllers of many kinds (not just music performance; also lighting, mixing, and much more). Supporting MIDI is also the obvious complement to Web Audio API support, allowing the browser to support a full spectrum of applications with real-time musical input from the user.
As the software lead at a MIDI controller manufacturer, we have already charted out new ways of reaching customers and customers reaching each other that uses MIDI in the browser. The potential is overwhelming! There is also a great cultural benefit in preserving and distributing non-fixed media artworks, such as algorithmic composition, many (if not most) of which are composed with MIDI. There is no reason that the only device that can control a browser is a keyboard, and there are millions of devices out there that speak MIDI. Beyond music applications, the industrial applications for show control would explode. Every major OS has seen fit to include MIDI natively - as the browser becomes the default "operating system" for screens around the world, this established and important hardware interface needs to be part of this growth.
We all agree this should be implemented. The question is who is going to do it and whether other things that are _also_ important should be dropped to implement this. Again, if people are willing to step up and help implement that would probably help a lot in terms of finding people to work on this.
I wanted to cast my vote of support for getting the MIDI API implemented. Although many seem to not understand the importance of MIDI and it's status as a long-standing standard for real-time I/O control, I personally cannot imagine us achieving the dream of the "browser being the platform" without both low-level audio AND MIDI support. If every OS on the market found the need to add MIDI functionality to their product, so should browser developers. From lighting and show control, to music education, to gaming, to controlling industrial machinery, MIDI has seemingly limitless applications. If you ponder MIDI’s potential in the home automation market alone, it is mind-boggling. Having said that, I will now tell you that I work for one of the biggest audio manufacturers in the world. Obviously, MIDI plays a huge part in most things we do...but that is probably not enough to have the API put in a browser. It should be in the browser because of the potential for developers to make applications that reach out of the browser and interact with the "real" world. Again, the applications are almost limitless. My company is a month away from shipping the follow-up to one of our biggest selling products in our history, with expected initial sales to be thousands and thousands of units a month. This products will ship with a PC/Mac desktop Editor/Librarian app. We have spent a considerable amount of time and resources developing this app. Before the product even ships, we are already beginning the process of porting the app over to the browser, utilizing the poly-fill/Jazz Soft midi plug in. If history repeats itself, that would be potentially 500,000 users of the web app. Strong stuff. In closing, I am not excited about this because of who I work for. I am excited about this because I spent 15 years as a web application developer and I am passionate about the potential of the platform. Scott Mire Peavey Electronics
Guys, this is not the correct place to advocate for this. Nobody has said that we _don't_ want to support Web MIDI. It's just that there are a lot of more important things for us to work on.
Responding to a feature request by advocating lower priority, and then telling the community that they're out of place for advocating higher priority, is unsupportable. This absolutely is the place to advocate. +1 for Web MIDI. It's a higher priority than Web Audio for lots of people. Web MIDI would allow the genesis of a new class of web based applications that leverage the vast ecosystem of MIDI-enabled technologies. Boris has the correct view that it's a matter of resourcing, and I get what Ehsan is saying. I want to be constructive here, so I'll see if I can drum up support from our OttawaJS meetup. We have a lot of good devs including a couple of Mozilla interns.
I'll give this a shot, though it won't be a full time initiative (I've still got lots of commitments on FxOS). I've done a bit of MIDI driver work (hi pete!) as well as WebAPI development (RIL and bluetooth for FxOS, gamepad oop still in development), so I'm capable of breaking lots of things. If there's anyone watching this that's interested in helping out (platform specific MIDI would be great, I've got some experience on mac and linux but haven't worked on MIDI code in windows), please let me know by commenting here. First task is probably dividing down into subtasks, getting WebAPI matching the current spec state and fleshing out HAL needs per platform.
Assignee: nobody → kyle
Kyle, I started looking into the rtmidi open source cross platform midi classes. Didn't get as far as writing an xpcom component but I'm still interested and would like to help.
Things have changed enough that there probably won't be XPCOM involved (See the recent post on dev.platform about no more xpcom for web facing objects). Off the top of my head, I'm guessing we'll do all of the platform work back in the HAL, have an IPDL protocol in front of that for IPC, then the DOM side will be WebIDL backed by whatever language is easiest and fulfills needs (been a while since I've touched DOM bindings so I'm not sure if we can deal with events in js impls yet). rtmidi looks neat, I'll check it out, though I doubt we'll be able to use it. Chrome's also got a webmidi implementation already, though it needs to be pref'd on. Not sure if that landed in Webkit before the fork or went with blink though. Something to look at though. Gonna try to get a branch running off my gecko-dev repo ASAP, will update bug once that's done. Will also break out above tasks into blocking bugs for this so it'll be easier to figure out what's needed.
Events in JS impls work fine, both in terms of subclassing Event and subclassing EventTarget.
Are we sure we want to implement this in javascript ? I'm all for implementing stuff in javascript, but the purpose of MIDI is kind of having very tight scheduling and super low latency.
You will definitely want the delivery of scheduled outgoing messages and timestamping of incoming messages to not be done in JS, as those need to be tightly scheduled*; but the actual delivery of messages ends up being events or JS method anyway, so not critically bad. * i.e. the WebMIDIAPIShim polyfill can never be good enough, because it has to use setTimeout to deliver scheduled-in-the-future send() messages. However, the underlying NPAPI control is delivering messages with system-level timestamps, so the input part is okay in terms of timing.
Yeah, that's the idea. The way it looks in my head currently is: OS API <-> HAL (C++) <-> IPDL <-> DOM API (JS) Everything that has to do with timing and communications to the world will happen in the HAL, as putting that before the JS/IPDL would be questionable in terms of scheduling. Doing the DOM work to get things out to content is mostly to keep things cleaner, though if it for some reasons causes massive issues it can always be rethought.
Right. The part you'll have to do yourself in that case is scheduling on Windows; the Windows MIDI API doesn't have a "send this message at xxx time" method, or a system-level timestamp on incoming messages. (CoreMIDI on OSX does; it has timestamps on MIDIPackets.) That will have to not be blocked on JS execution or GC, so it will need to happen on the left side of IPDL.
If we want to ever use that in workers (and I think it's likely) we can't use js. Easy decision ;)
Well, there you go then. Access to Web MIDI from Workers is fairly high priority in the issues list - https://github.com/WebAudio/web-midi-api/issues/99.
Ok, not a prob, glad this got hashed out now then. Not like there's much happening up there anyways, was just trying to make the boilerplate simpler. :)
Alias: webmidi
Summary: Implement the Web MIDI API → Implement the WebMIDI API
And so it begins. https://github.com/qdot/gecko-dev/tree/836897-webmidi Also posted intent to implement on dev-platform and dev-webapi. So we'll see how that goes. Right now I'm just getting WebIDL and DOM boilerplate in. Then it's on to IPDL (the protocol should be pretty simple), then a HAL template that'll send over test data so I can run mochitests. Once the HAL template is done, we can start creating platform specific versions to actually make things work. Going to break this out into multiple bugs, one for WebIDL and DOM implementation, and one for each platform for the HAL. Assuming this actually keeps momentum for more than like, today, I'll go ahead and schedule a sec review, though I imagine due to priority we'll sit on the bottom of the pile for a while.
Keywords: dev-doc-needed
A new draft was published last week: http://webaudio.github.io/web-midi-api/
Branch at https://github.com/qdot/gecko-dev/tree/836897-webmidi Is now up to date and back in motion. Right now, working on all of the DOM/IPC plumbing. Goal is to get that done with some sort of simple shim layer under it for testing/completeness sake, then starting work on the platform specific bits.
Just finished the permissions dialog implementation for this, filing for sec-review early since this is going to be an API used on desktop that hits hardware.
Flags: sec-review?
Yup, we've been talking to the Chrome people, super happy to see them finally get it shipped. Things are moving along on this side, bug 1123516 (which is a requirement for the WebIDL needs for WebMIDI) is almost done, at which point hopefully there will be more motion over here.
So, since I haven't updated this bug in forever... Assuming anyone really wants to follow code, my dev branch is at https://github.com/qdot/gecko-hg/tree/836897-webmidi now. I've been working on this full time for a while now, and am almost done with the gecko specific parts. Most of the WebAPI is implemented (both in-process and e10s'd), mainly down to filling out unit tests and making sure we comply with the spec. Once that's done, I'll be putting the patch up for review (and apologizing in advance to reviewers, it's gonna be a big one. :/ ), and starting work on the platform specific portions.
Really excited to see this. Ping me if there's anything I can do to help in testing, etc.
(In reply to Chris Wilson from comment #32) > Really excited to see this. Ping me if there's anything I can do to help in > testing, etc. Chris, are there web platform tests?
I am writing platform test. But it is not ready to submit yet.... Can you use that test which is still creating by me?? https://github.com/ryoyakawai/web-platform-tests/tree/submission/webmidi/webmidi
Component: Audio/Video → DOM: Device Interfaces
So, as some might've seen on twitter, I've got hardware interaction working on OS X. This includes device enumeration, and message sending/receiving. That said, this is still what I'd classify as a pre-alpha state, because there are quite a few hacks that need to be cleaned up before I can get this into reviews. The current todo list, off the top of my head: - DOM -- Packet scheduler (right now we ignore timestamps and just throw stuff over) -- Clear() handling to spec -- Fix port information handling/passing -- Cleanup and comment fixing -- Permissions string l10n - Mac -- Device notification handling (Having CFRunLoop issues) -- General cleanup -- Figuring out licensing (I adapted this off of chromium's midi_manager_mac object) - Windows -- Not Started - Linux -- Not Started - Android -- Not started, nor high priority. If you're an android dev that wants to do this, please feel free to take the bug! I've turned this bug into a metabug to track all of the MIDI work, so it can land separately and we can get testing happening ASAP.
This is awesome! I've been looking forward to this for some time. I am working on automated WebVR input latency testing (possibly to be used with Talos) and can see how the same is important for WebMIDI. The concept is to implement a puppet that can be scripted to fire events with precise timing and measure response time. Perhaps the same system could be used for WebMIDI as well.
Any update/target date on WebMIDI support? Talked to a number of manufacturers at NAMM last week who'd like to see it. :)
(In reply to Chris Wilson from comment #37) > Any update/target date on WebMIDI support? Talked to a number of > manufacturers at NAMM last week who'd like to see it. :) So far the project is Kyle's - I think there are a bunch of MIDI nerds in Mozilla ( myself included! ) that are excited about it and Kyle is making progress but it's unlikely to attract additional resources unless through community contribution.
So here's where things are: - Backend DOM Stuff: Like 97% done. Was gonna try to get the timestamp stuff cleaned up, got stalled on that in November, then got yanked onto some other stuff. I'm going to try to get it rebased, slightly more commented, and into review in the next couple of weeks, barring any major interruptions. I suspect reviews are going to take a while though, it's a big ol' chunk of code. - OSX Platform Specifics: Basically still working like it always did. With some of the work happening on moving the Gamepad API to our PBackground IPC mechanism, they solved one of the issues I'd run into with device hotplugging, so we need to integrate that. - Linux/Windows Platform Specifics: Unfortunately no real movement here yet. I've talked to Adam Goode from the Blink team about this a bit, and he said he'd at least possibly be available to answer questions. We'll probably use the chromium implementation for reference, much as we did the OS X backend.
You give me too much credit! I've not done much for Chromium/Blink outside of Linux WebMIDI. But happy to help as much as I can.
An update for anyone watching this, the DOM implementation of WebMIDI is now in for review on Bug 1201590. I would suspect this is going to take a few (or more) review rounds, and this doesn't involve anything platform specific, but it's a step in a direction!
Flags: sec-review?
Any news about that API?
Kiiiiiiiinda? Right now I'm busy on WebAuthn implementation, which was higher priority. However, it shares some aspects of this bug, in that the hardware access code will be rust, and it has a somewhat similar IPC setup. The architectural questions I have to answer to finish WebAuthn will at least give us a path for parts of WebMIDI implementation. That said, I still don't know when I'm gonna have time to work on it. :(
Now that version 52 has stripped out all NPAPI plugins, including mozplugger, this is more urgently needed. Firefox currently has no support for MIDI at all :(
We should be aware of, that MIDI susport means two different things. This issue is about Web MIDI API, that is an API to attach MIDI devices to the browser. It does however NOT cover playback of Standard MIDI files in any way. With mozplugger now dead, the playback of MIDI files has finally gone. Web sites may use http://www.midijs.net as a intermediate replacement to play MIDI files from their pages. However, a different issue shoud be openend (or does already exist?) to support playback of MIDI files via <audio>.
After reading the description: >it also enables access to external and internal synthesizers, such as the built-in software synthesizers on Windows and Mac OS I was under the impression the WebMIDI API could be used to pipe MIDI data to an external synthesizer. Do you mean there'd be no interface, internal to firefox, to connect MIDI files embedded in webpages to the Web MIDI API? If firefox is going to support this API, it doesn't seem so far-fetched that a way could be found to do that.
>external synthesizer It occurs to me that these terms are problematic; I don't mean a separate piece of hardware. This is how most programs using MIDI for audio work. One usually configures, within the program, a driver or port to send the MIDI data to, and then that driver or something listening on that port handles it from there. In Windows and MacOS they are shipping standardized software synthesizers; in Linux there are several available, but timidity++ is probably the best target for a default. MIDI is not rendered as audio data inside the program with MIDI music; it is exported to the driver or port and rendered into audio outside of the program. I think this can be achieved by WebMIDI API in firefox, if there would be an interface to connect embedded midi files to WebMIDI API and firefox had cross-platform ready configuration to pipe that data out to a synthesizer.
I was just wondering what the status of this is (from a user's perspective)? Are we talking months or years for this to be finished?
David, I haven't spoken to Kyle (who's the person working on this), but from his last comment ( https://bugzilla.mozilla.org/show_bug.cgi?id=836897#c43 ) I would expect years rather than months, unless someone else works in the feature. From what I read he is not doing work on it right now. We're doing lots of internal low level work in Firefox this year to make it blazing fast for lots of people and Web MIDI is less important as there's fewer people who might use it in comparison. Good news is part of Kyle's work right now might overlap with the work required for Web MIDI, so there's that.
Yeah, as :sole said, I'm mostly off working on things related to DOM/NPAPI at the moment. I'm also helping out with the U2F/WebAuthn project, which will most likely be our first project that uses Rust for the systems call side of hardware access. Once we've got that figured out, we'll at least have a roadmap for what the systems side of WebMIDI should look like. To get this going again, the patches here are going to need a cleanup at least, as well as bringing up to date with some of the features that've been added to gecko since I last worked on this. Then we'll need to write the Rust portions of the system to actually talk to MIDI hardware, though from what I've seen on crates.io, it looks like there's already code around to do that, so hopefully it'll mostly be a matter of code reuse.
Hello! I just ran into this today trying to connect my MIDI controller to some webapps. Found it very frustrating that the solution was to 'install Chrome' I've decided this aggression will not stand and that I would like to contribute. (Disclaimer: this would be my first time examining the project's source) If this is possible from an open source perspective, where can I get started collecting the necessary information? If not, are there any open applications for positions that could have me working on this? I currently do alot of Clojure & Java at work, and alot of music (MIDI!!!!) outside of work. Cheers, Rob
Hi! If you are interested in contributing code, a general introduction to the codebase is available here: https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Introduction As far as starting on the WebMIDI bits in particular, I would defer to Kyle.
Just for your interest: https://addons.mozilla.org/en-US/firefox/addon/midi-input-provider/ No, this is no full WebMIDI implementation. It is just for MIDI input. I just wanted some simple, easy solution to use piano learning platforms from Firefox. So far this is Linux only, but additional "native apps" for other operating systems would be possible. I'll update this addons maximum allowed Firefox version as soon as a builtin implementation is available, so the Addon gets incompatible with the first Firefox version with builtin MIDI support. I'm not able to do an implementation for Firefox directly as I don't know anything about Rust and couldn't find good beginners information, so far. I also could only do a Linux implementation anyway. To whoever wants/is able to implement this: Please don't forget to ask the user for permission to hand over its MIDI hardware. Google Chrome just gives permission immediately without asking. IMHO this is a very bad idea. It gives additional information for tracking users (only a few people have MIDI hardware) and potentially also allows to control MIDI devices even if the user didn't permit a website to do so.
Amazing job Manuel! (I will try asap)! Looking on Amo I discovered this one for the MIDI support, seems that add the webshim. https://addons.mozilla.org/en-US/firefox/addon/jazz-midi/?src=cb-dl-created In any case seems that the interesting on this API lately is getting interest but Firefox is behind to the others competitor :-/
(In reply to Manuel Reimer from comment #53) ... > To whoever wants/is able to implement this: Please don't forget to ask the > user for permission to hand over its MIDI hardware. Google Chrome just gives > permission immediately without asking. IMHO this is a very bad idea. It > gives additional information for tracking users (only a few people have MIDI > hardware) and potentially also allows to control MIDI devices even if the > user didn't permit a website to do so. This is an excellent point. Kyle, I realize this work is not a priority right now ( although I am a huge fan! ) - does the spec address permissions to midi devices?
Flags: needinfo?(kyle)
The spec requires permission for sysex access to devices, but that's it. As far as I remember, regular devices return with no prompt on Chrome, but there's nothing saying we can't do that here. The function that returns devices does so via a promise, so it's already async anyways. This is, of course, assuming this bug is ever resumed.
Flags: needinfo?(kyle)
Last I checked, Chrome plans to require MIDI permissions for non-sysex MIDI: https://bugs.chromium.org/p/chromium/issues/detail?id=585735 See https://googleprojectzero.blogspot.com/2016/02/racing-midi-messages-in-chrome.html for one motivation (there are other reasons).
Oh, great! That always kinda worried me, glad it's on the list of things to fix. Thanks for the update!
This is progressing on the Mozilla side again. Bug 1201590 should land within the next week (we're down to one review and me doing some test fixing), with the API pref'd off completely because we only have a test platform in that bug. WebMIDI will be pref'd on for nightly once we have actual platform support on at least one platform, most likely macOS. Note that we will be requiring user permissions for both non-sysex and sysex access. Also, we will only be exposing WebMIDI over secure contexts.
Priority: -- → P3
(In reply to Kyle Machulis [:qdot] [:kmachulis] (if a patch has no decent commit message, automatic r-) from comment #59) > This is progressing on the Mozilla side again. Bug 1201590 should land > within the next week (we're down to one review and me doing some test > fixing), with the API pref'd off completely because we only have a test > platform in that bug. WebMIDI will be pref'd on for nightly once we have > actual platform support on at least one platform, most likely macOS. As an implementer of one of the more complex Web MIDI applications (https://components.novationmusic.com) I would love to take this opportunity to thank you for picking this up, Kyle. We would love to help, if we can, specifically with testing - We actually found a couple of bugs in Chrome's implementation by simply using it and having a couple of thousand users finding every problem on the way :) There's two things I can immediately see where we could help: 1. Testing this in the earliest stage possible. As this bug turned into quite a complicated network of intertwined bugs, could you simply let us know as soon as MIDI comms are possible in nightlies on any of the platforms? Our app probably contains a couple of Chromisms and the earlier I can run our app against the real thing, the earlier I can get rid of those. 2. Supplying hardware so that you can check the implementation against a substantial real world app. If you're interested, ping me and I'll try to get you a test device. (Unless you already own a Novation Circuit, that is) Again, thanks for setting this in motion again, I'm really happy to see this bug spring back to life.
We are now discussing WebMIDI on the Mozilla platform position repo at https://github.com/mozilla/standards-positions/issues/58 This discussion includes coverage of some of the security aspects we may have issues with.
Just chiming in to say that Ableton would also be very excited to see this make its way into a release of Firefox. We can help by testing with preliminary builds as well as providing hardware for implementers to test with.
"Just chiming in" to say that about:config didn't show up in here, and that I found dom.webmidi.enabled, which was false by default on my Ubuntu installation (which I use as my primary computer, even though it's so old that I somehow chipped off the CD drive ... thingy), but when I changed it to true, my device still wouldn't work, but I won't blame it on Firefox, but instead that I can't find drivers ^_^
There's no platform specific code in firefox for WebMIDI currently, so it won't work on any platform. What's there is only stubs for testing.
status-firefox66: --- → affected
Flags: webcompat?
Whiteboard: [webcompat]
Whiteboard: [webcompat] → [webcompat][webcompat-revisit]

Migrating Webcompat whiteboard priorities to project flags. See bug 1547409.

Webcompat Priority: --- → ?

See bug 1547409. Migrating whiteboard priority tags to program flags.

Assignee: kyle → nobody

Are there (wiki?) pages somewhere that would allow a contributor to read up on what's in the tree right now, what's missing, and how they can help get support for basic MIDI written, tested, approved, and landed?

Kyle, do you have any pointers on that?

Flags: needinfo?(kyle)

Until https://github.com/mozilla/standards-positions/issues/58 has some sort of resolution, there's not really much to do code-wise. The project is mostly stalled on spec/security issues.

Assuming that does come to a resolution... The base WebMIDI API, according to the spec version around January 2018, has been minimally implemented, but there is currently no native platform support. While the MIDI tests still run on every build, there's probably going to be some fixing needed to bring the API up to current Gecko standards, as well as implementing anything new/different that might've been added to the WebMIDI spec since.

As I am leaving and will most likely not be engaged with the project, I've handed this to :padenot. He's fairly busy right now with WebAudio concerns, so I'm not sure when he'll be able to continue work on this, as AFAIK this is considered very low priority work.

Flags: needinfo?(kyle)

If we can find a group of enthusiastic people with some coding chops, we can probably hand the work off to them with minimal oversight on the Moz side. The spec discussion itself (ignoring the conversations and debate around implementation in a browser, let alone FF specifically) seems to have stalled a bit on what to do with respects to sysex, so it might make sense to simply follow the Chrome team's example and treat that part of the spec as "to be determined at some point in time but we're just not going to bother with it for now". Getting FF at least up to parity with Chrome feels like a worthwhile effort, as long as we can get outside effort on it so that no one in Moz has to feel bad about pushing on a low priority task?

Webcompat Priority: ? → revisit
Whiteboard: [webcompat][webcompat-revisit]
Flags: webcompat?

As a developer who's also been a music producer as long as they've been a developer (started on a c64..) I'm definitely excited to start seeing APIs like this become available.

Re: Pmax on Sysex: I really think that for most browser-based things that sysex would not commonly be used, so, "figuring it out at some point" is not a bad thing at all. It's mostly only used for downloading firmware data to devices and at that usually only older devices.

As for "someone at Moz feeling bad for pushing on a low priority task" .. if I worked for Moz - even though this is considered "low priority" - this would be something that I definitely wouldn't feel bad about, in fact I'd be quite excited to work on it, I'd love to contribute some code but I'm not sure I'll have the resources, though I definitely would be open to joining Moz in some aspect (working for moz would be cool I guess, cool enough to take it over my current gig), I may not be a 'group', but I am enthusiastic and have a lifetime of coding and digital music experience. I'd truly love to see real audio applications able to run in a browser, something along the lines of Logic or Cubase or ProTools in a browser could be quite interesting, especially with the collaborative possibilities.

I suppose, let me know if/how I can be of help.

Re: sysex, for me it's actually one of the most compelling use cases for WebMIDI.

I run a site (https://emu.tools/) which currently offers two sysex-based applications:

  • "e-loader" is a firmware update tool for various E-mu synths and samplers that are no longer supported and for which the original Windows .exe tool from E-mu has difficulties running under modern versions of Windows, and their Mac version only supported pre-OSX (i.e. MacOS 9).

  • "e-remote" is a web-based "virtual front panel" for E-mu EOS samplers which uses E-mu's "PEPtalk" protocol (carried inside sysex) to both control the sampler and also reproduce the current contents of the LCD display and LED panel buttons

At the moment I'm having to tell site users that Firefox isn't supported.

There are plenty other sites using sysex for things like patch editors and librarians, including the ability to share patch libraries over the 'net.

There is more and more web applications for sound composition and livecoding performances that support webMIDI. Here is a selection:

Also from former mozillians who hate having to recommend Chrome just to be able to use their stuff:

One more from another former mozillians who hate having to recommend Chrome just to be able to use their stuff:
https://www.scorio.com A score writer for creating sheet music with input via webMIDI

Actually I was thinking of starting an implementation in my spare time. This is an itch I really want to scratch, mostly because there are so many great artists using it these days and it's a shame for them to be stuck with Chrome. No promises regarding timing though, it will take a while as long as I'm on my own.

Does Mozilla do Google Summer of Code?

(In reply to Gabriele Svelto [:gsvelto] from comment #78)

Actually I was thinking of starting an implementation in my spare time. This is an itch I really want to scratch, mostly because there are so many great artists using it these days and it's a shame for them to be stuck with Chrome. No promises regarding timing though, it will take a while as long as I'm on my own.

Hi, Gabriele
Where is the like/love-button in Bugzilla?

My C coding skills are quite limited, but I would love to contribute.
Please contact me if you need help with simple programming tasks, testing or MIDI/WebMIDI API spec interpretations!

Best regards,
Erik

(In reply to Gabriele Svelto [:gsvelto] from comment #78)

Actually I was thinking of starting an implementation in my spare time. This is an itch I really want to scratch, mostly because there are so many great artists using it these days and it's a shame for them to be stuck with Chrome. No promises regarding timing though, it will take a while as long as I'm on my own.

Whatever progress can be made on the subject will result in me forever being greatful!

I'm also using MIDI in my own project for a long time now and I would really love to stop having to tell people to use Chrome when using the on-screen keyboard: https://the-monochord.com/listen/1:1-100.0-200.0-300.0-400.0-500.0-600.0-700.0-800.0-900.0-1000.0-1100.0-2:1/name:12ED(2[slash]1)/labels:C-C[hashtag]-D-D[hashtag]-E-F-F[hashtag]-G-G[hashtag]-A-B-B[hashtag]-C/baseFrequency:262

I'm not a C coder (I can read the code, but not so much write it), but if in any other way I can help, then I'm game!

In case it wasn't mentioned before: There is a way to use WebMIDI in Firefox.

The solution is called Jazz-Plugin https://jazz-soft.net/download/Jazz-Plugin/ and works for Firefox on Windows, MacOS and Linux.
One has to download and install the plugin (which technically isn't a plugin, but it had been so, some time ago) and then to install two different Firefox Extensions: "Jazz-MIDI" and "Web Midi API".

I've tested Jazz-Plugin with our music notation website http://www.scorio.com and found it to be compatible with Chrome's WebMIDI implementation. Jazz-Plugin ist free (no money), but doesn't seem to be open source.

Conclusion: It works fine but it's a nuisance to have to install three components in two different places.

Perhaps someone could talk to the creator(s) of Jazz-Plugin if they were willing to donate some of their code? And get an honourable mention?

(In reply to Gabriele Svelto [:gsvelto] from comment #78)

Actually I was thinking of starting an implementation in my spare time. This is an itch I really want to scratch, mostly because there are so many great artists using it these days and it's a shame for them to be stuck with Chrome. No promises regarding timing though, it will take a while as long as I'm on my own.

This is great - also count me in if you need a hand. I've integrated native MIDI more times than I can count on Windows and macOS, but have never looked into contributing into Firefox. I would be happy to provide the OS-side implementation from a stub on the mentioned platforms.

What's left to do is to integrate with the OSes APIs indeed. This is best done in Rust, because it's really easy to integrate in the Firefox build system (one line, and binding generation to C++ is automatic). Rust is the preferred language for new features, especially for leaf modules that are quite low-level like this with a rather narrow interface.

https://github.com/Boddlnagg/midir looks more or less what we need. It has everything we need except and Android backend, that we can do as a followup and contribute. Its license (MIT) is compatible with Firefox's.

One current limitation of midir on Windows is that the port names are currently derived only from what Windows provides for the MIDI ports, and what Windows provides is pretty bad for USB MIDI in terms of informing the user of what they're selecting. Blink/Chromium addresses this by locating the parent device. For the midir WinRT mapping, I've tried to approximate this (by using other WinRT APIs rather than Win32), but I believe until https://github.com/microsoft/windows-rs/issues/81 is addressed it's not possible to actually call the API's necessary to do this.

(In reply to Andrew Sutherland [:asuth] (he/him) from comment #85)

I've tried to approximate this (by using other WinRT APIs rather than Win32), but I believe until https://github.com/microsoft/windows-rs/issues/81 is addressed it's not possible to actually call the API's necessary to do this.

We can always roll our own temporary wrappers just for the calls we need. I did it recently for accessing the WER API for example.

FYI if someone starts hacking on this before I do please know that I can mentor and help you along the way. You can always find me hanging out in the #introduction channel in chat.mozilla.org.

(In reply to Paul Adenot (:padenot) from comment #84)

What's left to do is to integrate with the OSes APIs indeed. This is best done in Rust, because it's really easy to integrate in the Firefox build system (one line, and binding generation to C++ is automatic). Rust is the preferred language for new features, especially for leaf modules that are quite low-level like this with a rather narrow interface.

https://github.com/Boddlnagg/midir looks more or less what we need. It has everything we need except and Android backend, that we can do as a followup and contribute. Its license (MIT) is compatible with Firefox's.

This sounds really exciting!
I'd really love to contribute in some way to getting this done, and I have already previously messed with midir specifically, but I am completely lost on where to start if I wanted to integrate this into the firefox codebase....

Where would I start? Also would I actually have to build alllll of firefox to be able to develop something like this?

(In reply to mnzaki from comment #88)

This sounds really exciting!
I'd really love to contribute in some way to getting this done, and I have already previously messed with midir specifically, but I am completely lost on where to start if I wanted to integrate this into the firefox codebase....

I'm vendoring midir in bug 1728436. Currently it needs some light patching which I hope I can push upstream. Once that's done the actual implementation work can start.

Where would I start?

By building Firefox. Then you can look into the already existing DOM API under the dom/midi, that's where it will need to be wired up. I will be actively working on this and I'm always glad to get some help, but if you're unfamiliar with Firefox the work can be a bit daunting. One thing that I'd be very grateful for is testing if you have access to MIDI-enabled devices.

Also would I actually have to build alllll of firefox to be able to develop something like this?

Unfortunately yes.

Webcompat Priority: revisit → P3
See Also: → https://webcompat.com/issues/88210

Does this mean WebMIDI is a higher priority now?

I don't think as P3 is one of the lowest priority (yes is not P4 after all)...

Context: This is about WebCompat Priority. not the Core bug Priority (which was already P3)
The Webcompat Team looks at the report and if the signal of breakage or reports because Midi is not implemented and people are saying I can do this in Browser X but not in Browser Z. We raise the priority. That's just one of the signals.

(In reply to Gabriele Svelto [:gsvelto] from comment #89)

One thing that I'd be very grateful for is testing if you have access to MIDI-enabled devices.

Incidentally, I have a large pile of MIDI devices; I'd be happy to test support if you get a test build together and send me an email. I can run on Mac and Windows.

(In reply to Chris Wilson from comment #93)

(In reply to Gabriele Svelto [:gsvelto] from comment #89)

One thing that I'd be very grateful for is testing if you have access to MIDI-enabled devices.

Incidentally, I have a large pile of MIDI devices; I'd be happy to test support if you get a test build together and send me an email. I can run on Mac and Windows.

So do I ... Windows 10 and 11 and both x86_64 and arm64 Linux.

Excellent! I've already started pulling in the dependencies in bug 1728436 and I've been working on the implementation for the past two weeks. I still have to iron out some issues related to the object lifetimes but after that point it's testing time!

(In reply to znmeb from comment #94)

(In reply to Chris Wilson from comment #93)

(In reply to Gabriele Svelto [:gsvelto] from comment #89)

One thing that I'd be very grateful for is testing if you have access to MIDI-enabled devices.

Incidentally, I have a large pile of MIDI devices; I'd be happy to test support if you get a test build together and send me an email. I can run on Mac and Windows.

So do I ... Windows 10 and 11 and both x86_64 and arm64 Linux.

So do I ... Windows 10 x86_64 and Ubuntu 20

Depends on: 1741373
Depends on: 1742471
Depends on: 1742559
Depends on: 1742635
Depends on: 1743462

if you're unfamiliar with Firefox the work can be a bit daunting. One thing that I'd be very grateful for is testing if you have access to MIDI-enabled devices.

I have a number of MIDI devices and could help test and provide feedback. I was brought here because I recently started using https://flat.io again, but would like to use it's MIDI functionality within Firefox (instead of it's suggestion of using Chrome).

I expect the initial implementation to land behind a pref this week. To everybody who's willing to test, thank you! Your testing will be precious because the very first version will most likely have bugs and limitations; your testing will be precious.

For a Firefox non-aficionado who would like to test:

  • What does "behind a pref" mean?
  • Where/how can I download Firefox with your implementation?
  • Which platforms (Mac/Win77/Linux) should be tested?

(In reply to johannes.feulner from comment #99)

For a Firefox non-aficionado who would like to test:

  • What does "behind a pref" mean?

Not enabled by default, but it's possible to enabling by turning a flag. One will have to go to about:config, search for webmidi and turn the flag dom.webmidi.enabled to true. In any case, it's not in any released build at the moment, but will. It's going to be announced here.

  • Where/how can I download Firefox with your implementation?

This is going to be in Firefox Nightly. It is a version that's released every day and has a lot more experimental stuff than regular builds.

  • Which platforms (Mac/Win77/Linux) should be tested?

Linux and macOS seem to work well, according to my own tests. I need to test Windows (from 7 up to 11) a bit more. A known limitation (that we intend to adress) is device hot-plugging, but (afaik) everything else works.

First test results running nightly 97.0a1 (2021-12-22) (64-Bit), testing with https://www.scorio.com/new-score

Windows 10: Listing of devices and MIDI-Input/Output works fine. My Korg micoKEY identifies as expected as "microKEY". "Microsoft GS Wavetable Synth" is properly detected. Same as with Google Chrome.

Ubuntu 20.04.03: Listing of devices and MIDI-Input/Output works fine. My Korg micoKEY identifies as "microKEY:microKey MIDI 24:0". This doubling of the name is true for other MIDI devices as well. This is different from Google Chrome.

As Paul mentioned hot plugging is not yet supported on either platform.

Very good work, I love it.

(In reply to johannes.feulner from comment #101)

First test results running nightly 97.0a1 (2021-12-22) (64-Bit), testing with https://www.scorio.com/new-score

Windows 10: Listing of devices and MIDI-Input/Output works fine. My Korg micoKEY identifies as expected as "microKEY". "Microsoft GS Wavetable Synth" is properly detected. Same as with Google Chrome.

Ubuntu 20.04.03: Listing of devices and MIDI-Input/Output works fine. My Korg micoKEY identifies as "microKEY:microKey MIDI 24:0". This doubling of the name is true for other MIDI devices as well. This is different from Google Chrome.

As Paul mentioned hot plugging is not yet supported on either platform.

Very good work, I love it.

Thank you, that's excellent!

Will do more extensive testing soon, but I can tell you that sending MIDI from a DAW like Ableton Live to a page in Firefox via the IAC Driver seems to work great! Thanks for all of your hard work on this!

Amazing! From what I can gather we're looking at 97 branch? If that's the case I can doll up the MDN docs \o/

@paul(In reply to Paul Adenot (:padenot) from comment #100)

Happy new year everyone!

Linux and macOS seem to work well, according to my own tests. I need to test Windows (from 7 up to 11) a bit more. A known limitation (that we intend to adress) is device hot-plugging, but (afaik) everything else works.

I just started testing on Windows - There are some promising beginnings but to be honest, I didn't expect Novation Components to work out of the box - There may be other Chromium dependencies we need to resolve within the app before Components will actually work on FF.

I can't make any promises (I'm an external contractor anyway) but I'll try my best to convice our QA team to do some initial testing soon on both Mac and Windows.

Paul, what specifically did you mean by "hotplugging will not work"? Does that mean I should restart the browser after plugging in a browser or does that simply refer to the relevant callbacks not being called, so reload should suffice?

In any case, y'all you can't imagine how happy this push for getting it done makes me! Well done to everyone involved!

(In reply to Jan Krutisch from comment #105)

Paul, what specifically did you mean by "hotplugging will not work"? Does that mean I should restart the browser after plugging in a browser or does that simply refer to the relevant callbacks not being called, so reload should suffice?

Hi Jan! Reloading the page is enough, this forces a re-enumeration of the devices attached to the machine. MidiAccess.onstatechange isn't called right now when a device is plugged to the machine, that's right.

I'm working on logging at the moment, so debugging complex applications should be easier in a few days. In any case, thanks for trying it out!

During my testing I found the implementation to be rather solid on Linux and Windows but I've encountered stability issues on macOS. I wouldn't recommend testing on that platform until we have sorted them out.

Depends on: 1748641

Do we want to add a relnote for this to the Fx97 beta relnotes? Please nominate with suggested wording if yes.

status-firefox66: affected → wontfix
Flags: needinfo?(gsvelto)

Running Version 98.0a1 (2022-01-13) (64-bit) on Ubuntu 20.04.3 LTS.
dom.webmidi.enabled = true
dom.webmidi.gated = true (also tried with false)
midi.testing = false (also tried with true)

When I open https://www.scorio.com/new-score, the only MIDI ports I get are:
Input device: Test Control MIDI Device Input Port
Output device: Test State MIDI Device Output Port / Always Closed MIDI Device Output Port / Test Control MIDI Device Output Port

On Chrome, I see the Keystation 49es MIDI 1 keyboard that I plugged in.

I've also tried with Timidity, snd-virmidi, etc. - none of those ports show up.

Can anyone help me figure out why I am not seeing any ports?

@Karim
For me it works well with Version 98.0a1 (2022-01-13) (64-bit) on Ubuntu 20.04.3 LTS.

midi.testing must be false. If true, the "Test ..." dummy devices will be shown.

Thanks, midi.testing = false does remove the dummy devices. However, I'm still not seeing what I am expecting. Here's my `aconnect -l`: ```

Thanks, midi.testing = false does remove the dummy devices. However, I'm still not seeing what I am expecting.

Below is the output of aconnect -l. Notice client 28: 'Keystation 49es' [type=kernel,card=3]. This does not show up in the devices list.

I am also testing with https://blog.karimratib.me/demos/sheetplayer/ which lists output ports and only shows me the 4 "Midi Through" ports, and http://mmontag.github.io/dx7-synth-js/ which detects no input port whatsoever (even though both work on Chrome).

client 0: 'System' [type=kernel]
    0 'Timer           '
    1 'Announce        '
	Connecting To: 129:0
client 14: 'Midi Through' [type=kernel]
    0 'Midi Through Port-0'
	Connecting To: 129:0, 128:0, 135:0, 131:0
	Connected From: 130:0, 141:0[real:0], 136:0[real:0]
    1 'Midi Through Port-1'
	Connecting To: 129:0, 138:0, 132:0
	Connected From: 130:1, 142:0[real:0], 145:0[real:0]
    2 'Midi Through Port-2'
	Connecting To: 129:0, 139:0, 133:0
	Connected From: 130:2, 143:0[real:0], 146:0[real:0]
    3 'Midi Through Port-3'
	Connecting To: 129:0, 140:0, 134:0
	Connected From: 130:3, 144:0[real:0], 151:0[real:0]
client 24: 'Virtual Raw MIDI 2-0' [type=kernel,card=2]
    0 'VirMIDI 2-0     '
	Connecting To: 129:0
	Connected From: 130:4
client 25: 'Virtual Raw MIDI 2-1' [type=kernel,card=2]
    0 'VirMIDI 2-1     '
	Connecting To: 129:0
	Connected From: 130:5
client 26: 'Virtual Raw MIDI 2-2' [type=kernel,card=2]
    0 'VirMIDI 2-2     '
	Connecting To: 129:0
	Connected From: 130:6
client 27: 'Virtual Raw MIDI 2-3' [type=kernel,card=2]
    0 'VirMIDI 2-3     '
	Connecting To: 129:0
	Connected From: 130:7
client 28: 'Keystation 49es' [type=kernel,card=3]
    0 'Keystation 49es MIDI 1'
	Connecting To: 129:0
client 128: 'TiMidity' [type=user,pid=15388]
    0 'TiMidity port 0 '
	Connected From: 130:8, 14:0
    1 'TiMidity port 1 '
	Connected From: 130:9
    2 'TiMidity port 2 '
	Connected From: 130:10
    3 'TiMidity port 3 '
	Connected From: 130:11
client 131: 'WebMIDI input' [type=user,pid=14732]
    0 'Input connection'
	Connected From: 14:0, 130:16
client 132: 'WebMIDI input' [type=user,pid=14732]
    0 'Input connection'
	Connected From: 14:1, 130:17
client 133: 'WebMIDI input' [type=user,pid=14732]
    0 'Input connection'
	Connected From: 14:2, 130:18
client 134: 'WebMIDI input' [type=user,pid=14732]
    0 'Input connection'
	Connected From: 14:3, 130:19
client 135: 'WebMIDI input' [type=user,pid=14732]
    0 'Input connection'
	Connected From: 14:0, 130:12
client 136: 'WebMIDI output' [type=user,pid=14732]
    0 'Output connection'
	Connecting To: 14:0[real:0], 129:0
client 138: 'WebMIDI input' [type=user,pid=14732]
    0 'Input connection'
	Connected From: 14:1, 130:13
client 139: 'WebMIDI input' [type=user,pid=14732]
    0 'Input connection'
	Connected From: 14:2, 130:14
client 140: 'WebMIDI input' [type=user,pid=14732]
    0 'Input connection'
	Connected From: 14:3, 130:15
client 141: 'WebMIDI output' [type=user,pid=14732]
    0 'Output connection'
	Connecting To: 14:0[real:0], 129:0
client 142: 'WebMIDI output' [type=user,pid=14732]
    0 'Output connection'
	Connecting To: 14:1[real:0], 129:0
client 143: 'WebMIDI output' [type=user,pid=14732]
    0 'Output connection'
	Connecting To: 14:2[real:0], 129:0
client 144: 'WebMIDI output' [type=user,pid=14732]
    0 'Output connection'
	Connecting To: 14:3[real:0], 129:0
client 145: 'WebMIDI output' [type=user,pid=14732]
    0 'Output connection'
	Connecting To: 14:1[real:0], 129:0
client 146: 'WebMIDI output' [type=user,pid=14732]
    0 'Output connection'
	Connecting To: 14:2[real:0], 129:0
client 151: 'WebMIDI output' [type=user,pid=14732]
    0 'Output connection'
	Connecting To: 14:3[real:0], 129:0

This might be related to an issue I've spotted while working on bug 1748647. It will be interesting to know if the issue persists after I fix that issue sometimes next week.

I turned OFF dom.webmidi.gated and it seems to show ports better on various web apps. Some additional observations:

  • I was surprised to see "WebMIDI input" in the output of aconnect -o, and "WebMIDI output" in the output of aconnect -i. I would have expected the opposite.

  • My main testing experiment is to hook the output of https://blog.karimratib.me/demos/sheetplayer/ (via a Midi Through port) to the input of https://mmontag.github.io/dx7-synth-js/ (via the same port). In Chrome, this works reliably, but in FF 98.0a1 I only hear a short sound then silence.

(In reply to Karim Ratib from comment #114)

I reproduced the issue on my machine and filed bug 1750763 to address it.

Flags: needinfo?(gsvelto)

Release Note Request (optional, but appreciated)
[Why is this notable]: This feature had been requested by users for a long time
[Affects Firefox for Android]: No, this is only for desktop
[Suggested wording]: We added experimental support for the Web MIDI API allowing musicians to interact with their instruments and tools via the browser
[Links (documentation, blog post, etc)]: https://www.w3.org/TR/webmidi/

relnote-firefox: --- → ?
Depends on: 1752584

Docs work for this can be tracked here: https://github.com/mdn/content/issues/12792

I'm trying to confirm what has been implemented, in what release, and behind what preferences. I've been using browserstack and first testing for the existence of midi string in about:config. If present, I have then been trying https://blog.karimratib.me/demos/sheetplayer/

Testing against dom.webmidi.enabled indicates it first appears in FF60.

  • This bug implies that this is just the "front end": a backend implementation is needed for each platform to actually do anything. Is that correct?
  • The linked bugs seem to indicate that backends were created for Linux, Windows and MacOS in FF97, and that Android does not yet have an implementation - is that correct?
  • What about iOS?
  • Are there any other platforms that still require backends?
  • Is there something better than https://blog.karimratib.me/demos/sheetplayer/ for testing this API?
  • Is the API specification complete? If not, which APIs are not implemented?

I did some testing, and it seems a little at odds with the above, raising some more questions:
2. Testing shows three midi preferences dom.webmidi.enabled, midi.tested, dom.webmidi.gated.

  • My assumption is that dom.webmidi.enabled is what turns on Midi. What do the others do? Note that only dom.webmidi.gated appears to be set true in any build by default.
  • dom.webmidi.enabled is false for all versions, including FF98 - my assumption is that this is not yet released in any build, including nightly. Correct?
  1. Testing shows that the preference dom.webmidi.enabled first appears on Windows in FF60 and the test sheetplayer outputs audio in FF60 on Browserstack
    • If the backend was provided in FF97, how can this test page play on browserstack in FF60?
  2. Similarly the test page does not play on FF97 on macOS on Browserstack. what might be going on?
Flags: needinfo?(ryanvm)

I'll answer the questions regarding the test page, which I have written:

If the backend was provided in FF97, how can this test page play on browserstack in FF60?

Note that the test page can send its output to either "local synth", or to some MIDI port that it has detected. For the purposes of this present work, we are not interested in "local synth" output, because this mode does not exercise the MIDI implementation. Instead, we expect that the "output" dropdown would show additional entries, corresponding to the output MIDI ports that FF is detecting via this implementation. So what you saw on FF60 is probably the "local synth" option working as expected.

Similarly the test page does not play on FF97 on macOS on Browserstack. what might be going on?

I tested on BrowserStack with FF97 on both macOS Monterey and Windows 11. After enabling dom.webmidi.enabled, I loaded the test page and saw the JS error: Web MIDI not enabled: SecurityError: The operation is insecure. This does not happen on my local Linux FF98, so I'm not sure if this is a FF97 issue, a BrowserStack issue, or an operating system issue - needs more investigation. But again, note that we are not interested in the "local synth" output of the test page - we're looking for MIDI ports instead.

In general, though, I doubt that BrowserStack can be a suitable test environment for this functionality. Because MIDI backends are tied to the host environment, the functionality that we're expecting here depends on BrowserStack's VM setup, which adds layers of uncertainty to the testing process. For example, running latest Chrome on Windows 11 in BrowserStack does not throw the security error above, but still does not show any MIDI port in the dropdown - which makes testing MIDI functionality impossible.

Is there something better than https://blog.karimratib.me/demos/sheetplayer/ for testing this API?

Ideally, we would want a test site that exercises the full Web MIDI API. My test page does not cover all cases. If there's interest, I'd be happy to work on specifying the test cases and writing a more comprehensive test site.

(In reply to Hamish Willee from comment #117)

Testing against dom.webmidi.enabled indicates it first appears in FF60.

  • This bug implies that this is just the "front end": a backend implementation is needed for each platform to actually do anything. Is that correct?
  • The linked bugs seem to indicate that backends were created for Linux, Windows and MacOS in FF97, and that Android does not yet have an implementation - is that correct?

Yes. We were hoping to be able to ship it in 97 but had to turn it off at the last minute due to two significant issues that cropped up during last-minute testing.

  • What about iOS?

I don't know the answer to that question but given that Firefox on iOS uses Apple's WebKit backend I suppose the answer is that it's not supported.

  • Are there any other platforms that still require backends?

No.

  • Is the API specification complete? If not, which APIs are not implemented?

We've implemented everything even though some events will never be delivered yet. Existing code should work. The implementation followed the public API draft but fell back on Chrome's existing behavior wherever it was different from the spec. We've been posting PRs to amend the spec so that it's consistent with what developers' have been using until now.

I did some testing, and it seems a little at odds with the above, raising some more questions:
2. Testing shows three midi preferences dom.webmidi.enabled, midi.tested, dom.webmidi.gated.

  • My assumption is that dom.webmidi.enabled is what turns on Midi. What do the others do? Note that only dom.webmidi.gated appears to be set true in any build by default.

dom.webmidi.gated causes the Web MIDI implementation to be gated by a new add-on based security policy. As you will see this will be the default.

  • dom.webmidi.enabled is false for all versions, including FF98 - my assumption is that this is not yet released in any build, including nightly. Correct?

Yes. We're now trying to enable it in version 98.

  1. Testing shows that the preference dom.webmidi.enabled first appears on Windows in FF60 and the test sheetplayer outputs audio in FF60 on Browserstack
    • If the backend was provided in FF97, how can this test page play on browserstack in FF60?

All versions prior to version 97 only contained a dummy implementation of Web MIDI so they can't really be tested against real code.

  1. Similarly the test page does not play on FF97 on macOS on Browserstack. what might be going on?

I'll have to look into that.

@Karim @Gabriele Thanks so much for taking the time to respond.

FYI only
I'm exploring right way to report this in browser compatibility data in https://github.com/mdn/browser-compat-data/pull/14874.
At the moment I am assuming that the versions prior to v97 should be ignored, except perhaps for a note that a dummy implementation exists.

@karim With respect to "I'd be happy to work on specifying the test cases and writing a more comprehensive test site."
That's a conversation to have with the dev team. My personal feeling is that for development teams, investing in wpt live tests would be a better use of time than any standalone site, because those are designed to test compatibility and is where people should be looking. At the moment there is nothing there except some sort of IDL test.

My colleague Ruth will be doing most of the rest of the work on this.

Flags: needinfo?(ryanvm)

Here is a simple Web MIDI test page with basic device hotplug support. It show the state and properties of the connected devices. It appears that the 'manufacturer' field is missing:
https://versioduo.com/webmidi-test/

A screenshot of Firefox 97.0 and the source repositiory is here:
https://github.com/versioduo/webmidi-test

See Also: → 1756549
Depends on: 1757153
Depends on: 1757796

I cannot make it work at all. When the navigator.requestMIDIAccess gets called I get this rejection all the time:

DOMException: The operation is insecure.

I tried (on three different environments - macOS 10.15.7, MacOS 12.2.1 and Fedora 33):

  • FF v97
  • FF Beta v98

Everywhere I enabled dom.webmidi.enabled flag and tried all combinations of dom.webmidi.gated and midi.testing flags - no luck.

The test page shared earlier (https://versioduo.com/webmidi-test/) shows the same error when loading from FF.

What am I missing?

(In reply to szmydadam from comment #122)

I cannot make it work at all. When the navigator.requestMIDIAccess gets called I get this rejection all the time:

DOMException: The operation is insecure.

I tried (on three different environments - macOS 10.15.7, MacOS 12.2.1 and Fedora 33):

  • FF v97
  • FF Beta v98

Everywhere I enabled dom.webmidi.enabled flag and tried all combinations of dom.webmidi.gated and midi.testing flags - no luck.

The test page shared earlier (https://versioduo.com/webmidi-test/) shows the same error when loading from FF.

What am I missing?

Can you try with a nightly version that includes the fix for bug 1757153?

(In reply to Gabriele Svelto [:gsvelto] from comment #123)

(In reply to szmydadam from comment #122)

I cannot make it work at all. When the navigator.requestMIDIAccess gets called I get this rejection all the time:

DOMException: The operation is insecure.

I tried (on three different environments - macOS 10.15.7, MacOS 12.2.1 and Fedora 33):

  • FF v97
  • FF Beta v98

Everywhere I enabled dom.webmidi.enabled flag and tried all combinations of dom.webmidi.gated and midi.testing flags - no luck.

The test page shared earlier (https://versioduo.com/webmidi-test/) shows the same error when loading from FF.

What am I missing?

Can you try with a nightly version that includes the fix for bug 1757153?

Unfortunately, it's still the same error (This operation is insecure). Tried v99.0a1 on macOS 10.15.7, MacOS 12.2.1 and Fedora 33.

(In reply to szmydadam from comment #122)

I cannot make it work at all. When the navigator.requestMIDIAccess gets called I get this rejection all the time:

DOMException: The operation is insecure.

Does it work if you manually grant the "Access MIDI devices *' permissions in 'Page Info'/Permissions?

It appears FF will never ask the user for it and just refuse the request right away (if it wasn't granted before). I just noticed that it stopped working after clearing all site settings.

I think that is the case! After I manually allowed the permissions as @Kai Sievers suggested, things are working (in v98 and v99).

Depends on: 1758468
status-firefox66: wontfix → ---
status-firefox98: --- → disabled
status-firefox99: --- → fixed
Target Milestone: --- → 99 Branch

Even with the manual permissions allowed I am seeing a different list of MIDI controllers that I am on Chrome. There are no outgoing messages sent or incoming messages received.

Release note wording tracked under Bug 1752906

relnote-firefox: ? → -

(In reply to theran.brigowatz from comment #127)

Even with the manual permissions allowed I am seeing a different list of MIDI controllers that I am on Chrome. There are no outgoing messages sent or incoming messages received.

I tried FF98 on MacOS 12.3, Windows 11, Fedora 35. All need manual assignment of permissions.

MacOS:

  • The currently connected devices work after FF startup
  • Reloading the working web page will show the devices, but appear to render all of them non-functional
  • No hotplug support, device lists will not be updated when devices are connected/disconnected

Windows:

  • The currently connected devices work after FF startup
  • Reloading the web page works
  • No hotplug support, device lists will not be updated when devices are connected/disconnected

Linux:

  • No devices show up

(In reply to Kay Sievers from comment #129)

MacOS:

  • Reloading the working web page will show the devices, but appear to render all of them non-functional

This is likely bug 1748696.

Linux:

  • No devices show up

This is odd, can you check if you have the /dev/snd/seq device and what its permissions are? Can you open a separate bug and we'll investigate it.

(In reply to Gabriele Svelto [:gsvelto] from comment #130)

This is odd, can you check if you have the /dev/snd/seq device and what its permissions are?

Yes, this should be fine on all more recent systems, filesystem AccessControlLists for logged in users grant access to all sound devices; getfacl shows them.

Oh, just realized that I tested with http instead of https on Linux. Instead of refusing the WebMIDI access, just the device lists seem empty in that case. Using http fails with a security error on MacOS.

(In reply to Kay Sievers from comment #129)

(In reply to theran.brigowatz from comment #127)

Even with the manual permissions allowed I am seeing a different list of MIDI controllers that I am on Chrome. There are no outgoing messages sent or incoming messages received.

I tried FF98 on MacOS 12.3, Windows 11, Fedora 35. All need manual assignment of permissions.

MacOS:

  • The currently connected devices work after FF startup
  • Reloading the working web page will show the devices, but appear to render all of them non-functional
  • No hotplug support, device lists will not be updated when devices are connected/disconnected

Windows:

  • The currently connected devices work after FF startup
  • Reloading the web page works
  • No hotplug support, device lists will not be updated when devices are connected/disconnected

Linux:

  • No devices show up

I was able to get rid of this strange device naming by setting midi.testing to false.

Depends on: 1760585

Status for enabling is in bug 1752906. Support is in the code, behind a pref. We didn't flip the pref yet because we want to do one more round of fixes.

Keywords: dev-doc-needed → dev-doc-complete
Severity: normal → S3

Apologies if this is in the wrong spot. It seems I need a site permission add-on to use Webmidi on Firefox and the links I’ve found to generate one return 404. Could anyone point me in the right direction?

What is the correct link to request site permission add-on (for Webmidi)?
The link on the MDN docs is https://extensionworkshop.com/documentation/publish/site-permission-add-on/ , and that gives you a 404 error.

The associated issue is MDN/content#21832 and the page is Web_MIDI_API. This is the same question as the comment above ^^^.

Flags: needinfo?(cmartin)

Changing NI to gsvelto, as I don't know almost anything about WebMIDI

Flags: needinfo?(cmartin) → needinfo?(gsvelto)

(In reply to Hamish Willee from comment #135)

What is the correct link to request site permission add-on (for Webmidi)?
The link on the MDN docs is https://extensionworkshop.com/documentation/publish/site-permission-add-on/ , and that gives you a 404 error.

The associated issue is MDN/content#21832 and the page is Web_MIDI_API. This is the same question as the comment above ^^^.

The add-on is not required anymore, Firefox generates one on-the-fly, as web page authors you shouldn't have to do anything else. The MDN page is outdated, I'll make sure it gets updated. In the meantime WebMIDI in the latest nightly builds should work out-of-the-box on your pages, it will just prompt users to install the auto-generated add-on once, and then will work transparently.

Flags: needinfo?(gsvelto)

Super happy to test this in the latest Beta. I can confirm that it works with Novation Components - up to a point. I have two questions:

  1. Is the wording during the addon installation still be tinkered with or are these considered to be final? They are potentially very scary without giving particularly good explanations to users what the app will actually do until relatively late in the process.
  2. It seems to me that hotplug support is still not working right now. I couldn't find (to be fair I've only skimmed this bug superficially) an issue specifically for this, so if someone could point it out to me so that I can track it and help with testing, that would be great.

(In reply to Jan Krutisch from comment #138)

Super happy to test this in the latest Beta. I can confirm that it works with Novation Components - up to a point.

Thanks!

  1. Is the wording during the addon installation still be tinkered with or are these considered to be final? They are potentially very scary without giving particularly good explanations to users what the app will actually do until relatively late in the process.

We recently tweaked it, so the current panel should begin with: "This site is requesting access to your devices. Device access can be enabled by installing an add-on." If you tested on an earlier beta you might have seen some more generic text.

It's difficult to articulate the threat model (malicious payload to non-hardened device, resulting in compromise and potential escalation to the host machine) in user-understandable terms, but we did our best with "Make sure you trust the site" and a bit more explanation in the "Learn More" link.

Happy to hear suggestions for improvements to the language, feel free to shoot me an email.

  1. It seems to me that hotplug support is still not working right now. I couldn't find (to be fair I've only skimmed this bug superficially) an issue specifically for this, so if someone could point it out to me so that I can track it and help with testing, that would be great.

Gabriele, can you link to the relevant bug?

Flags: needinfo?(gsvelto)
Depends on: 1802149
Flags: needinfo?(gsvelto)

(In reply to Bobby Holley (:bholley) from comment #139)

(In reply to Jan Krutisch from comment #138)

It's difficult to articulate the threat model (malicious payload to non-hardened device, resulting in compromise and potential escalation to the host machine) in user-understandable terms, but we did our best with "Make sure you trust the site" and a bit more explanation in the "Learn More" link.

Happy to hear suggestions for improvements to the language, feel free to shoot me an email.

Why is there a two step process to activate WebMIDI at all? First there is the intimidating warning without even mentioning WebMIDI. And only then follows the real question "This add-on gives example.com access to your MIDI devices" followed by another warning. The second warning is well unterstable in the context of allowing MIDI.

I'd suggest to get fully rid of the first step. From a user perspective it serves no purpose besides being intimidating.

It's really exciting to see WebMIDI in the 108 release! Really amazing work!

With the 108 release (and the nightly 110) I was initially not getting the site permission add-on prompt. This was working for me in nightly 109, but now navigator.requestMIDIAccess is pending for about a second, then throws an exception. No dialog asking for approval to install the add-on appears.

This seems to fix it:

  1. toggle midi.testing on, close browser, open browser. Now I get the site permission add-on dialog and can enumerate the fake devices
  2. toggle midi.testing off, close browser, open browser. The fix sticks. The site permission add-on dialog still works as it did in 109 and I can enumerate real devices

Initially I thought this was some settings problem I created in testing previous versions, but I tried it on a clean Windows 11 install and I get the same problem and the workaround fixes it. I also tried this out on Ubuntu 22 (official Firefox snap) and MacOS. Both show the same behavior.

Side note: as noted in comment 129, in MacOS you have to completely close Firefox to enumerate new devices. On Windows 11 and Ubuntu a simple reload sees the new interfaces.

(In reply to maccailein.mor from comment #141)

It's really exciting to see WebMIDI in the 108 release! Really amazing work!

With the 108 release (and the nightly 110) I was initially not getting the site permission add-on prompt. This was working for me in nightly 109, but now navigator.requestMIDIAccess is pending for about a second, then throws an exception. No dialog asking for approval to install the add-on appears.

Thanks for the detailed report. Can you file a new bug and we'll see if we can get to the bottom of it?

(In reply to Bobby Holley (:bholley) from comment #142)

(In reply to maccailein.mor from comment #141)

It's really exciting to see WebMIDI in the 108 release! Really amazing work!

With the 108 release (and the nightly 110) I was initially not getting the site permission add-on prompt. This was working for me in nightly 109, but now navigator.requestMIDIAccess is pending for about a second, then throws an exception. No dialog asking for approval to install the add-on appears.

Thanks for the detailed report. Can you file a new bug and we'll see if we can get to the bottom of it?

Ok done. Thanks.
https://bugzilla.mozilla.org/show_bug.cgi?id=1805582

Depends on: 1805582
Depends on: 1805735

I want to echo the concern over the prompts displayed to the user when installing the add-on in 108. The first prompt sets the developer up as worthy of suspicion without giving them a chance to display the requested resource. A user can only properly evaluate the threat model after getting to the second prompt, and they may leave before then out of fear, uncertainty, or doubt.

The first prompt lists these potential consequences:

This add-on could be used to steal your data or attack your computer.

This statement is a bit heavy-handed and may scare users away from indie web projects that do not have name recognition. I get that it will also scare them away from malicious sites, but perhaps there is a more balanced approach.

One alternative would be to make explicit to the user what is being requested and why it is more dangerous than their other interactions with the browser.

Here is my take on how this could be written as a single prompt:

This site is requesting access to your MIDI devices. Device access can be enabled by installing an add-on.

This access can be dangerous because it allows the site to run software and access resources outside of the browser. Only continue if you trust this site.

[Learn more about installing add-ons safely](Link to more info)

{Don't Allow/Never Allow} {Install add-on}

Hopefully, that is accurate technically. I think it would be enough to inform the user that they are taking a risk that requires additional trust. The "Learn More" link could get into more detail about the potential consequences.

I also had a question about the add-on scope. Does the add-on only grant access to top-level domains or is it scoped by subdomain? If the former is true, it could be an issue because services such as Netlify and Vercel publish sites using subdomains on a TLD they control.

A user could install the add-on for one Netlify app and grant permission to all apps on Netlify. Similarly, one malicious app could ruin everything for innocent apps published on one of these platforms.

(In reply to Brian G from comment #144)

One alternative would be to make explicit to the user what is being requested and why it is more dangerous than their other interactions with the browser.

Here is my take on how this could be written as a single prompt:

Thanks for the thoughtful comment. Would you mind filing a separate bug with your proposed language changes and we can discuss the trade-offs there?

I also had a question about the add-on scope. Does the add-on only grant access to top-level domains or is it scoped by subdomain? If the former is true, it could be an issue because services such as Netlify and Vercel publish sites using subdomains on a TLD they control.

A user could install the add-on for one Netlify app and grant permission to all apps on Netlify. Similarly, one malicious app could ruin everything for innocent apps published on one of these platforms.

It's scoped to subdomain. So for example, installing the add-on for https://webmidi-examples.glitch.me will not grant permissions on https://webmidi-examples-2.glitch.me , and vice-versa (those sites are live so you can try it out).

(In reply to Bobby Holley (:bholley) from comment #145)

Thanks for the thoughtful comment. Would you mind filing a separate bug with your proposed language changes and we can discuss the trade-offs there?

Yep, you bet! Filed it here: https://bugzilla.mozilla.org/show_bug.cgi?id=1808431

It's scoped to subdomain. So for example, installing the add-on for https://webmidi-examples.glitch.me will not grant permissions on https://webmidi-examples-2.glitch.me , and vice-versa (those sites are live so you can try it out).

Ah great, glad to hear it works that way.

It's really exciting that this is in Firefox now, thank you to everybody who made it happen.

Today I visited a site with WebMIDI enabled and a scary looking popup came up that didn't even mention anything about WebMIDI. I'll leave a report over on the other bug addressing the wording but I just wanted to say thanks for the feature itself, super cool.

Thank you all. I'm just leaving a note for people who arrive here via web search.

On Ubuntu GNU/Linux. In case your midi devices are not detected/listed, in my case it was because I was using a snap version. The MIDI functionality only worked the with the native deb version of Firefox. I'm not sure why.

The MIDI functionality only worked the with the native deb version of Firefox. I'm not sure why.

Can you file a bug about this? Can be here (Firefox Build System: Third Party Packaging) or with Ubuntu (this is probably on their end?). I think the Ubuntu Snap config will have to be changed to poke a hole for the MIDI devices in the Snap sandbox.

Depends on: 1817610
See Also: → 1834075
Depends on: 1857420

The MIDI functionality only worked the with the native deb version of Firefox. I'm not sure why.

Can you file a bug about this?

I did file a bug for that here: 1864243

Whiteboard: meta
Keywords: meta
Summary: Implement the WebMIDI API → [meta@ Implement the WebMIDI API
Whiteboard: meta
Summary: [meta@ Implement the WebMIDI API → [meta] Implement the WebMIDI API

Re the prompts displayed when MIDI support is requested.
See:
https://github.com/WebAudio/web-midi-api/issues/261#issuecomment-2125255875
Rejecting the MIDI permission should use NotAllowedError, not SecurityError.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: