1660936 - Display media key on the top layer of the touch bar

Status: NEW

(Core :: Widget: Cocoa, enhancement, P3)

Description

[Alastor Wu [:alwu]]

When enabling media control [1], we can see media keys showing on the touch bar (play/pause/next/prev). However, they are hiding in the second layer, users have to click "show more" button (a little arrow button) in order to see media keys.

I wonder if it's possible to display them on the top layer while Firefox is playing media, which can help users to control media easily.

[1] https://docs.google.com/document/d/1c4FivJpvAjjw9Uw-jn7X1UjGOoWkANXOulNyqDWs83w/edit#heading=h.k4qnw5lik9gx

Comment 1

[Alastor Wu [:alwu]]

Hi, harry,
I saw you worked on MacTouchBar.js before, do you know if it's possible to do that?
Thank you.

Comment 2

[Harry Twyford]

Touch Bar buttons have a limited ability to change based on the user's context. We do this with the "Search or enter address" button that turns into "Exit Fullscreen" when the user is in DOM fullscreen. If you could tell from a privileged JS context that the user is playing media, then you could create a button that turns into a play/pause button when the user is playing media, for example. There's no way that I know of to move Apple's "official" media buttons up a layer when media is playing.

If you didn't care about showing/hiding media buttons based on whether the page contained playing media, then this would be a lot easier. All you'd have to do then is add the media buttons to gBuiltInInputs in MacTouchBar.js. Buttons that didn't change based on context probably wouldn't be a great UX, but we could potentially add them as optional buttons, accessible in the Customize Touch Bar... menu. NB that we used to collect telemetry on Touch Bar buttons, and we learned that almost no one customizes the Touch Bar.

I wrote some documentation on the MacTouchBar API if you're interested in taking a look!

Comment 3

[Alastor Wu [:alwu]]

Thank you for the detailed information. We can do that from a privileged JS context, and we would try to see if we can get designed icons for media key from UX.

Comment 4

[Alastor Wu [:alwu]]

Hi, Harry,

Now I'm able to replace OpenLocation with media control button (use the way like exit fullscreen), however, the space allocated for OpenLocation is too large, it's weird to put only one media button on there.

I wonder how I can put multiple different buttons to replace OpenLocation (ex. play/pause/prev song/next song)? Or add a new button but modify the current button layout.

Thank you.

Comment 5

[Harry Twyford]

We don't currently have a way to replace a button with multiple buttons based on user context. I see two ways around this:

• Change the default button layout, like you said. You do this by modifying this array in nsTouchBar.mm.
• Introduce a popover with media buttons. A popover is a button that shows a new set of buttons when pressed. You can show/hide the contents of a popover programmatically: the set of search filters that appear when the Urlbar is focused are contained in a popover that is opened on the Urlbar focus event. It's defined in MacTouchBar.js as SearchPopover. There's actually a corresponding button for that popover; we just hide it by burying it at the end of the all the other buttons (it's a bit hacky :). So you could define a new input like

MediaPopover: {
  title: ...
  image: ...
  type: kInputTypes.POPOVER,
  children: {
    Play: {
      title: ...
      image: ...
      type: kInputTypes.BUTTON,
      callback: () => // Play media
    },
    Pause: {
      ...
    },
  },
}

Then you could show/hide it with gTouchBarUpdater.showPopover, maybe when a tab with media is foregrounded. Otherwise, if a media tab was in the background, the user could still access these controls by clicking the popover button. We collected telemetry on which Touch Bar buttons were used until 77. Now I can't seem to view that data for some reason... but iirc hardly anyone used the Forward, Reload, and Bookmark buttons. Your media popover could maybe replace one of those in the default set.

Whatever you decide, please talk to UX if you're changing the default layout of the Touch Bar! I'm happy to answer any other questions.

Comment 6

[Alastor Wu [:alwu]]

Following are some example of other applications' touch bar UI, while playing media.

Chrome : https://photos.app.goo.gl/CAv6RpFJqJWJpsiy7 (don't have media keys)
Firefox : https://photos.app.goo.gl/NY6AT3PCGXM9RnMs5
Safari : https://photos.app.goo.gl/9zibygNV8ikHYWCz5
Spotify : https://photos.app.goo.gl/wTVve8DJR1gzchWm6

---

Change the default button layout, like you said.
From my perpective, I don't want to always show media keys, I would like to show them only when Firefox is playing media, and the website has corresponding hanlder for media keys (for example, some sites support prev song and next song, but some don't. So we should only show those keys if sites support them) Is default layout in nsTouchBar.mm modifiable? Or it's fixed?

So there are some possible approaches,
(1) replace search bar with play/pause key : https://photos.app.goo.gl/Me1Ruteqr9XWxH8g7
But it looks not really good in visual.

(2) replace other default keys (prev page/next page/refresh) with play/pause key
Because other default keys are smaller, which seems more proper to put media keys. However, we have to decide if we want to put next song/prev song because they are not always valid. Or we just want to put play/pause key?

In addition, I've tried to use same way in (1), but didn't succeed to replace other default key.

(3) put media keys on popover (current search popover : https://photos.app.goo.gl/UsNKyaiNeS7tZPM48)
Showing media keys on pop up seems an easiest way, which allows us to dynamic change the layout depending on what media keys site supports. But media keys doesn't need a whole space, we just need a small space to show some buttons, so I don't know if it's proper or not.

In addition, I found that my popover can only work if I set title: "search-popover", if I rename my popover name to media-popover, then I can show my popover correctly. I wonder if it's because we have fixed popover id in nsTouchBar.mm [1].

The media popover has to be changed dynamically based on the playback state (showing pause or play button), but gBuiltInInputs seems a fixed object, so how can I change its child button dynamically?

[1] https://searchfox.org/mozilla-central/rev/27932d4e6ebd2f4b8519865dad864c72176e4e3b/widget/cocoa/nsTouchBar.mm#25-26

Comment 7

[Alastor Wu [:alwu]]

Hi, Adam, could you NI UX into this discussion? I've conclude some ideas in comment6.

Comment 8

[Alastor Wu [:alwu]]

Hi, Harry,
Before we receive further response from UX, I'm trying to prototype something by different approches, but got some issues. I mentioned them in (2) and (3) in comment6, could you help me check them?
Thank you.

Comment 9

[Harry Twyford]

(In reply to Alastor Wu [:alwu] from comment #6)

In addition, I found that my popover can only work if I set title: "search-popover", if I rename my popover name to media-popover, then I can show my popover correctly. I wonder if it's because we have fixed popover id in nsTouchBar.mm [1].

You would need to create a new popover ID. For example,

static NSTouchBarItemIdentifier MediaPopoverIdentifier =
    [TouchBarInput nativeIdentifierWithType:@"popover" withKey:@"media-popover"];

Then add MediaPopoverIdentifier to defaultItemIdentifiers. You could try replacing the Reload button, like so:

     NSArray* defaultItemIdentifiers = @[
        [TouchBarInput nativeIdentifierWithType:@"button" withKey:@"back"],
        [TouchBarInput nativeIdentifierWithType:@"button" withKey:@"forward"],
        MediaPopoverIdentifier,
        [TouchBarInput nativeIdentifierWithType:@"mainButton" withKey:@"open-location"],
        [TouchBarInput nativeIdentifierWithType:@"button" withKey:@"new-tab"],
        ShareScrubberIdentifier, SearchPopoverIdentifier
      ];

The media popover has to be changed dynamically based on the playback state (showing pause or play button), but gBuiltInInputs seems a fixed object, so how can I change its child button dynamically?

All Touch Bar inputs are updated in the observe method in MacTouchBar.js. Look at how we update the bookmark button when a page is bookmarked. We change the properties of the object representing the button, then we call this._updateTouchBarInputs(<input name>). So you could have a button in your media popover called PlayPause for example. When you detect via an event that media has started playing or was just paused, you could change the button. I think you'd then need to call _updateTouchBarInputs on the entire popover. Here's some pseudocode:

      case "media-playing":
        gBuiltInInputs.MediaPopover.PlayPause.image = "chrome://browser/skin/pause.svg";
        gBuiltInInputs.MediaPopover.PlayPause.callback = () => // Pause media
        this._updateTouchBarInputs("MediaPopover");
        break;
      case "media-paused":
        gBuiltInInputs.MediaPopover.PlayPause.image = "chrome://browser/skin/play.svg";
        gBuiltInInputs.MediaPopover.PlayPause.callback = () => // Play media
        this._updateTouchBarInputs("MediaPopover");
        break;

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

Redirect a needinfo that is pending on an inactive user to the triage owner.
:spohl, since the bug doesn't have a severity set, could you please set the severity or close the bug?

For more information, please visit auto_nag documentation.