704229 - [meta] Refine touch video controls

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect, P3)

Description

[Jared Wein [:jaws] (please needinfo? me)]

Please see bug 470628. We probably need a full-screen button on Fennec Native video controls.

Comment 1

[:Margaret Leibovic]

I don't think this needs to block. It wasn't a feature in XUL fennec, and you can still get to full screen from the context menu.

Comment 2

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Margaret Leibovic [:margaret] from comment #1)
> I don't think this needs to block. It wasn't a feature in XUL fennec, and
> you can still get to full screen from the context menu.

solid reasoning

Comment 3

[Mark Finkle (:mfinkle) (use needinfo?)]

Not sure this needs to track 16. Maybe tracking "+" ?

Comment 4

[:Margaret Leibovic]

I haven't had time to work on this myself, so I'm marking it as a mentor bug. It may not be a good *first* bug, but it should be relatively straightforward. A good starting point is definitely to take a look at bug 470628.

Comment 5

[Sucheta]

I wish to work on this particular bug.

Comment 6

[:Margaret Leibovic]

(In reply to Sucheta from comment #5)
> I wish to work on this particular bug.

Awesome!

Sucheta, do you have a build environment set up for Fennec? If not, you should see: https://wiki.mozilla.org/Mobile/Fennec/Android#Building_Fennec

Like I mentioned, this bug will be very similar to bug 470628, and actually touch some of the same files. The video controls are in a file shared by desktop and mobile [1], although there is a separate "touchControls" binding for mobile [2], which is what you'll want to change for this patch. I would start by trying to get a full screen button to show up in there, using the same button icon that was used for desktop. Once we see what that looks like we can get some help from UX to see if there's a different button style we want for mobile.

Feel free to leave questions in here, or attach a WIP ("work in progress") patch for feedback. You can also join us in #mobile on irc.mozilla.org for more real-time help.

[1] http://mxr.mozilla.org/mozilla-central/source/toolkit/content/widgets/videocontrols.xml
[2] http://mxr.mozilla.org/mozilla-central/source/toolkit/content/widgets/videocontrols.xml?force=1#1458

Comment 7

[Wesley Johnston (:wesj)]

Sucheta, hey. Just curious if you've had any time to play with this and have any questions. Happy to help!

Comment 8

[Jared Wein [:jaws] (please needinfo? me)]

Wes, can you take this bug? Without it, fullscreen videos on B2G lack a visible way to exit them.

If you can't take it, please reassign to someone who can get to it quickly.

Comment 9

[Wesley Johnston (:wesj)]

No problem. On paternity now, but I'll be back in next week and can take this. We'll at least need some artwork though (and these controls ideally need a fuller redesign on mobile, but we can put that off for a different bug).

Comment 10

[David Flanagan [:djf]]

I'm taking this bug because I've been assigned to work on bug 834931

Comment 11

[Wesley Johnston (:wesj)]

Since bug 834931 looks to be fixing this only for b2g, I'm going to hijack this to move the shared styles/images into toolkit. I'm also going to try and rework these to be a bit prettier. I'll post screenshots and hopefully ian can give me some feedback.

We'll put that into toolkit as our default "touch" theme, and products that want to alter it (maybe Android will want to use Android stock icons for play/pause/etc eventually) can extend or alter those.

Comment 12

[David Flanagan [:djf]]

It turns out we can fix this for FirefoxOS and Fennec separately, so bug 834931 no longer depends on this one. I'm reassigning to Wesley.

Wesley: note that for FirefoxOS, you can't rework stuff to be prettier without sending everything through the visual design team. I know that the FirefoxOS team would like to change these controls dramatically to make them similar to what we have in the FirefoxOS Video app. I don't know if Fennec has its own team of visual designers. If so, maybe you can get the Fennec and B2G designers to agree on a shared design.

Otherwise, it would be best to continue to give FirefoxOS a way to override the default styles and images, or you'll risk the wrath of the FirefoxOS visual design team.

Comment 13

[Peter La]

I've been told that Ian Barlow is working on media controls for Fennec currently, so I'll sync up with him to decide on a unified direction.  Please hold off on any work until we figure out the visuals.

Thanks!

Comment 14

[Wesley Johnston (:wesj)]

Attachment: WIP Patch

This is a WIP. We're actually getting closer and closer to just using the desktop controls with some interaction tweaks here. This switches things so that the touch controls look and work mostly fine using just the desktop theme images. We show the clickToPlay button for video's instead of having a play button in the control bar. It also moves the fullscreen button to the left and makes some tweaks to the scrubber bar.

It needs a better scrubber image (I faked one here using CSS), a better "pause" image for the click to play button, and potentially some larger icons for fullscreen/mute. Also, better rtl support, and there's some issue with the controls not showing at all in fullscreen...

I'll post some images to get UX feedback.

Comment 15

[Wesley Johnston (:wesj)]

Attachment: Video (paused)

Comment 16

[Wesley Johnston (:wesj)]

Attachment: Audio (paused)

Comment 17

[Jared Wein [:jaws] (please needinfo? me)]

Comment on attachment 711033 [details]
Audio (paused)

Both the audio and video look good to me (note that the reason the audio image is showing a larger than necessary box is due to bug 704326).

Does the volume control need to be present for touch? As far as I know, most touch devices have hardware volume controls. If the video element has focus, can we use the hardware volume controls to control the video volume? (or just set video volume to 100% and then let hardware controls modify system volume?)

Comment 18

[Wesley Johnston (:wesj)]

Attachment: WIP v2

Getting better. This fixes the default images and sorta fixes the fullscreen bug (our controls are getting reinitialized, but the clickToPlay button isn't hidden correctly leading to crazyness, I fixed it by force-showing the controls any time you click and they aren't currently shown).

Biggest fix is getting some slightly better images. I did some SVG spriting tricks to reuse the click-to-play svg stuff.

Comment 19

[Wesley Johnston (:wesj)]

Whoops. Meant to post, build with these is at:

https://dl.dropbox.com/u/72157/videoControls.apk

Comment 20

[Wesley Johnston (:wesj)]

This is blocked on getting feedback from fennec+b2g ux teams. In the long run, we can slip images/size/border/color/etec differences in and our for different platforms. Initially, we just need something that looks good enough to be our "base" theme, and more importantly, to have the correct layout/interactions.

Comment 21

[Wesley Johnston (:wesj)]

Maybe peter can help unfreeze this?

Comment 22

[Wesley Johnston (:wesj)]

Ping?

Comment 23

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

Will mock up player controls for both small embedded and full screen videos that take into consideration FFOS controls and current FFX Android/Desktop controls.

Comment 24

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

As per ^, http://cl.ly/image/1h42163O121T

Based off of ibarlow's mockup, included new play and pause icons that I made but didn't get the chance to redo the 'full screen' icon yet so I'm using the FFOS one I found for the time being.

Wesj is in the process of building out a quick prototype to test it out on actual devices.

Comment 25

[Wesley Johnston (:wesj)]

Attachment: Screenshot

Still trying to fix this up a lot, but wanted to give UX something. WIP:
http://people.mozilla.org/~wjohnston/video.apk

Some test files;
http://people.mozilla.org/~wjohnston/videos/big_buck_bunny.webm
http://people.mozilla.org/~wjohnston/videos/gizmo.mp4
http://people.mozilla.org/~wjohnston/videos/oceans_clip.webm

Note the play/pause buttons are wrong. I can move in some images that look more like the screenshot. We have a similar image, but its a java resource, not available here. Its also not SVG like the image desktop uses here.

I think if we're going to change these images, we should do it uniformly across Firefox Desktop/OS/Android, so I'm a bit nervous about changing them at all :(

I also think I want to hide the play button based on the size of the video (currently its a portrait/landscape decision).

Comment 26

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

As per wesj's comments, as it stands right now we're going to try and use the FX Desktop icons for the media controls and get a build going to test out the issues with different (namely really small) video sizes. 

Reusing icons also helps build some visual consistency between our products.

Comment 29

[Kevin Brosnan [Ex-Mozilla]]

We should reconsider this as part of the video improvements.

Comment 30

[Mark Finkle (:mfinkle) (use needinfo?)]

Let's stop trying to unify and just get these refinements done.

Comment 31

[Ian Barlow (:ibarlow)]

Help us antlam you're our only hope

Comment 32

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

Attachment: prev_videocont_mock2.png

Attaching a WIP for this first...

Main points: FS icon (unFS icon not in the mock but I have one), mute/unmute icon, new icons all around (hand crafted just for us).

As a middle ground (since we know the scrubber is finicky, I came up with a compromise in the middle but I think we can push for something a little sweater (far right). 

On that note, Far right mock opens up space along the bottom for future "cast" icon, additional sliders (volume?), etc. This is also still missing video clip progress and length indicators so I will be trying to add those in. But I also foresee a lot of tweaking especially around smaller video sizes where we can't surface all these controls.

Stay tuned!

Comment 33

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

Attachment: prev_mediacont_icons.png

Also attaching a quick snap shot at what our lib of media controller icons are looking like ATM

Comment 34

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

Attachment: prev_videocont_mock3.png

Talked to Ian about this a little bit and made some iterations. Attaching a WIP so at least we can have something to move the needle on.

Takeaways from this are mainly the "Proposed" and "Proposed 2". I want to see if we can handle "medium" sized videos like this (i.e. move the controls around). The visuals are still not 100% yet doing some crits on it with Ian about the look & feel but this is the direction. "Proposed 2" is the state that shows when the scrubber is active, putting into the foreground the spot in the video in which the scrubber is at (0:07 in the mock).

"Landscape FS" is simply full screen mode. Only thing out of the ordinary here is maybe the double "play" buttons. I simply want to leverage the entire video as a "play/pause button" so it would be easier to trigger as this is a common interaction.

Lastly, "Proposed (small)" is where it gets interesting and essentially what we talked about Wes. We sometimes run into this situation where the video is too small to put all the controls in there so I want to explore if we can do something smarter and display the appropriate controls in the appropriate videos. 

In this particular WIP, I'm experimenting with pulling out all the controls and floating a media control bar at the bottom (not unlike our treatment with second screen initiatives like Roku). I'm open to some feedback here of course as I realize this would have many implementation implications.

Comment 35

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

Attachment: prev_mediacont_icons3.png

Attaching an updated overview of our current media icons as it stands

Comment 36

[Ian Barlow (:ibarlow)]

(In reply to Anthony Lam (:antlam) from comment #34)
> Created attachment 8439543 [details]
> prev_videocont_mock3.png

Looks awesome!

Comment 37

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

Attachment: prev_videocont_mock4.png

Wanted to nudge this bug along as it's a relatively quick win it seems.

Attaching design mocks so we can move forward with this... Please ignore the dotted line around the video that was on the screen shot hah. :P  

This mock will contain info just on videos that are large enough to put controls on. I'll follow this up with what we can hopefully do for smaller videos in my next comment.

The controls in the middle mock sit on a #363B40, 0.45 opacity background matte that goes from the bottom of the scrubber track to the bottom of the screen (this is to help ensure controls visibility). 

Proposed 2 (Right mock) -  
When the user picks up the scrubber, the orange surrounding it enlarges, and the time/place is displayed in Roboto, Light on white (42 sp in the mock). The overlay is the same (#363B40) but at 0.7 opacity and is over the entire video. Letting go fades away the coverlay and resumes the video.

Comment 38

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

Attachment: prev_videocont_mock4b.png

^Part 2!

Here is the mock for full screen and smaller videos.

I would like to try and pull out the video controls on websites that have tiny videos playing. Not unlike our interface for our second screen work, the bar that contains our video controls would be pinned to the bottom (#363B40 1.0 opacity).

For full screen, the background matte for the controls bar (#363B40) would revert to 0.45 opacity. Ideally, the entire area above the bar would be used as a hit area for play/pause as well.

Let me know as questions pop up.

Comment 39

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

Friendly ping on the status here :)

Comment 40

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

Attachment: icon_pbar_mob.zip

Attaching some icons (+ more) that I've used over on bug 1055809.. we could re-use these for our video player (when the video is too small, as outlined above :)

Comment 41

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

Attachment: icon_fs_mob.zip

adding icons for FS mode (bigger)

Comment 42

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

NI-ing Mark :)

Just wondering - these video controls seem to surface themselves more often now that we have the second screen projects picking up for 34.. should we spend some time on these controls now?

Comment 43

[Mark Finkle (:mfinkle) (use needinfo?)]

Just be careful. The video element controls can become a slipper slope. Focus on small sets of changes that give incremental improvements. Also note that even though the Media ControlBar (for casting) and the video element controls look similar, they are different formats and have different UI limitations.

In general, I think it's fine to work to make the controls look the same, but the work is not a high priority (does not need to ship with fx33 or fx34).

Comment 47

[:Margaret Leibovic]

What is the state of the mock-ups in this bug? Are they still up to date?

What are the changes we want to make here? If it's just about updating the icons, we can do that in bug 1065076.

Maybe we should turn this into a meta bug, and track more concrete pieces of work in separate bugs.

Comment 48

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

(In reply to :Margaret Leibovic from comment #47)
> What is the state of the mock-ups in this bug? Are they still up to date?
> 
> What are the changes we want to make here? If it's just about updating the
> icons, we can do that in bug 1065076.
> 
> Maybe we should turn this into a meta bug, and track more concrete pieces of
> work in separate bugs.

WFM.

Trying to remember right now and I think the changes we want to make here are focused around the player's behaviours as a whole and bug 1065076 is just about icons..

Comment 49

[:Margaret Leibovic]

I don't think there's anything else left to do here. If there are more improvements we want to me, let's file a new bug.

Comment 50

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Added to Fx48 Aurora release notes

Comment 51

[Sorina Florean [:sflorean]]

Hi all, I will take this feature as a QA. Here is the Test plan based on which the testing is made: 
https://wiki.mozilla.org/QA/Fennec/Update_icons_for_video_touch_controls