938149 - Screenrecorder utility for FirefoxOS

Status: RESOLVED DUPLICATE of bug 1144103

(Firefox OS Graveyard :: General, defect, P4)

Description

[William Lachance (:wlach)]

Android 4.4 apparently adds the ability to do a screen recording via a command-line adb utility:

It would be nice to have a similar capability in FirefoxOS, for Eideticker and other QA purposes.

See:

http://developer.android.com/about/versions/kitkat.html#44-screen-recording
http://androidxref.com/4.4_r1/xref/frameworks/av/cmds/screenrecord/screenrecord.cpp

Comment 1

[William Lachance (:wlach)]

See also bug 933987, which tracks adding Eideticker support for this new capability in Android.

Comment 2

[cmtalbert]

I'd be really cautious with this. We'll need to measure the performance impact of the recording system versus using a camera. But, if we can get good enough data out of it that is comparable to our camera/HDMI runs, it could be a big win, of course we'll also have to wait for a android 4.4 based firefox OS port.

Comment 3

[Sotaro Ikeda [:sotaro]]

FYI:
DisplayDevice and VirtualDisplaySurface related diagram for android 4.4.
https://github.com/sotaroikeda/android-diagrams/blob/master/graphics/DisplayDevice_4.4.pdf?raw=true

Another android related diagrams.
  https://github.com/sotaroikeda/android-diagrams/wiki/Android-Diagrams

Comment 5

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

FWIW, I don't think we'll get anywhere near the HDMI/eideticker style recording on the current generation FxOS devices.  But we may need to do something similar for some other partners.

Comment 6

[Chiajung Hung [:chiajung]]

I am working on make snapshot function goes to Compositor to enhance performance and avoid OMTA problem. For full utility, we may need a WebAPI to trigger recording or wait for Bug 952359 to create new GetUserMedia source to let JS side able to streaming captured frame.

Comment 7

[William Lachance (:wlach)]

(In reply to Chiajung Hung [:chiajung] from comment #6)
> I am working on make snapshot function goes to Compositor to enhance
> performance and avoid OMTA problem. For full utility, we may need a WebAPI
> to trigger recording or wait for Bug 952359 to create new GetUserMedia
> source to let JS side able to streaming captured frame.

A screencasting feature that you could access via WebAPI would be cool, but to be clear I filed this bug about a developer-only feature that you can access via adb. Even if we do expose this functionality with a WebAPI, I would still want a command-line version because that's much easier to test and automate.

Comment 8

[Peter Chang[:pchang]]

(In reply to William Lachance (:wlach) from comment #7)
> (In reply to Chiajung Hung [:chiajung] from comment #6)
> > I am working on make snapshot function goes to Compositor to enhance
> > performance and avoid OMTA problem. For full utility, we may need a WebAPI
> > to trigger recording or wait for Bug 952359 to create new GetUserMedia
> > source to let JS side able to streaming captured frame.
> 
> A screencasting feature that you could access via WebAPI would be cool, but
> to be clear I filed this bug about a developer-only feature that you can
> access via adb. Even if we do expose this functionality with a WebAPI, I
> would still want a command-line version because that's much easier to test
> and automate.

Right now I purpose to use getUserMedia API for screen recording and I would like to put the screen recorder switch on status bar(system app) and associate with a preference.

william, any comment about this approach? If you still require the adb commandline, need to figure out how to change the system app's preference.

Comment 9

[William Lachance (:wlach)]

Hi Peter, I am afraid that a webapi is not sufficient for my needs here and will need an adb commandline utility. I know this is more work. I'm not sure what the system app's preference has to do with this though?

Comment 10

[Chiajung Hung [:chiajung]]

(In reply to William Lachance (:wlach) from comment #9)
> Hi Peter, I am afraid that a webapi is not sufficient for my needs here and
> will need an adb commandline utility. I know this is more work. I'm not sure
> what the system app's preference has to do with this though?

To use command line to trigger such function, Compositor will need some change, which we can discuss what change to do later.

Discuss how to trigger it first:
(1) If you insist a command line binary like Android's system/bin/screenrecorder we will need a program like rilproxy to trigger screenrecoder. As B2G IPC is total different from Android Binder, in which any process can access system services via simple API. However, I think we have no way to build IPC channel if the process is not forked from B2G process(and or Nuwa process).
(2) If you think Android's setprop(which an called from command line, too) or something similar to that is okay for you, then it would be easier to do.
(3) It you think trigger from Settings app is okay for you, then Peter's proposal should be good for you.

Comment 11

[Peter Chang[:pchang]]

(In reply to William Lachance (:wlach) from comment #9)
> Hi Peter, I am afraid that a webapi is not sufficient for my needs here and
> will need an adb commandline utility. I know this is more work. I'm not sure
> what the system app's preference has to do with this though?

William, I just got the idea how to enable screen recorder function through system app.
The following is the simple flow to enable, let me know if you have any concern
a. Turn on "Screen Recorder support" from Developer option
b. Then new icon for "screen recorder" added when you drag down status bar, just like the wifi/3g icons...
c. Turn on/off the screen recorder icon

Comment 12

[William Lachance (:wlach)]

(In reply to peter chang[:pchang][:peter] from comment #11)
> (In reply to William Lachance (:wlach) from comment #9)
> > Hi Peter, I am afraid that a webapi is not sufficient for my needs here and
> > will need an adb commandline utility. I know this is more work. I'm not sure
> > what the system app's preference has to do with this though?
> 
> William, I just got the idea how to enable screen recorder function through
> system app.
> The following is the simple flow to enable, let me know if you have any
> concern
> a. Turn on "Screen Recorder support" from Developer option
> b. Then new icon for "screen recorder" added when you drag down status bar,
> just like the wifi/3g icons...
> c. Turn on/off the screen recorder icon

I don't think this will work well for what I need to do. For eideticker, I really need console output of the frame # being captured (similar to how this utility works: https://github.com/mozilla/eideticker/blob/master/src/videocapture/videocapture/pointgrey/pointgrey-capture.cpp#L201), for precise correlation between the start/end of a test and the videocapture. I am worried about the latency involved in communicating with the gaia layer with something like marionette will make what we're capturing too non-deterministic.

If a command-line utility seems like it would be a lot of work (it sounds like it is), maybe we should just defer this for now. I plan to prototype a version of eideticker which uses kitkat's screenrecorder (bug 933987). The results that we get from doing that should give us some indication on whether this is really worth the effort.

Comment 13

[Chiajung Hung [:chiajung]]

For B2G case, if we don't care architecture, we can let CompositorThread read a property, and render to SurfaceMediaSource get from mediaserver and make it done. 

I think Peter is propose a more generic way to do ScreenRecording, which may be later leveraged by GetUserMedia and avoid platform dependency at all. Since it will use standard API for video compression(MediaRecoder API) and trigger screen capturing(I think screen source is not in GetUserMedia spec currently, but it seems that Chrome supports it already).

Comment 14

[Peter Chang[:pchang]]

(In reply to William Lachance (:wlach) from comment #12)
> (In reply to peter chang[:pchang][:peter] from comment #11)
> > (In reply to William Lachance (:wlach) from comment #9)
> > > Hi Peter, I am afraid that a webapi is not sufficient for my needs here and
> > > will need an adb commandline utility. I know this is more work. I'm not sure
> > > what the system app's preference has to do with this though?
> > 
> > William, I just got the idea how to enable screen recorder function through
> > system app.
> > The following is the simple flow to enable, let me know if you have any
> > concern
> > a. Turn on "Screen Recorder support" from Developer option
> > b. Then new icon for "screen recorder" added when you drag down status bar,
> > just like the wifi/3g icons...
> > c. Turn on/off the screen recorder icon
> 
> I don't think this will work well for what I need to do. For eideticker, I
> really need console output of the frame # being captured (similar to how
> this utility works:
> https://github.com/mozilla/eideticker/blob/master/src/videocapture/
> videocapture/pointgrey/pointgrey-capture.cpp#L201), for precise correlation
> between the start/end of a test and the videocapture. I am worried about the
> latency involved in communicating with the gaia layer with something like
> marionette will make what we're capturing too non-deterministic.
> 
> If a command-line utility seems like it would be a lot of work (it sounds
> like it is), maybe we should just defer this for now. I plan to prototype a
> version of eideticker which uses kitkat's screenrecorder (bug 933987). The
> results that we get from doing that should give us some indication on
> whether this is really worth the effort.

I just checked the eideticker, so you want the frame number and the same frame rate as display, like 60 fps. Am I right? I think the screen recorder from android doesn't provide the frame num and also the frame rate is limited to mp4 which is less than 60 fps.

Comment 15

[William Lachance (:wlach)]

(In reply to peter chang[:pchang][:peter] from comment #14)
> (In reply to William Lachance (:wlach) from comment #12)
> > (In reply to peter chang[:pchang][:peter] from comment #11)
> > > (In reply to William Lachance (:wlach) from comment #9)
> > > > Hi Peter, I am afraid that a webapi is not sufficient for my needs here and
> > > > will need an adb commandline utility. I know this is more work. I'm not sure
> > > > what the system app's preference has to do with this though?
> > > 
> > > William, I just got the idea how to enable screen recorder function through
> > > system app.
> > > The following is the simple flow to enable, let me know if you have any
> > > concern
> > > a. Turn on "Screen Recorder support" from Developer option
> > > b. Then new icon for "screen recorder" added when you drag down status bar,
> > > just like the wifi/3g icons...
> > > c. Turn on/off the screen recorder icon
> > 
> > I don't think this will work well for what I need to do. For eideticker, I
> > really need console output of the frame # being captured (similar to how
> > this utility works:
> > https://github.com/mozilla/eideticker/blob/master/src/videocapture/
> > videocapture/pointgrey/pointgrey-capture.cpp#L201), for precise correlation
> > between the start/end of a test and the videocapture. I am worried about the
> > latency involved in communicating with the gaia layer with something like
> > marionette will make what we're capturing too non-deterministic.
> > 
> > If a command-line utility seems like it would be a lot of work (it sounds
> > like it is), maybe we should just defer this for now. I plan to prototype a
> > version of eideticker which uses kitkat's screenrecorder (bug 933987). The
> > results that we get from doing that should give us some indication on
> > whether this is really worth the effort.
> 
> I just checked the eideticker, so you want the frame number and the same
> frame rate as display, like 60 fps. Am I right? I think the screen recorder
> from android doesn't provide the frame num and also the frame rate is
> limited to mp4 which is less than 60 fps.

Yes, on the frame number. As far as frame rate, it depends on the test. We can generate useful results for things like startup tests (e.g. http://eideticker.mozilla.org/b2g/#/inari/b2g-contacts-startup/timetostableframe) with much less than 60fps as we don't need 1/60 precision.

Comment 16

[Peter Chang[:pchang]]

(In reply to William Lachance (:wlach) from comment #15)
> (In reply to peter chang[:pchang][:peter] from comment #14)
> > (In reply to William Lachance (:wlach) from comment #12)
> > > (In reply to peter chang[:pchang][:peter] from comment #11)
> > > > (In reply to William Lachance (:wlach) from comment #9)
> > I just checked the eideticker, so you want the frame number and the same
> > frame rate as display, like 60 fps. Am I right? I think the screen recorder
> > from android doesn't provide the frame num and also the frame rate is
> > limited to mp4 which is less than 60 fps.
> 
> Yes, on the frame number. As far as frame rate, it depends on the test. We
> can generate useful results for things like startup tests (e.g.
> http://eideticker.mozilla.org/b2g/#/inari/b2g-contacts-startup/
> timetostableframe) with much less than 60fps as we don't need 1/60 precision.

From eideticker, it looks like you want 
a. the frame number count from compositor
b. the related timestamp
c. the raw frame
For the startup tests, I think item a and b could help. But what is the purpose for item c?

william ,except startup testing, are there any other scenarios to use screenrecorder?

Comment 17

[William Lachance (:wlach)]

Sorry for the late reply...

(In reply to peter chang[:pchang][:peter] from comment #16)

> > > I just checked the eideticker, so you want the frame number and the same
> > > frame rate as display, like 60 fps. Am I right? I think the screen recorder
> > > from android doesn't provide the frame num and also the frame rate is
> > > limited to mp4 which is less than 60 fps.
> > 
> > Yes, on the frame number. As far as frame rate, it depends on the test. We
> > can generate useful results for things like startup tests (e.g.
> > http://eideticker.mozilla.org/b2g/#/inari/b2g-contacts-startup/
> > timetostableframe) with much less than 60fps as we don't need 1/60 precision.
> 
> From eideticker, it looks like you want 
> a. the frame number count from compositor
> b. the related timestamp
> c. the raw frame
> For the startup tests, I think item a and b could help. But what is the
> purpose for item c?

So usually you start recording before the actual test (and there is also a bit of lag between stopping the test and stopping the recording), so we need to be able to correlate the frame number when the test started to crop the capture to the area of interest (so we can actually measure how long things take).

> william ,except startup testing, are there any other scenarios to use
> screenrecorder?

It's hard to say for sure, but likely we could still do some of the tests we currently run to measure checkerboarding and jank even with a lower frame rate. Obviously a higher frame rate would be highly desirable for these.

Comment 18

[:gerard-majax]

Bug 1144103 made it possible to use screenrecord; maybe something can be built on top of that?

Comment 19

[William Lachance (:wlach)]

Yeah bug 1144103 basically delivers what I was asking for here.

I'm pretty sure you could make Eideticker work with that. Pretty neat.